Experimental evidence shows that site fertility is a key modulator underlying plant community changes under climate change. Communities on fertile sites, with species having fast dynamics, have been found to react more strongly to climate change than communities on infertile sites with slow dynamics. However, it is still unclear whether this generally applies to high-latitude plant communities in natural environments at broad spatial scales. We tested a hypothesis that vegetation of fertile sites experiences greater changes over several decades and thus would be more responsive under contemporary climate change compared to infertile sites that are expected to show more resistance. We resurveyed understorey communities (vascular plants, bryophytes, and lichens) of four infertile and four fertile forest sites along a latitudinal bioclimatic gradient. Sites had remained outside direct human disturbance. We analyzed the magnitude of temporal community turnover, changes in the abundances of plant morphological groups and strategy classes, and changes in species diversity. In agreement with our hypothesis, temporal turnover of communities was consistently greater on fertile sites compared to infertile sites. However, our results suggest that the larger turnover of fertile communities is not primarily related to the direct effects of climatic warming. Furthermore, community changes in both fertile and infertile sites showed remarkable variation in terms of shares of plant functional groups and strategy classes and measures of species diversity. This further emphasizes the essential role of baseline environmental conditions and nonclimatic drivers underlying vegetation changes. Our results show that site fertility is a key determinant of the overall rate of high-latitude vegetation changes but the composition of plant communities in different ecological contexts is variously impacted by nonclimatic drivers over time.