Community-scale impacts of glyphosate-based herbicides on wetland plant communities and the magnitude of those impacts that should be considered biologically relevant are poorly understood. We contrast three different thresholds for setting biologically meaningful critical effect sizes for complex ANOVA study designs. We use each of the of the critical effect sizes to determine optimal α levels for assessment of how different concentrations of glyphosate-based herbicides affect wetland plant communities over two years of herbicide application (alone and in combination with agricultural fertilizers) and two subsequent years without herbicide (or fertilizer) application. The application of glyphosate-based herbicides was found to result in a decrease in macrophyte species richness, an increase in macrophyte species evenness, a decrease in macrophyte cover and a reduction in community similarity. There was little evidence that nutrient additions directly or indirectly affected plant community endpoints. The glyphosate effects were evident in the first year of herbicide application in 2009, and became more pronounced in the second year of herbicide application in 2010. However, when herbicides were not applied in 2011, recovery was observed in most endpoints, with the exception being species evenness, for which partial recovery was not observed until 2012. Optimal α levels differed among the three critical effect sizes for each ANOVA term and endpoint combination, however regardless of differences in α levels, conclusions were generally consistent across all critical effect sizes.