Many zeolitic imidazolate frameworks (ZIFs) are promising candidates for use in separation technologies. Comprising large cavities interconnected by small windows they can be used, at least in principle, as molecular sieves where molecules smaller than the window size are able to diffuse into the material while larger molecules are rejected. However, "swing effect" or "gate opening" phenomena resulting in an enlargement of the windows have proven to be detrimental. Here, we present the first systematic experimental and computational study of the effect of chemical functionalization of the imidazole linker on the framework dynamics. Using high-pressure (HP) single-crystal X-ray diffraction, density functional theory, and grand canonical Monte Carlo simulations, we show that in the isostructural ZIF-8, ZIF-90, and ZIF-65 functional groups of increasing polarity (-CH, -CHO, and -NO) on the imidazole linkers provide control over the degree of rotation and thus the critical window diameter. On application of pressure, the substituted imidazolate rings rotate, resulting in an increase in both pore volume and content. Our results show that the interplay between the guest molecules and the chemical function of the imidazole linker is essential for directing the swing effect in ZIF frameworks and therefore the adsorption performance.