In shallow lakes, different primary producers might impact the physiochemical characteristics of the sediment and the associated microbial communities. Until now, little was known about the features of sediment Archaea and their variation across different primary producer-dominated ecosystems. Lake Taihu provides a suitable study area with cyanobacteria- and macrophyte-dominated zones co-occurring in one ecosystem. The composition of the sediment archaeal community was assessed using 16S rRNA gene amplicon sequencing technology, based on which the potential variation with respect to the physiochemical characteristics of the sediment was analyzed. Euryarchaeota (30.19% of total archaeal sequences) and Bathyarchaeota (28.00%) were the two most abundant phyla, followed by Crenarchaeota (11.37%), Aigarchaeota (10.24%) and Thaumarchaeota (5.98%). The differences found in the composition of the archaeal communities between the two zones was significant (p = 0.005). Sediment from macrophyte-dominated zones had high TOC and TN content and an abundance of archaeal lineages potentially involved in the degradation of complex organic compounds, such as the order Thermoplasmatales. In the area dominated by cyanobacteria, archaeal lineages related to sulfur metabolism, for example, Sulfolobales and Desulfurococcales, were significantly enriched. Among Bathyarchaeota, subgroups MCG-6 and MCG-15 were significantly accumulated in the sediment of areas dominated by macrophytes whereas MCG-4 was consistently dominant in both type of sediments. The present study contributes to the knowledge of sediment archaeal communities with different primary producers and their possible biogeochemical functions in sediment habitats.