How do speakers design what they say in order to communicate effectively with groups of addressees who vary in their background knowledge of the topic at hand? Prior findings indicate that when a speaker addresses a pair of listeners with discrepant knowledge, that speakers Aim Low, designing their utterances for the least knowledgeable of the two addressees. Here, we test the hypothesis that speakers will depart from an Aim Low approach in order to efficiently communicate with larger groups of interacting partners. Further, we ask whether the cognitive demands of tracking multiple conversational partners' perspectives places limitations on successful audience design. We find that speakers can successfully track information about what up to four of their partners do and do not know in conversation. When addressing groups of 3-4 addressees at once, speakers design language based on the combined knowledge of the group. These findings point to an audience design process that simultaneously represents the perspectives of multiple other individuals and combines these representations in order to design utterances that strike a balance between the different needs of the individuals within the group.