Neurofibromatosis type 1 (NF1) is a common genetic condition associated with cognitive and social dysfunction as well as abnormal brain structure. The pathophysiology underlying social dysfunction in NF1 is poorly understood. Here, we investigate for the first time whether there is a broad deficit of social cognition in NF1 and explore the neural correlates for these deficits. Twenty-nine adults with NF1 and 30 controls were administered an ecologically based test of social cognition, The Awareness of Social Inference Test (TASIT), to identify deficits in emotion recognition and sarcasm detection. We employed voxel-based morphometry in a subset of NF1 patients (n = 16) and 16 additional controls to examine the neural correlates of these deficits. Results indicated that adults with NF1 were impaired in their ability to understand paradoxical sarcasm and their capacity to recognize emotion, particularly anger. TASIT performance was not associated with measures of attention, visuospatial skills or executive function. Relative to controls, gray matter (GM) volume within the right superior temporal gyrus (STG) was decreased, after controlling for total brain volume. Decreased volume in this region was significantly associated with social cognitive deficits in adults with NF1. We conclude that patients with NF1 are at high risk for a social cognitive deficit and provide evidence for a neuroanatomical basis for this deficit; GM volumetric reductions in the right STG. These findings improve our understanding of the nature of social interaction impairments in NF1 and add to the growing body of literature indicating the STG as a critical brain region for social cognition.