Growing evidence suggests that a broad range of behavioral anomalies in people with autism spectrum disorder (ASD) can be linked with morphological and functional alterations in the brain. However, the neuroanatomical underpinnings of ASD have been investigated using either structural magnetic resonance imaging (MRI) or diffusion tensor imaging (DTI), and the relationships between abnormalities revealed by these two modalities remain unclear. This study applied a multimodal data-fusion method, known as linked independent component analysis (ICA), to a set of structural MRI and DTI data acquired from 46 adult males with ASD and 46 matched controls in order to elucidate associations between different aspects of atypical neuroanatomy of ASD. Linked ICA identified two composite components that showed significant between-group differences, one of which was significantly correlated with age. In the other component, participants with ASD showed decreased gray matter (GM) volumes in multiple regions, including the bilateral fusiform gyri, bilateral orbitofrontal cortices, and bilateral pre- and post-central gyri. These GM changes were linked with a pattern of decreased fractional anisotropy (FA) in several white matter tracts, such as the bilateral inferior longitudinal fasciculi, bilateral inferior fronto-occipital fasciculi, and bilateral corticospinal tracts. Furthermore, unimodal analysis for DTI data revealed significant reductions of FA along with increased mean diffusivity in those tracts for ASD, providing further evidence of disrupted anatomical connectivity. Taken together, our findings suggest that, in ASD, alterations in different aspects of brain morphology may co-occur in specific brain networks, providing a comprehensive view for understanding the neuroanatomy of this disorder.