Occurrence of language impairment in mesial temporal lobe epilepsy (mTLE) patients is common and left mTLE patients always exhibit a primary problem with access to names. To explore different neuropsychological profiles between left and right mTLE patients, the study investigated both structural and effective functional connectivity changes within the semantic cognition network between these two groups and those from normal controls. We found that gray matter atrophy of left mTLE patients was more severe than that of right mTLE patients in the whole brain and especially within the semantic cognition network in their contralateral hemisphere. It suggested that seizure attacks were rather targeted than random for patients with hippocampal sclerosis (HS) in the dominant hemisphere. Functional connectivity analysis during resting state fMRI revealed that subregions of the anterior temporal lobe (ATL) in the left HS patients were no longer effectively connected. Further, we found that, unlike in right HS patients, increased causal linking between ipsilateral regions in the left HS epilepsy patients cannot make up for their decreased contralateral interaction. It suggested that weakened contralateral connection and disrupted effective interaction between subregions of the unitary, transmodal hub of the ATL may be the primary cause of anomia in the left HS patients.