Contextual relevance, which is vital for understanding conversational implicatures (CI), engages both the frontal-temporal language and theory-of-mind networks. Here we investigate how contextual relevance affects CI processing and regulates the connectivity between CI-processing-related brain regions. Participants listened to dialogues in which the level of contextual relevance to dialogue-final utterance (reply) was manipulated. This utterance was either direct, indirect but relevant, irrelevant with contextual hint, or irrelevant with no contextual hint. Results indicated that compared with direct replies, indirect replies showed increased activations in bilateral IFG, bilateral MTG, bilateral TPJ, dmPFC, and precuneus, and increased connectivity between rTPJ/dmPFC and both IFG and MTG. Moreover, irrelevant replies activated right MTG along an anterior-posterior gradient as a function of the level of irrelevance. Our study provides novel evidence concerning how the language and theory-of-mind networks interact for pragmatic inference and how the processing of CI is modulated by level of contextual relevance.