This article introduced an fMRI (functional magnetic resonance imaging) experiment which investigated the brain mechanisms underlying perceiving conversational implicatures (CI) and how contextual relevance affects the processing of CI and the brain connectivity that supports it. The contextual relevance to dialogue-final utterance (the reply to a question) was varied such that the reply was direct, indirect but relevant, irrelevant with contextual hint, or irrelevant with no contextual hint. Relative to the direct reply, the indirect reply enhanced the activities in bilateral fronto-temoral regions, bilateral TPJ and preneuneus, and the connectivity between the right TPJ/dmPFC and both IFG and MTG. The irrelevant reply showed enhanced activities in right MTG along an anterior-posterior gradient, depending on whether the irrelevance was also accompanied with some contextual hint. These findings revealed 1) how the language and the theory-of-mind networks interplay during pragmatic inference; 2) how the brain mechanisms underlying CI processing is affected by different factors of contextual relevance.