Impairment of naming function is a critical problem for temporal lobe epilepsy patients, yet the neural correlates of the disruption of temporal lobe language networks are poorly understood. Using functional MRI, we investigated the activation and task-related functional connectivity of left temporal lobe language networks and their relation to clinical naming performance and disease characteristics. We studied 59 adult patients with temporal lobe epilepsy (35 left temporal lobe epilepsy) and 32 healthy controls with auditory and visual naming functional MRI tasks. Time series of activation maxima in the left posterior inferior temporal lobe were extracted to create a psychophysiological interaction regressor for subsequent seed-based whole-brain task-related functional connectivity analyses. Correlational analyses were performed to assess the association of functional MRI activation and functional connectivity with clinical naming scores, age of onset of epilepsy, and duration of epilepsy. Auditory naming elicited activation in the left posterior inferior temporal gyrus and visual naming in the left fusiform gyrus across all groups. Activations in the left inferior temporal gyrus, left thalamus and left supplementary motor region during auditory naming as well as left fusiform activations during picture naming correlated with better clinical naming performance. Functional connectivity analyses indicated coupling of left posterior inferior temporal regions to bilateral anterior and posterior temporal lobe regions and the bilateral inferior precentral gyrus as well as contralateral occipital cortex. Stronger functional connectivity was associated with better clinical naming performance in all groups. In patients with left temporal lobe epilepsy only, functional connectivity increased with later age of onset of epilepsy and shorter disease duration. This suggests that onset of seizures early in life and prolonged disease duration lead to disrupted recruitment of temporal lobe networks ipsilateral to the seizure focus, which might account for naming deficits in temporal lobe epilepsy.