Selective amygdalohippocampectomy is an effective treatment for patients with therapy-refractory temporal lobe epilepsy but may cause visual field defect (VFD). Here, we aimed to describe tissue-specific pre- and postoperative imaging correlates of the VFD severity using whole-brain analyses from voxel- to network-level. Twenty-eight patients with temporal lobe epilepsy underwent pre- and postoperative MRI (T1-MPRAGE and Diffusion Tensor Imaging) as well as kinetic perimetry according to Goldmann standard. We probed for whole-brain gray matter (GM) and white matter (WM) correlates of VFD using voxel-based morphometry and tract-based spatial statistics, respectively. We furthermore reconstructed individual structural connectomes and conducted local and global network analyses. Two clusters in the bihemispheric middle temporal gyri indicated a postsurgical GM volume decrease with increasing VFD severity (FWE-corrected p < 0.05). A single WM cluster showed a fractional anisotropy decrease with increasing severity of VFD in the ipsilesional optic radiation (FWE-corrected p < 0.05). Furthermore, patients with (vs. without) VFD showed a higher number of postoperative local connectivity changes. Neither in the GM, WM, nor in network metrics we found preoperative correlates of VFD severity. Still, in an explorative analysis, an artificial neural network meta-classifier could predict the occurrence of VFD based on presurgical connectomes above chance level.