The neurophysiological basis of variability in the latency of evoked neural responses has been of interest for decades. We describe a method to identify white matter pathways that may contribute to inter-individual variability in the timing of neural activity. We investigated the relation of the latency of peak visual responses in occipital cortex as measured by magnetoencephalography (MEG) to fractional anisotropy (FA) in the entire brain as measured with diffusion tensor imaging (DTI) in eight healthy young adults. This method makes no assumptions about the anatomy of white matter connections. Visual responses were evoked during a saccadic paradigm and were time-locked to arrival at a saccadic goal. The latency of the peak visual response was inversely related to FA in bilateral parietal and right lateral frontal white matter adjacent to cortical regions that modulate early visual responses. These relations suggest that biophysical properties of white matter affect the timing of early visual responses. This preliminary report demonstrates a non-invasive, unbiased method to relate the timing information from evoked-response experiments to the biophysical properties of white matter measured with DTI.