Subjects with brainstem lesions due to either an infarct or multiple sclerosis (MS) underwent two types of binaural testing (lateralization testing and interaural discrimination) for three types of sounds (clicks and high and low frequency narrow-band noise) with two kinds of interaural differences (level and time). Two major types of abnormalities were revealed in the lateralization performances: perception of all stimuli, regardless of interaural differences (time and/or level) in the center of the head (center-oriented), or lateralization of all stimuli to one side or the other of the head (side-oriented). Similar patterns of abnormal lateralization (center-oriented and side-oriented) occurred for MS and stroke patients. A subject's pattern of abnormal lateralization testing was the same regardless of the type of stimulus or type of interaural disparity. Lateralization testing was a more sensitive test than interaural discrimination testing for both types of subjects. Magnetic resonance image (MRI) scanning in three orthogonal planes of the brainstem was used to detect lesions. A semi-automated algorithm superimposed the auditory pathway onto each MRI section. Whenever a lesion overlapped the auditory pathway, some binaural performance was abnormal and vice versa. Given a lateralization test abnormality, whether the pattern was center-oriented or side-oriented was mainly determined by lesion site. Center-oriented performance was principally associated with caudal pontine lesions and side-oriented performance with lesions rostral to the superior olivary complex. For lesions restricted to the lateral lemniscus and/or inferior colliculus, whether unilateral or bilateral, just noticeable differences (JNDs) were nearly always abnormal, but for caudal pontine lesions JNDs could be normal or abnormal. MS subjects were more sensitive to interaural time delays than interaural level differences particularly for caudal pontine lesions, while stroke patients showed no differential sensitivity to the two kinds of interaural differences. These results suggest that neural processing of binaural stimuli is multilevel and begins with independent interaural time and level analyzers in the caudal pons.