Recent studies of multiple sclerosis (MS) and stroke patients suggested a correlation between two patterns of abnormal performance in lateralization tasks and two sites of pontine lesions. Most patients who had lesions below or at the superior olivary complex (SOC) perceived all interaural differences in binaural stimuli as small, while most patients who had lesions above the SOC perceived all interaural differences as large. The two abnormal performance patterns occurred for interaural time differences (ITD) and/or for interaural level differences (ILD). The present model proposes a multi-level hierarchical brainstem structure that estimates ITD and ILD. The first level seeks dissimilarity between the left and right inputs and a second level looks for similarity between the two sides' inputs. Each level is modeled as an ensemble of neural arrays in which each unit performs a logic or arithmetic function. The inputs are simulations of auditory nerve responses to broadband stimuli. Simulations yield good correspondence to the effect of both locations of pontine lesions on binaural performance.