Normal listeners are often surprisingly poor at processing pitch changes. The neural bases of this difficulty were explored using magnetoencephalography (MEG) by comparing participants who obtained poor thresholds on a pitch-direction task with those who obtained good thresholds. Source-space projected data revealed that during an active listening task, the poor threshold group displayed greater activity in the left auditory cortical region when determining the direction of small pitch glides, whereas there was no difference in the good threshold group. In a passive listening task, a mismatch response (MMNm) was identified for pitch-glide direction deviants, with a tendency to be smaller in the poor listeners. The results imply that the difficulties in pitch processing are already apparent during automatic sound processing, and furthermore suggest that left hemisphere auditory regions are used by these listeners to consciously determine the direction of a pitch change. This is in line with evidence that the left hemisphere has a poor frequency resolution, and implies that normal listeners may use the sub-optimal hemisphere to process pitch changes.