In many multitalker listening tasks, the degradation in performance that occurs when the number of interfering talkers increases from one to two is much larger than would be predicted from the corresponding decrease in the signal-to-noise ratio (SNR). In this experiment, a variety of contextually-relevant speech maskers, contextually-irrelevant speech maskers and non-speech maskers were used to examine the impact that the characteristics of the interfering sound sources have on the magnitude of this "multimasker penalty." The results show that a significant multimasker penalty only occurred in cases where two specific conditions were met: 1) the stimulus contained at least one contextually-relevant masker that could be confused with the target; and 2) the signal-to-noise ratio of the target relative to the combined masker stimulus was less than 0 dB. Remarkably, in cases where one masker was contextually relevant, the specific characteristics of the second masker had virtually no impact on the size of the multimasker penalty. Indeed, when the results were corrected for random guessing, there was essentially no difference in performance between conditions with three contextually-relevant talkers and those with two contextually-relevant talkers and one irrelevant talker. The results of a second experiment suggest that the listeners are generally able to hear keywords spoken by all three talkers even in situations where the multimasker penalty occurs, implying that the primary cause of the penalty is a degradation in the listener's ability to use prosodic cues and voice characteristics to link together words spoken at different points in the target phrase.