Affective (emotional) prosody is a neuropsychological function that encompasses non-verbal aspects of language that are necessary for recognizing and conveying emotions in communication, whereas non-affective (linguistic) prosody indicates whether the sentence is a question, an order or a statement. Considerable evidence points to a dominant role for the right hemisphere in both aspects of prosodic function. However, it has yet to be established whether separate parts of the right hemisphere are involved in processing different kinds of emotional intonation. The aim of this study was to answer this question. In addition, the issue of sex differences in the ability to understand prosody was considered. Fifty-two patients with damage to frontal, temporo-parietal or subcortical (basal) parts of the right hemisphere and 26 controls were tested for their ability to assess prosody information in normal (well-formed) sentences and in pseudo-sentences. General impairment of prosody processing was seen in all patient groups but the effect of damage was more apparent for emotional rather than linguistic prosody. Interestingly, appreciation of emotional prosody appeared to depend on the type of emotional expression and the location of the brain lesion. The patients with frontal damage were mostly impaired in comprehension of happy intonations; those with temporo-parietal damage in assessment of sad intonations, while subcortical lesions mostly affected comprehension of angry intonations. Differential effects of lesion location on the performance of men and women were also observed. Frontal lesions were more detrimental to women, whereas subcortical lesions led to stronger impairment in men. This suggests sex differences in brain organization of prosodic functions.