Sexual dimorphism, including differences in morphology, behavior and physiology between females and males, is widespread in animals and plants and is shaped by gene expression differences between the sexes. Such expression differences may also underlie sex-specific responses of hosts to pathogen infections, most notably when pathogens induce partial sex reversal in infected hosts. The genetic changes associated with sex-specific responses to pathogen infections on the one hand, and sexual dimorphism on the other hand, remain poorly understood. The dioecious White Campion (Silene latifolia) displays sexual dimorphism in floral traits and infection with the smut fungus Micobrotryum lychnidis-dioicae induces a partial sex reversal in females. We find strong sex-specific responses to pathogen infection and reduced sexual dimorphism in infected S. latifolia. This provides a direct link between pathogen-mediated changes in sex-biased gene expression and altered sexual dimorphism in the host. Expression changes following infection affected mainly genes with male-biased expression in healthy plants. In females, these genes were up-regulated, leading to a masculinization of the transcriptome. In contrast, infection in males was associated with down-regulation of these genes, leading to a demasculinization of the transcriptome. To a lesser extent, genes with female-biased expression in healthy plants were also affected in opposite directions in the two sexes. These genes were overall down-regulated in females and up-regulated in males, causing, respectively, a defeminization in infected females and a feminization of the transcriptome in infected males. Our results reveal strong sex-specific responses to pathogen infection in a dioecious plant and provide a link between pathogen-induced changes in sex-biased gene expression and sexual dimorphism.