The present study investigated when emotion modulates working memory from the perspective of neural activation. Using fMRI, we measured brain activity during the encoding and retrieval phases of a reading span test (RST) that used emotional contexts. The emotional RST required participants to read sentences that elicited negative, neural or positive emotional states while they were memorizing target words from the sentences. Compared with the neutral RST, the negative RST activated the right amygdala during the reading phase. Significant activation was also found in the parahippocampal gyrus, albeit only after activation of the amygdala became comparable to that in the neutral RST. In contrast, the positive RST activated the substantia nigra during the reading phase relative to the neutral RST. These findings suggest that negative and positive emotions modulate working memory through distinctive neural circuits. We also discuss possible relationships between emotional modulation and working memory capacity.