When sounds occur with temporally structured patterns, we can feel a rhythm. To memorize a rhythm, perception of its temporal patterns and organization of them into a hierarchically structured sequence are necessary. On the other hand, rhythm perception can often cause unintentional body movements. Thus, we hypothesized that rhythm information can be manifested in two different ways; temporal and motor representations. The motor representation depends on effectors, such as the finger or foot, whereas the temporal representation is effector-independent. We tested our hypothesis with a working memory paradigm to elucidate neuronal correlates of temporal or motor representation of rhythm and to reveal the neural networks associated with these representations. We measured brain activity by fMRI while participants memorized rhythms and reproduced them by tapping with the right finger, left finger, or foot, or by articulation. The right inferior frontal gyrus and the inferior parietal lobule exhibited significant effector-independent activations during encoding and retrieval of rhythm information, whereas the left inferior parietal lobule and supplementary motor area (SMA) showed effector-dependent activations during retrieval. These results suggest that temporal sequences of rhythm are probably represented in the right fronto-parietal network, whereas motor sequences of rhythm can be represented in the SMA-parietal network.