In this paper, we propose a novel multi-atlas based longitudinal label fusion method with temporal sparse representation technique to segment hippocampi at all time points simultaneously. First, we use groupwise longitudinal registration to simultaneously (1) estimate a group-mean image of a subject image sequence and (2) register its all time-point images to the estimated group-mean image consistently over time. Then, by registering all atlases with the group-mean image, we can align all atlases longitudinally consistently to each time point of the subject image sequence. Finally, we propose a longitudinal label fusion method to propagate all atlas labels to the subject image sequence by simultaneously labeling a set of temporally-corresponded voxels with a temporal consistency constraint on sparse representation. Experimental results demonstrate that our proposed method can achieve more accurate and consistent hippocampus segmentation than the state-of-the-art counterpart methods.