Sex invariance of a six-factor, higher order model of declarative memory (two second-order factors: episodic and semantic memory; and four first-order factors: recall, recognition, fluency, and knowledge) was established for 1,796 participants (35-85 years). Metric invariance of first- and second-order factor loadings across sex was demonstrated. At the second-order level, a female advantage was observed for both episodic and semantic memory. At the first-order level, sex differences in episodic memory were apparent for both recall and recognition, whereas the differences in semantic memory were driven by a female superiority in fluency. Additional tests of sex differences in three age groups (35-50, 55-65, and 70-85 years of age) indicated that the female superiority in declarative memory diminished with advancing age. The factor-specific sex differences are discussed in relation to sex differences in hippocampal function.