The Sexual Desire Inventory (SDI), developed by Spector, Carey, and Steinberg in 1996, has been widely used to assess sexual desire in men and women throughout the world. This questionnaire categorizes sexual desire in two dimensions: dyadic sexual desire and solitary sexual desire. Our study addressed the factorial structure of the SDI, an aspect that until now has been largely neglected. We recruited two samples of Spanish men and women involved in stable heterosexual relationships. The first sample consisted of 3,417 subjects (1,600 males and 1,817 females), ages 18 to 84; the second sample consisted of 677 subjects (285 males and 392 females), ages 18 to 50. The results of an exploratory factor analysis (EFA) showed that instead of two dimensions the SDI should have three: (1) partner-focused dyadic sexual desire, (2) general dyadic sexual desire for an attractive person, and (3) solitary sexual desire. A confirmatory factor analysis (CFA) supported the robustness of this new three-factor structure. No gender differences were revealed, except for dyadic sexual desire for an attractive person, for which men reported higher scores. Good validity and reliability values were obtained. Moreover, standard scores for men and women by different age groups were developed.