Adipose tissues regulate metabolism, reproduction, and life span. The development and growth of adipose tissue are due to increases of both adipocyte cell size and cell number; the latter is mediated by adipocyte progenitors. Various markers have been used to identify either adipocyte progenitors or mature adipocytes. The fatty acid binding protein 4 (FABP4), commonly known as adipocyte protein 2 (aP2), has been extensively used as a marker for differentiated adipocytes. However, whether aP2 is expressed in adipogenic progenitors is controversial. Using Cre/LoxP-based cell lineage tracing in mice, we have identified a population of aP2-expressing progenitors in the stromal vascular fraction (SVF) of both white and brown adipose tissues. The aP2-lineage progenitors reside in the adipose stem cell niche and express adipocyte progenitor markers, including CD34, Sca1, Dlk1, and PDGFRα. When isolated and grown in culture, the aP2-expressing SVF cells proliferate and differentiate into adipocytes upon induction. Conversely, ablation of the aP2 lineage greatly reduces the adipogenic potential of SVF cells. When grafted into wild-type mice, the aP2-lineage progenitors give rise to adipose depots in recipient mice. Therefore, the expression of aP2 is not limited to mature adipocytes, but also marks a pool of undifferentiated progenitors associated with the vasculature of adipose tissues. Our finding adds to the repertoire of adipose progenitor markers and points to a new regulator of adipose plasticity.