The reproductive endocrine systems are vastly different between males and females. This sexual dimorphism of the endocrine milieu originates from sex-specific differentiation of the somatic cells in the gonads during fetal life. Most gonadal somatic cells arise from the adrenogonadal primordium. After separation of the adrenal and gonadal primordia, the gonadal somatic cells initiate sex-specific differentiation during gonadal sex determination with the specification of the supporting cell lineages: Sertoli cells in the testis vs granulosa cells in the ovary. The supporting cell lineages then facilitate the differentiation of the steroidogenic cell lineages, Leydig cells in the testis and theca cells in the ovary. Proper differentiation of these cell types defines the somatic cell environment that is essential for germ cell development, hormone production, and establishment of the reproductive tracts. Impairment of lineage specification and function of gonadal somatic cells can lead to disorders of sexual development (DSDs) in humans. Human DSDs and processes for gonadal development have been successfully modeled using genetically modified mouse models. In this review, we focus on the fate decision processes from the initial stage of formation of the adrenogonadal primordium in the embryo to the maintenance of the somatic cell identities in the gonads when they become fully differentiated in adulthood.