We assessed the interplay of artificial and natural selection in rice adaptation in low-input farming systems in West Africa. Using 20 morphological traits and 176 molecular markers, 182 farmer varieties of rice (Oryza spp.) from 6 West African countries were characterized. Principal component analysis showed that the four botanical groups (Oryza sativa ssp. indica, O. sativa ssp. japonica, O. glaberrima, and interspecific farmer hybrids) exhibited different patterns of morphological diversity. Regarding O. glaberrima, morphological and molecular data were in greater conformity than for the other botanical groups. A clear difference in morphological features was observed between O. glaberrima rices from the Togo hills and those from the Upper Guinea Coast, and among O. glaberrima rices from the Upper Guinea Coast. For the other three groups such clear patterns were not observed. We argue that this is because genetic diversity is shaped by different environmental and socio-cultural selection pressures. For O. glaberrima, recent socio-cultural selection pressures seemed to restrict genetic diversity while this was not observed for the other botanical groups. We also show that O. glaberrima still plays an important role in the selection practices of farmers and resulting variety development pathways. This is particularly apparent in the case of interspecific farmer hybrids where a relationship was found between pericarp colour, panicle attitude and genetic diversity. Farmer varieties are the product of long and complex trajectories of selection governed by local human agency. In effect, rice varieties have emerged that are adapted to West African farming conditions through genotype × environment × society interactions. The diversity farmers maintain in their rice varieties is understood to be part of a risk-spreading strategy that also facilitates successful and often serendipitous variety innovations. We advocate, therefore, that farmers and farmer varieties should be more effectively involved in crop development.