Understanding the adaptation mechanisms of sorghum to drought and the underlying genetic architecture may help to improve its production in a wide range of environments. By crossing a high yielding parent (HYP) and a drought tolerant parent (DTP), we obtained 140 recombinant inbred lines (RILs), which were genotyped with 120 DArT and SSR markers covering 14 linkage groups (LGs). A subset of 100 RILs was evaluated three times in control and drought treatments to genetically dissect their response to water availability. Plants with early heading date (HD) in the drought treatment maintained yield (YLD) level by reducing seed number SN and increasing hundred seed weight (HSW). In contrast, early HD in the control treatment increased SN, HSW and YLD. In total, 133 significant QTL associated with the measured traits were detected in ten hotspot regions. Antagonistic, pleiotropic effects of a QTL cluster mapped on LG-6 may explain the observed trade-offs between SN and HSW: Alleles from DTP reduced SN and the alleles from HYP increased HSW under drought stress, but not in the control treatment. Our results illustrate the importance of considering genetic and environmental factors in QTL mapping to better understand plant responses to drought and to improve breeding programs.