Common bean productivity is reduced by several abiotic stress factors like drought and low soil fertility, leading to yield losses particularly in low input smallholder farming systems in the tropics. To understand the genetics of stress tolerance, and to improve adaptation of common bean to adverse environments, the BAT 881 x G21212 population of 95 recombinant inbred lines (RILs) was evaluated under different abiotic stress conditions in 15 trials across four locations in Colombia, representing two higher altitude (Darién, Popayán) and two lower altitude (Palmira, Quilichao) locations. Stress vs non-stress treatments showed that yields were reduced in drought trials in Palmira by 13 and 31%, respectively, and observed yield reductions in low phosphorus stress were 39% in Quilichao, 16% in Popayán, and 71% in Darién, respectively. Yield components and biomass traits were also reduced. Traits linked to dry matter redistribution from stems, leaves and pods to seed, such as pod harvest index and total non-structural carbohydrates, were found to be important factors contributing to yield in all conditions. In contrast, early maturity was correlated with improved yield only in lower altitude locations, whereas in higher altitudes delayed maturity promoted yield. Superior RILs that combine stress tolerance and high cross-location productivity were identified. Lines that showed good yield under strong stress conditions also performed well under non-stress conditions, indicating that breeder's selection can be applied for both conditions at the same time. Quantitative trait loci (QTL) analyses revealed a stable yield QTL on chromosome Pv04, detected individually in all locations, several stress treatments and in best linear unbiased predictions (BLUPs) across all trials. Furthermore, two QTL hotspots for maturity traits were identified on Pv01 and Pv08, which are the most stable QTL. The constitutive yield QTL could serve as a good candidate for marker development and could be used in marker assisted selection. Increased understanding of the physiology of abiotic stress tolerance, combined with the availability of superior germplasm and molecular tools, will aid breeding efforts for further improvement of these plant traits.