Heat stress is a major abiotic stress factor reducing crop productivity and climate change models predict increasing temperatures in many production regions. Common bean (Phaseolus vulgaris L.) is an important crop for food security in the tropics and heat stress is expected to cause increasing yield losses. To study physiological responses and to characterize the genetics of heat stress tolerance, we evaluated the recombinant inbred line (RIL) population IJR (Indeterminate Jamaica Red) x AFR298 of the Andean gene pool. Heat stress (HS) conditions in the field affected many traits across the reproductive phase. High nighttime temperatures appeared to have larger effects than maximum daytime temperatures. Yield was reduced compared to non-stress conditions by 37% and 26% in 2016 and 2017 seasons, respectively. The image analysis tool HYRBEAN was developed to evaluate pollen viability (PolVia). A significant reduction of PolVia was observed in HS and higher viability was correlated with yield only under stress conditions. In susceptible lines the reproductive phase was extended and defects in the initiation of seed, seed fill and seed formation were identified reducing grain quality. Higher yields under HS were correlated with early flowering, high pollen viability and effective seed filling. Quantitative trait loci (QTL) analysis revealed a QTL for both pod harvest index and PolVia on chromosome Pv05, for which the more heat tolerant parent IJR contributed the positive allele. Also, on chromosome Pv08 a QTL from IJR improved PolVia and the yield component pods per plant. HS affected several traits during the whole reproductive development, from floral induction to grain quality traits, indicating a general heat perception affecting many reproductive processes. Identification of tolerant germplasm, indicator traits for heat tolerance and molecular tools will help to breed heat tolerant varieties to face future climate change effects.