Single crystals represent a benchmark for understanding the bulk properties of halide perovskites. We have indeed studied the dielectric function of lead bromide perovskite single crystals (MAPbBr3, CsPbBr3 and for the first time FAPbBr3) by spectroscopic ellipsometry in the range of 1-5 eV while varying the temperature from 183 to 440 K. An extremely low absorption coefficient in the sub-band gap region was found, indicating the high optical quality of all three crystals. We extracted the band gap values through critical point analysis showing that Tauc-based values are systematically underestimated. The two structural phase transitions, i.e., orthorhombic-tetragonal and tetragonal-cubic, show distinct optical behaviors, with the former having a discontinuous character. The cross-correlation of optical data with DFT calculations evidences the role of octahedral tilting in tailoring the value of the band gap at a given temperature, whereas differences in the thermal expansion affect the slope of the band gap trend as a function of temperature.