We demonstrate the use of spectrograms of the field-induced second-harmonic (FISH) signal generated in ambient air, to reconstruct the absolute temporal electric field of ultra-broadband terahertz-infrared (THz-IR) pulses with bandwidths exceeding 100 THz. The approach is applicable even with relatively long (150-femtosecond) optical detection pulses, where the relative intensity and phase can be extracted from the moments of the spectrogram, as demonstrated by transmission spectroscopy of very thin samples. Auxiliary EFISH/ABCD measurements are used to provide the absolute field and phase calibration, respectively. We take into account the beam-shape/propagation effects about the detection focus on the measured FISH signals, which affect the field calibration, and show how an analysis of a set of measurements vs. truncation of the unfocused THz-IR beam can be used to correct for these. This approach could also be applied to the field calibration of ABCD measurements of conventional THz pulses.