The problem of characterizing and estimating the radiometric noise of satellite high spectral resolution infrared spectrometers from Earth views is addressed in this paper. A methodology has been devised which is based on the common concept of spectral residuals (Observations-Calculations) obtained after spectral radiance inversion for atmospheric and surface parameters. An in-depth analytical assessment of the statistical covariance matrix of the spectral residuals has been performed which is based on the optimal estimation theory. It has been mathematically demonstrated that the use of spectral residuals to assess instrument noise leads to an effective estimator, which is largely independent of possible departures of the observational covariance matrix from the true covariances. Application to the Infrared Atmospheric Sounder Interferometer has been considered. It is shown that Earth-view-derived observation errors agree with blackbody in-flight calibration. The spectral residuals approach also proved to be effective in characterizing noise features due to mechanical microvibrations of the beam splitter of the IASI instrument.