How related are the ergodic properties of the over- and underdamped Langevin equations driven by fractional Gaussian noise? We here find that for massive particles performing fractional Brownian motion (FBM) inertial effects not only destroy the stylized fact of the equivalence of the ensemble-averaged mean-squared displacement (MSD) to the time-averaged MSD (TAMSD) of overdamped or massless FBM, but also dramatically alter the values of the ergodicity-breaking parameter (EB). Our theoretical results for the behavior of EB for underdamped or massive FBM for varying particle mass m, Hurst exponent H, and trace length T are in excellent agreement with the findings of stochastic computer simulations. The current results can be of interest for the experimental community employing various single-particle-tracking techniques and aiming at assessing the degree of nonergodicity for the recorded time series (studying, e.g., the behavior of EB versus lag time). To infer FBM as a realizable model of anomalous diffusion for a set single-particle-tracking data when massive particles are being tracked, the EBs from the data should be compared to EBs of massive (rather than massless) FBM.