The random diffusivity, initially proposed to explain Brownian yet non-Gaussian diffusion, has garnered significant attention due to its capacity not only for elucidating the internal physical mechanism of non-Gaussian diffusion, but also for establishing an analytical framework to characterize particle motion in complex environments. In this paper, based on the correlation function C(t_{1},t_{2})=〈D(t_{1})D(t_{2})〉 of random diffusivity D(t), we quantitatively propose a general criterion of determining the ergodic property of the Langevin equation with the arbitrary random diffusivity D(t). Due to the critical role of correlation function C(t_{1},t_{2}), we derive the criterion for the two cases with stationary diffusivity or nonstationary diffusivity, respectively. By utilizing the quantitative criterion, we can directly judge the ergodic properties of the random diffusivity model based on the correlation function C(t_{1},t_{2}) of random diffusivity D(t). Several typical diffusivities, including the common square of the Brownian motion and of the (fractional) Ornstein-Uhlenbeck process, are found to contribute to different ergodic properties, which validates our proposed criterion built on the correlation function C(t_{1},t_{2}).