Heterodyne-based continuous-variable source-independent quantum random number generator (CV-SI-QRNG) can produce true random numbers without any assumptions on source. However, practical implementations always contain imperfections, which will greatly influence the extractable randomness and even open loopholes for eavesdroppers to steal information about the final output. In this work, based on the theoretical model, we systematically analyzed the effects of imperfect implementations on the practical security of heterodyne-based CV-SI-QRNG. The influences of local oscillator (LO) fluctuation under imbalanced heterodyne detection are first analyzed. The simulation results show that the lower bound of extractable randomness will be overestimated without considering the influence of LO fluctuation, which will threaten the practical security of CV-SI-QRNG system. Moreover, we analyze the effects of the degree of imbalance and the magnitude of LO fluctuation on evaluating the extractable randomness. Finally, we investigate the impact of an imperfect phase modulator on the practical security of CV-SI-QRNG and find it will reduce the extractable randomness. Our analysis reveals that one should carefully consider the imperfections in the actual implementations of CV-SI-QRNGs.