Capturing videos under the extremely dark environment is quite challenging for the extremely large and complex noise. To accurately represent the complex noise distribution, the physics-based noise modeling and learning-based blind noise modeling methods are proposed. However, these methods suffer from either the requirement of complex calibration procedure or performance degradation in practice. In this paper, we propose a semi-blind noise modeling and enhancing method, which incorporates the physics-based noise model with a learning-based Noise Analysis Module (NAM). With NAM, self-calibration of model parameters can be realized, which enables the denoising process to be adaptive to various noise distributions of either different cameras or camera settings. Besides, we develop a recurrent Spatio-Temporal Large-span Network (STLNet), constructed with a Slow-Fast Dual-branch (SFDB) architecture and an Interframe Non-local Correlation Guidance (INCG) mechanism, to fully investigate the spatio-temporal correlation in a large span. The effectiveness and superiority of the proposed method are demonstrated with extensive experiments, both qualitatively and quantitatively.