Background subtraction has attracted enormous interest in the field of moving object detection. However, when there are complex scenarios such as illumination changes, dynamic background, and noise, the moving object area obtained by background subtraction often has holes, noise, and shadows. This paper proposes a novel background update model based on matrix factorization, which uses the temporal continuity of video content to solve the problems of holes, noise, and shadows. Moreover, in some cases, the texture consistency of the object is also a factor worth considering. The neighborhood weighed local binary pattern (NWLBP) is introduced to optimize the background update model, which is very effective for suppressing background or foreground shadow. The effectiveness of our method is confirmed by extensive experiments on public data sets and real shot video. Compared with the existing state-of-the-art moving object detection methods, the proposed methods can accurately establish the background model and locate the moving object region robustly.