In this paper, we propose a novel object tracking method that can work well in challenging scenarios such as appearance changes, motion blurs, and especially partial occlusions and noise. Our method applies bilateral two-dimensional principal component analysis (Bi-2DPCA) for efficient object modeling and real-time computation requirement. An incremental Bi-2DPCA learning algorithm is proposed for characterizing the appearance changes of newly tracked objects. Also, to account for noise and occlusions, a sparse structure is introduced into our Bi-2DPCA object representation model. With this sparse structure, the appearance of an object can be represented by a linear combination of basis images and an additional noise image. The noise image, which indicates the location of noise and occlusions, can be used to effectively eliminate the influence caused by noise and occlusions and lead to a robust tracker. Instead of the reconstruction error commonly used in eigen-based tracking methods, a more accurate method is adopted for the computation of observation likelihood. The method is based on the energy distribution of coefficient matrix projected by Bi-2DPCA. Experimental results on challenging image sequences demonstrate the effectiveness of the proposed tracking method.