This paper presents a new method to reconstruct the beating heart surface based on the non-rigid structure from motion technique using preprocessed endoscopic images. First the images captured at the same phase within each heart cycle are automatically extracted from the original image sequence to reduce the dimension of the deformation subspace. Then the remaining residual non-rigid motion is restricted to lie within a low-dimensional subspace and a probabilistic model is used to recover the 3D structure and camera motion simultaneously. Outliers are removed iteratively based on the reprojection error, Missing data are also recovered with an Expectation Maximization algorithm. As a result the camera can move around the operation scene to build a 3D surface with a wide field-of-view for intra-operative procedures. The method has been evaluated with synthetic data, heart phantom data, and in vivo data from a da Vinci surgical system.