Respiratory motion introduces uncertainties when planning and delivering radiotherapy for lung cancer patients. Cone-beam projections acquired in the treatment room could provide valuable information for building motion models, useful for gated treatment delivery or motion compensated reconstruction. We propose a method for estimating 3D+T respiratory motion from the 2D+T cone-beam projection sequence by including prior knowledge about the patient's breathing motion. Motion estimation is accomplished by maximizing the similarity of the projected view of a patient specific model to observed projections of the cone-beam sequence. This is done semi-globally, considering entire breathing cycles. Using realistic patient data, we show that the method is capable of good prediction of the internal patient motion from cone-beam data, even when confronted with interfractional changes in the breathing motion.