Respiratory motion management is a huge challenge in radiation therapy. Respiratory motion induces temporal anatomic changes that distort the tumor volume and its position. In this study, a markerless tumor-tracking algorithm was investigated by performing phase recognition during stereotactic body radiation therapy (SBRT) using four-dimensional cone-beam computer tomography (4D-CBCT) obtained at patient registration, and in-treatment cone-beam projection images. The data for 20 treatment sessions (five lung cancer patients) were selected for this study. Three of the patients were treated with conventional flattening filter (FF) beams, and the other two were treated with flattening filter-free (FFF) beams. Prior to treatment, 4D-CBCT was acquired to create the template projection images for 10 phases. In-treatment images were obtained at near real time during treatment. Template-based phase recognition was performed for 4D-CBCT re-projected templates using prior 4D-CBCT based phase recognition algorithm and was compared with the results generated by the Amsterdam Shroud (AS) technique. Visual verification technique was used for the verification of the phase recognition and AS technique at certain tumor-visible angles. Offline template matching analysis using the cross-correlation indicated that phase recognition performed using the prior 4D-CBCT and visual verification matched up to 97.5% in the case of FFF, and 95% in the case of FF, whereas the AS technique matched 83.5% with visual verification for FFF and 93% for FF. Markerless tumor tracking based on phase recognition using prior 4D-CBCT has been developed successfully. This is the first study that reports on the use of prior 4D-CBCT based on normalized cross-correlation technique for phase recognition.