Automatic alignment estimation from projection images has a range of applications, but misaligned cameras induce inaccuracies. Calibration methods for optical cameras requiring calibration bodies or detectable features have been a matter of research for years. Not so for image guided therapy, although exact patient pose recovery is crucial. To image patient anatomy, X-ray instead of optical equipment is used. Feature detection is often infeasible. Furthermore, a method not requiring a calibration body, usable during treatment, would be desirable to improve accuracy of the patient alignment. We present a novel approach not relying on image features but combining intensity based calibration with 3D pose recovery. A stereoscopic X-ray camera model is proposed, and effects of erroneous parameters on the patient alignment are evaluated. The relevant camera parameters are automatically computed by comparison of X-ray to CT images and are incorporated in the patient alignment computation. The methods were tested with ground truth data of an anatomic phantom with artificially produced misalignments and available real-patient images from a particle therapy machine. We show that our approach can compensate patient alignment errors through mis-calibration of a camera from more than 5 mm to below 0.2 mm. Usage of images with artificial noise shows that the method is robust against image degradation of 2-5%. X-ray camera self-calibration improves accuracy when cameras are misaligned. We could show that rigid body alignment was computed more accurately and that self-calibration is possible, even if detection of corresponding image features is not.