The advent of large of multi-modal imaging databases opens up the opportunity to learn how local intensity patterns covariate between multiple modalities. These models can then be used to describe expected intensities in an unseen image modalities given one or multiple observations, or to detect deviations (e.g. pathology) from the expected intensity patterns. In this work, we propose a template-based multi-modal generative mixture-model of imaging data and apply it to the problems of inlier/outlier pattern classification and image synthesis. Results on synthetic and patient data demonstrate that the proposed method is able to synthesise unseen data and accurately localise pathological regions, even in the presence of large abnormalities. It also demonstrates that the proposed model can provide accurate and uncertainty-aware intensity estimates of expected imaging patterns.