Manifold alignment can be used to reduce the dimensionality of multiple medical image datasets into a single globally consistent low-dimensional space. This may be desirable in a wide variety of problems, from fusion of different imaging modalities for Alzheimer's disease classification to 4DMR reconstruction from 2D MR slices. Unfortunately, most existing manifold alignment techniques require either a set of prior correspondences or comparability between the datasets in high-dimensional space, which is often not possible. We propose a novel technique for the 'self-alignment' of manifolds (SAM) from multiple dissimilar imaging datasets without prior correspondences or inter-dataset image comparisons. We quantitatively evaluate the method on 4DMR reconstruction from realistic, synthetic sagittal 2D MR slices from 6 volunteers and real data from 4 volunteers. Additionally, we demonstrate the technique for the compounding of two free breathing 3D ultrasound views from one volunteer. The proposed method performs significantly better for 4DMR reconstruction than state-of-the-art image-based techniques.