With the increased prevalence of retinal pathologies, automating the detection and progression measurement of these pathologies is becoming more and more relevant. Color fundus photography is the leading modality for assessing retinal pathologies. Because eye fundus cameras have a limited field of view, multiple photographs are taken from each retina during an eye fundus examination. However, operators usually don't indicate which photographs are from the left retina and which ones are from the right retina. This paper presents a novel algorithm that automatically assigns each photograph to one retina and builds a composite image (or "mosaic") per retina, which is expected to push the performance of automated diagnosis forward. The algorithm starts by jointly forming two mosaics, one per retina, using a novel graph theoretic approach. Then, in order to determine which mosaic corresponds to the left retina and which one corresponds to the right retina, two retinal landmarks are detected robustly in each mosaic: the main vessel arch surrounding the macula and the optic disc. The laterality of each mosaic derives from their relative location. Experiments on 2790 manually annotated images validate the very good performance of the proposed framework even for highly pathological images.