Age related Macular Degeneration (AMD) is a disease of the retina associated with aging. AMD progression in patients is characterized by drusen, pigmentation changes, and geographic atrophy, which can be seen using fundus imagery. The level of AMD is characterized by standard scaling methods, which can be somewhat subjective in practice. In this work we propose a statistical image processing approach to segment drusen with the ultimate goal of characterizing the AMD progression in a data set of longitudinal images. The method characterizes retinal structures with a statistical model of the colors in the retina image. When comparing the segmentation results of the method between longitudinal images with known AMD progression and those without, the method detects progression in our longitudinal data set with an area under the receiver operating characteristics curve of 0.99.