Limited depth-of-focus is a problem in many fields of optics, e.g., microscopy and macro-photography. We propose a new physically based method with a space variant point spread function (PSF) to accomplish all-in-focus reconstruction (image fusion) from a multi-focus image sequence in order to extend the depth-of-field. The proposed method works well under strong defocus conditions for color image stacks of arbitrary length. Experimental results are provided to demonstrate that our method outperforms state-of-the-art image fusion algorithms for strong defocus on both synthetic as well as real data images.