We present a novel approach to the structure-from-motion problem which combines the search for correspondences and geometric reconstruction, rather than treating these as separate steps. Through the combination of the two steps, we achieve an implicit feedback of 3D information to aid the correspondence search, and at the same time we avoid an explicit model for tracking errors. The reconstruction results are therefore optimal in case of, for example, Gaussian noise on image intensities. We also present an efficient online framework for structure-from-motion with our combined approach, thoroughly evaluate the method in experiments and compare the results to state-of-the-art methods.