Moiré topography is a well-established optical technique to measure the shape of three-dimensional surfaces, based on the geometric interference between an optical grid and its image deformed by an object surface. The technique produces fringes that represent contours of equal height, and from the recordings of several phase-shifted topograms surface height coordinates can be calculated. To perform these calculations, it is assumed that object height variation is small in comparison with the measurement setup dimensions, and this approximation leads to systematic errors in measurement accuracy. We present the mathematical description of the fringe formation process in projection moiré topography, and on the basis of these equations we establish the relation between setup geometry and upper limits of the systematic measurement errors. We derive the equations that determine design specifications needed to reduce the effects of approximations to be below the measurement resolution of the setup. It is shown that setup geometry should be adapted to the gray-scale measurement resolution of the imaging system. We show that, using an iterative correction from one fringe order to the next, measurement accuracy can be maintained over the entire object depth.