When fringe projection profilometry is used for measuring texture on rough surfaces, the measurement resolution is subject to the spatial frequency response of the instrument. The instrument transfer function (ITF) is a good metric to quantify this property. A valid ITF analysis requires the system to be linear. In this paper, we investigate the validity of using ITF to characterize the spatial resolution of a fringe projection system. Approximate linearity is shown through a mathematical analysis and simulation. We also demonstrate a practical method for measuring ITF using a stepped surface. The measured ITF is compared with an ITF prediction, which is simulated with a theoretical model.