The possibility that infantile nystagmus (IN) may reflect a failure in early sensorimotor integration has been proposed for more than a century, but is only recently being borne out in animal studies. The underlying neural and genetic substrate for this plasticity is complex. We propose that, in most cases, IN develops as a developmental response to reduced contrast sensitivity to high-spatial frequencies in an early "critical period," however caused, whether by structural malformations (e.g. foveal hypoplasia) or poor optics (e.g. cataract). As shown by psychophysics, contrast sensitivity to low spatial frequencies is enhanced by motion of the image across the retina. Based on our previous theoretical study (Harris & Berry, Nonlinear Dynamics, 2006), we argue that the best compromise between moving the image and maintaining the image near the fovea (or its remnant) is to oscillate the eyes with jerk nystagmus with increasing velocity waveforms, as seen empirically. The generation of jerk waveforms relies heavily on the saccadic system, which is immature in infancy. Pendular waveforms may therefore provide an alternative to jerk waveforms, and may explain why they are seen more often in young infants. We discuss the implications of this developmental model for the need to synchronize sensory and motor developments in normal development. Failure of this synchronization may also explain some idiopathic cases.