The ultrafast optical Kerr effect (OKE) is widely used to investigate the structural dynamics and interactions of liquids, solutions, and solids by observing their intrinsic nonlinear temporal responses through nearly collinear four-wave mixing. Non-degenerate mixing schemes allow for background free detection and can provide information on the interplay between a material's internal degrees of freedom. Here, we show a source of temporal dynamics in the OKE signal that is not reflective of the internal degrees of freedom but arises from a group index and momentum mismatch. It is observed in two-color experiments on condensed media with sizable spectral dispersion, a common property near an optical resonance. In particular, birefringence in crystalline solids is able to entirely change the character of the OKE signal via the off-diagonal tensor elements of the nonlinear susceptibility. We develop a detailed description of the phase-mismatched ultrafast OKE and show how to extract quantitative information on the spectrally resolved birefringence and group index from time-resolved experiments in one and two dimensions.