The wavelength dependence of the retardation induced by a photoelastic modulator (PEM) is a central issue in multichannel modulator-based spectroscopic ellipsometry and reflectance difference spectroscopy (RDS), where the optical signal is detected simultaneously at different wavelengths. Here we present a refined analysis of the modulator crystal's retardation and its effect on the signal quality. Two retardation correction schemes that take into account the actual wavelength dependence of the stress-optic coefficient are introduced. It is demonstrated experimentally that both methods provide a better correction than the procedure currently used in multichannel RDS. We define quality factors to evaluate the actual performance of the multichannel detection system as compared with a wavelength adaptive single-channel experiment. These quality factors thus provide a useful guideline for choosing the appropriate PEM retardation or reference wavelength in a multichannel experiment.