Multi-spectral quantitative phase imaging (QPI) is an emerging imaging modality for wavelength dependent studies of several biological and industrial specimens. Simultaneous multi-spectral QPI is generally performed with color CCD cameras. Here, we present a new approach for accurately measuring the color crosstalk of 2D area detectors, without needing prior information about camera specifications. Color crosstalk is systematically studied and compared using compact interference microscopy on two different cameras commonly used in QPI, single chip CCD (1-CCD) and three chip CCD (3-CCD). The influence of color crosstalk on the fringe width and the visibility of the monochromatic constituents corresponding to three color channels of white light interferogram are studied both through simulations and experiments. It is observed that presence of color crosstalk changes the fringe width and visibility over the imaging field of view. This leads to an unwanted non-uniform background error in the multi-spectral phase imaging of the specimens. The color crosstalk of the detector is observed to be the limiting factor for phase measurement accuracy of simultaneous multi-spectral QPI systems.