For developing color difference formulas, there are several choices to be made on the psychophysical method used for gathering visual (observer) data. We tested three different psychophysical methods: gray scales, constant stimuli, and two-alternative forced choice (2AFC). Our results show that when using gray scales or constant stimuli, assessments of color differences are biased toward lightness differences. This bias is particularly strong in LCD monitor experiments, and also present when using physical paint samples. No such bias is found when using 2AFC. In that case, however, observer responses are affected by other factors that are not accounted for by current color difference formulas. For accurate prediction of relative color differences, our results show, in agreement with other works, that modern color difference formulas do not perform well. We also investigated if the use of digital images as presented on LCD displays is a good alternative to using physical samples. Our results indicate that there are systematic differences between these two media.