How Does the Valence of Wording Affect Features of a Scale? The Method Effects in the Undergraduate Learning Burnout Scale

Abstract

A combination of both positively and negatively worded items is often employed in a survey to reduce participants' acquiescence bias, but such a combination may hurt the validity of the survey. The current study investigated the effect of valence of wording on participants' (N = 1132) responses to four versions of the Undergraduate Learning Burnout (ULB) scale. The results showed that the valence of wording affected a number of features of the scale. The internal consistency of both the original and the original-reverse versions (consisted of both positively and negatively worded items) was lower than that of the positive-only and the negative-only versions. The original and the original-reverse versions also had more factors than the positive-only and the negative-only versions. The original and the original-reverse versions showed method effects from both the positively and the negatively worded items, and those from the negatively worded items were stronger than those from the positively worded items. The method effects were predicted by participants' subjective well-being and future academic career plans. Together, this study suggests that using a combination of positively and negatively worded items can lead to a predictable response style and significant method effects, which reduce the scale's internal consistency and change the factor structure of the scale.

Keywords: learning burnout; method effect; multitrait-multimethod model; parallel analysis; valence of wording.