Objective: The suitability of self-ratings and observer ratings within organisational management approaches is controversial. The aim of this study was to compare the degree of agreement between self-rated and observer-rated occupational psychosocial demands. The comparison took place within a work-activity and not worker-centred assessment, according to official policies for psychosocial risk assessment. Through simultaneous application of two versions of the same instrument, we aimed to reduce the rating bias to a minimum demonstrating the suitability of self-ratings and observer ratings in companies of all kinds.
Methods: A multimethod online assessment of 22 different work activities was conducted in Germany from October 2016 to October 2017. Workers (self-ratings) and occupational safety and health (OSH) committees (observer ratings) rated the occupational psychosocial risks of each activity with the same instrument (N = 669). The instrument measured psychosocial risk conditions at work. Reliability and agreement indices were computed.
Results: The within-group agreement (WGA; r wg,mean = .42) of the workers' self-ratings was good for each psychosocial risk and the interrater reliability (IRR) was excellent on average (ICC 2 = .77) with a medium effect size of ICC 1 = .15. The interrater agreement (IRA) between the two groups varied across the activities depending on rating group and activity composition (from ICC unjust,mean = .39 to ICC unjust,mean = .86) but was good to excellent on average (ICC unjust,mean = .71).
Conclusion: The reasonable agreement and excellent reliability in workers' self-ratings justify aggregation of item means at the group level. Furthermore, if the work activities are homogenous and the committee consists of members from different OSH specialties, observer ratings and self-ratings provide comparable results. According to this study's results, both methods are reliable assessment strategies in the context of psychosocial risk assessment. The observer rating approach is especially suitable for small-to-medium enterprises that do not have access to a large anonymous survey assessment.