Ted Matherly (2019), European Journal of Marketing, 53,2, 195-223. (Preprint)
The purpose of this paper is to investigate how the effectiveness of systems for ensuring cooperation in online transactions is impacted by a positivity bias in the evaluation of the work that is produced. The presence of this bias can reduce the informativeness of the reputation system and negatively impact its ability to ensure quality. Using survey and experimental methods, data was collected from 1,875 Mechanical Turk (MTurk) workers in five studies designed to investigate the informativeness of the MTurk reputation system. I demonstrate the presence of a positivity bias in evaluations of workers on MTurk, which leaves them undifferentiated, except at the extremity of the reputation system and by status markers. For researchers, the results suggest that requiring 99 per cent approval rates (rather than the previously advised 95 per cent) should be used to identify high-quality workers on MTurk. The research provides insights into the design and use of reputation systems and demonstrates how design decisions can exacerbate the effect of naturally occurring biases in evaluations to reduce the utility of these systems.