Matherly, Ted (2018), “A Panel For Lemons? Positivity bias, reputation systems and data quality on MTurk,” fortchoming at European Journal of Marketing. (Preprint)
The purpose of this research is to investigate how the effectiveness of systems for ensuring cooperation in online transactions are impacted by a positivity bias in the evaluation of the work that is produced. The presence of this bias can reduce the informativeness of the reputation system, and negatively impact its ability to ensure quality. Data from 1875 workers in five studies demonstrate the presence of a positivity bias in evaluations of workers on MTurk, which leaves them undifferentiated, except at the extremity of the reputation system and by status markers. This shows how design decisions in reputation systems can exacerbate the effect of naturally occurring biases in evaluations to reduce the utility of these systems. For researchers, the results suggest that requiring 99% approval rates (rather than the previously advised 95%) should be used to identify high quality workers on MTurk.