While prior work has generally found that data from Amazon’s Mechanical Turk (MTurk) is of reasonable quality, researchers frequently employ reputation metrics to screen out workers who are believed to be of low quality. The purpose of this research is to investigate these reputation systems as a means for improving data quality on MTurk, and to explore the potential implications of the information – or lack thereof – this system provides. The findings indicate that MTurk’s reputation system is largely uninformative about worker quality, presenting conditions that could lead to adverse selection of workers. However, no evidence is observed for this problem, suggesting that most MTurk workers are of relatively high quality. The research further validates the use of online data collection methods, suggests that there may be little value in using existing reputation metrics as screening criteria, and provides evidence that adverse selection is unlikely to be a problem for most researchers on MTurk.

Share this Project