Title |
Coding Psychological Constructs in Text Using Mechanical Turk: A Reliable, Accurate, and Efficient Alternative
|
---|---|
Published in |
Frontiers in Psychology, May 2016
|
DOI | 10.3389/fpsyg.2016.00741 |
Pubmed ID | |
Authors |
Jennifer Tosti-Kharas, Caryn Conley |
Abstract |
In this paper we evaluate how to effectively use the crowdsourcing service, Amazon's Mechanical Turk (MTurk), to content analyze textual data for use in psychological research. MTurk is a marketplace for discrete tasks completed by workers, typically for small amounts of money. MTurk has been used to aid psychological research in general, and content analysis in particular. In the current study, MTurk workers content analyzed personally-written textual data using coding categories previously developed and validated in psychological research. These codes were evaluated for reliability, accuracy, completion time, and cost. Results indicate that MTurk workers categorized textual data with comparable reliability and accuracy to both previously published studies and expert raters. Further, the coding tasks were performed quickly and cheaply. These data suggest that crowdsourced content analysis can help advance psychological research. |
X Demographics
Geographical breakdown
Country | Count | As % |
---|---|---|
United States | 4 | 80% |
Hong Kong | 1 | 20% |
Demographic breakdown
Type | Count | As % |
---|---|---|
Members of the public | 4 | 80% |
Science communicators (journalists, bloggers, editors) | 1 | 20% |
Mendeley readers
Geographical breakdown
Country | Count | As % |
---|---|---|
United States | 2 | 6% |
Unknown | 31 | 94% |
Demographic breakdown
Readers by professional status | Count | As % |
---|---|---|
Student > Ph. D. Student | 10 | 30% |
Researcher | 6 | 18% |
Student > Master | 5 | 15% |
Student > Bachelor | 2 | 6% |
Professor | 2 | 6% |
Other | 4 | 12% |
Unknown | 4 | 12% |
Readers by discipline | Count | As % |
---|---|---|
Psychology | 10 | 30% |
Social Sciences | 5 | 15% |
Medicine and Dentistry | 3 | 9% |
Design | 2 | 6% |
Neuroscience | 2 | 6% |
Other | 5 | 15% |
Unknown | 6 | 18% |