Do Mechanical Turks Dream of Square Pie Charts?

556 views

Published on

Presenter: Robert Kosara, Caroline Ziemkiewicz
BELIV 2010 Workshop
http://www.beliv.org/beliv2010/

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
556
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Do Mechanical Turks Dream of Square Pie Charts?

  1. 1. Do Mechanical Turks Dream of Square Pie Charts? Robert Kosara and Caroline Ziemkiewicz
  2. 2. University vs. Online Studies <ul><li>University </li></ul><ul><li>Online </li></ul><ul><li>Controlled environment </li></ul><ul><li>Easy to recruit </li></ul><ul><li>Standard incentives </li></ul><ul><li>Homogenous users </li></ul><ul><li>Slow, work-intensive </li></ul><ul><li>Small samples </li></ul><ul><li>Broader demographics </li></ul><ul><li>Fast, low effort </li></ul><ul><li>Larger numbers </li></ul><ul><li>Environmental control </li></ul><ul><li>Vote flooding </li></ul><ul><li>Recruitment and advertising </li></ul>Caroline Ziemkiewicz
  3. 3. Mechanical Turk <ul><li>Amazon’s micro-payment job market </li></ul><ul><li>Addresses many issues with online studies! </li></ul><ul><ul><li>Prevents vote flooding </li></ul></ul><ul><ul><li>Manages payment automatically </li></ul></ul><ul><ul><li>Handles recruitment </li></ul></ul>Caroline Ziemkiewicz
  4. 4. Our experiences Turking studies <ul><li>We’ve done a lot of these by now! </li></ul><ul><ul><li>2 years </li></ul></ul><ul><ul><li>7 studies </li></ul></ul><ul><ul><li>Over 550 participants </li></ul></ul><ul><li>What have we learned? </li></ul>Caroline Ziemkiewicz
  5. 5. Participants <ul><li>Who are the Turkers? </li></ul><ul><ul><li>57.4% female, 42.6% male (in our studies) </li></ul></ul><ul><ul><ul><li>Contrast with student populations </li></ul></ul></ul><ul><ul><li>Mean age of 32, and a broad distribution… </li></ul></ul>Caroline Ziemkiewicz
  6. 6. Participants <ul><li>Generally faster at tasks than student populations </li></ul><ul><li>Some performance differences </li></ul><ul><li>Personality </li></ul>Caroline Ziemkiewicz
  7. 7. Designing studies <ul><li>Preventing cheaters </li></ul><ul><ul><li>Bonus system allows for accuracy/quality incentives </li></ul></ul><ul><ul><li>Building checks into data collection </li></ul></ul><ul><ul><li>Pre-testing/calibration </li></ul></ul><ul><li>Other issues </li></ul><ul><ul><li>Informed consent </li></ul></ul><ul><ul><li>Good instructions </li></ul></ul><ul><ul><li>Feedback </li></ul></ul><ul><ul><li>Audit trails </li></ul></ul>Caroline Ziemkiewicz
  8. 8. Running studies <ul><li>Participation varies over time of day… </li></ul><ul><ul><li>Highest rate during the afternoon + evening EST </li></ul></ul>Caroline Ziemkiewicz
  9. 9. Running Studies <ul><li>Timing affects speed of completion </li></ul><ul><li>Rate of participation starts high and decreases over time </li></ul><ul><ul><li>Placement on front page is crucial! </li></ul></ul><ul><li>Longer tasks get fewer participants </li></ul><ul><li>Turkers like doing studies! </li></ul>Caroline Ziemkiewicz
  10. 10. To Turk or not to Turk <ul><li>Studies that lend themselves well to Turking </li></ul><ul><ul><li>Simple interactions and data collection </li></ul></ul><ul><ul><li>Short performance time </li></ul></ul><ul><ul><li>Responses that are hard to fake </li></ul></ul><ul><li>Studies that don’t </li></ul><ul><ul><li>Accuracy of self-reported data is crucial </li></ul></ul><ul><ul><li>Complex tasks </li></ul></ul><ul><ul><li>Very long task duration </li></ul></ul>Caroline Ziemkiewicz
  11. 11. Thanks! Let’s discuss!

×