data processing withmechanical turkKelly OBrien @klm427; github.com/kellyobMichael Becker @beckerfuffle; github.com/mdbeck...
Mechanical Turk                  #ptw2013
"The Turk"             #ptw2013
Lets focus on the crowdsourcing... Relatively cheap means of getting random  samples of input for small, tedious tasks    ...
"Nothing is a waste of time if you use the experience wisely."                                                      ~Augus...
The business challenge....                             #ptw2013
The solution....                   #ptw2013
Let start with the basics                                              Template Data          Template                    ...
Use cases● Classification● Transcription● Content Generation● Surveys                       #ptw2013
Do people actually use this?                               #ptw2013
AOL      #ptw2013
Twitter          #ptw2013
CardMunch @LinkedIn                      #ptw2013
The Sheep Market                   #ptw2013
Development Tools● Requester user interface● Amazon offers four official APIs  ○ Ruby, .NET, Perl, and Java● AWS API● Boto...
Create a HIT● A title● A description● Keywords, used to help Workers find the HITs with a search● The amount of the reward...
Process Results●   Assignment id●   Worker id●   HIT id●   Assignment status●   Auto approval time●   Accept time●   Submi...
What was the question?● Question forms● External questions● HTML questions                         #ptw2013
Formatting HITs●   Compact●   Coherent●   Cost-effective                     #ptw2013
Bad Actors  "Unfortunately, since manually verifying the quality of the submitted results is hard, malicious workers often...
Quality Control● Manually spot  check● Qualifications● Multiple agreement● Gold HITs● Calculate worker  error             ...
Quality Control: Manually CheckLook through the results of some workers and manuallyreject/ban those which look bad       ...
Quality Control: Multiple Agreement1. Submit HITs to multiple turks (3-10)2. Reject/throw out all HITs below some   agreem...
Quality Control: Qualifications  ● Pay extra for "superior"         turks  ●      Build your own custom         qualificat...
Quality Control: Gold HITs1. Give turks HITs which we know the correct answer to2. Reject/Ban turks with high error ratesT...
Quality Control: Calculate ErrorCalculate each workers error rate based solely on their agreement with otherworkers. Use a...
Auto-approval      "Quick approval is important, too. Watching that money pile up is a seriousmotivator; I’ll sometimes ch...
Turkopticon"Turkopticon lets you REPORT and AVOID shadyemployers"                                          #ptw2013
Turkernation"If you want to make a living on AmazonMechanical Turk, this is the forum for you"                            ...
Dos and Donts                  #ptw2013
What exactly do I do with this?                                  #ptw2013
A demo in python                   #ptw2013
Requirements               #ptw2013
Data Details               #ptw2013
Question template                    #ptw2013
Build a custom qualification                               #ptw2013
Post HITs....                #ptw2013
Success.           #ptw2013
Let the work begin.                      #ptw2013
To get results...                    #ptw2013
AWeber         Were hiring.                         aweber.jobs
....and we have slides.           aweberopenhouse.eventbrite.com
Data Processing with Mechanical Turk
Upcoming SlideShare
Loading in …5
×

Data Processing with Mechanical Turk

893 views

Published on

Looking at a photo and deciding whether the person depicted is happy, angry or sad may seem like a trivial task for anyone to do. However, differing contexts and other subtle factors make it very costly for a computer to do the same.

Being able to analyze subjective information automatically is an invaluable tool for small businesses. This data can be used to shape business decisions and drive profits.

One way to achieve this goal is through crowdsourcing. In other words, getting a large group of volunteers to participate in a common problem and combining their contirbutions. Actually organizing, funding, and managing a project like this can be daunting and expensive, this is where Amazon's Mechanical Turk comes in.

This talk explains how Mechanical Turk works and cover various ways in which it can be leveraged by anyone. We will cover use cases that have been successful, the mechanics of posting, processing and testing tasks, and specific tools for accomplishing these goals.

This talk was given by Michael Becker and Kelly O'Brien at the 2013 Philly Tech Week on April 23, 2013.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
893
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Data Processing with Mechanical Turk

  1. 1. data processing withmechanical turkKelly OBrien @klm427; github.com/kellyobMichael Becker @beckerfuffle; github.com/mdbecker #ptw2013
  2. 2. Mechanical Turk #ptw2013
  3. 3. "The Turk" #ptw2013
  4. 4. Lets focus on the crowdsourcing... Relatively cheap means of getting random samples of input for small, tedious tasks "Crowdsourced labor can cost companies less than half as much as typical outsourcing" -- Panagiotis G. Ipeirotis, an associate professor at NYUs Stern School of Business #ptw2013
  5. 5. "Nothing is a waste of time if you use the experience wisely." ~Auguste Rodin #ptw2013
  6. 6. The business challenge.... #ptw2013
  7. 7. The solution.... #ptw2013
  8. 8. Let start with the basics Template Data Template Template Requesters Template Data HITs Workers (Turkers) #ptw2013
  9. 9. Use cases● Classification● Transcription● Content Generation● Surveys #ptw2013
  10. 10. Do people actually use this? #ptw2013
  11. 11. AOL #ptw2013
  12. 12. Twitter #ptw2013
  13. 13. CardMunch @LinkedIn #ptw2013
  14. 14. The Sheep Market #ptw2013
  15. 15. Development Tools● Requester user interface● Amazon offers four official APIs ○ Ruby, .NET, Perl, and Java● AWS API● Boto mturk ○ Python● Houdini, Clockwork Raven, Crowdflower, QuikTurKit #ptw2013
  16. 16. Create a HIT● A title● A description● Keywords, used to help Workers find the HITs with a search● The amount of the reward● An amount of time in which the Worker must complete the HIT● An amount of time after which the HIT will no longer be available to Workers● The number of Workers needed to submit results for the HIT before the HIT is considered complete● Qualification requirements● All of the information required to answer the question #ptw2013
  17. 17. Process Results● Assignment id● Worker id● HIT id● Assignment status● Auto approval time● Accept time● Submit time● Approval time● Rejection time● Deadline● Answer● Requester feedback #ptw2013
  18. 18. What was the question?● Question forms● External questions● HTML questions #ptw2013
  19. 19. Formatting HITs● Compact● Coherent● Cost-effective #ptw2013
  20. 20. Bad Actors "Unfortunately, since manually verifying the quality of the submitted results is hard, malicious workers often take advantage of the verification difficulty and submit answers of low quality." [1] #ptw2013
  21. 21. Quality Control● Manually spot check● Qualifications● Multiple agreement● Gold HITs● Calculate worker error #ptw2013
  22. 22. Quality Control: Manually CheckLook through the results of some workers and manuallyreject/ban those which look bad #ptw2013
  23. 23. Quality Control: Multiple Agreement1. Submit HITs to multiple turks (3-10)2. Reject/throw out all HITs below some agreement threshold #ptw2013
  24. 24. Quality Control: Qualifications ● Pay extra for "superior" turks ● Build your own custom qualification"Thought Masters was just bad for non-blessed workers? Its even worse for requesters [1]" #ptw2013
  25. 25. Quality Control: Gold HITs1. Give turks HITs which we know the correct answer to2. Reject/Ban turks with high error ratesThis technique is used by CrowdFlower #ptw2013
  26. 26. Quality Control: Calculate ErrorCalculate each workers error rate based solely on their agreement with otherworkers. Use an expectation-maximization algorithm as described by Dawidand Skene.Lots of math, consider using 3rd party service like Project Troia #ptw2013
  27. 27. Auto-approval "Quick approval is important, too. Watching that money pile up is a seriousmotivator; I’ll sometimes choose a lower-paying task that approves in close to real time over a higher-paying one that won’t pay out for several days." -worker[1] #ptw2013
  28. 28. Turkopticon"Turkopticon lets you REPORT and AVOID shadyemployers" #ptw2013
  29. 29. Turkernation"If you want to make a living on AmazonMechanical Turk, this is the forum for you" #ptw2013
  30. 30. Dos and Donts #ptw2013
  31. 31. What exactly do I do with this? #ptw2013
  32. 32. A demo in python #ptw2013
  33. 33. Requirements #ptw2013
  34. 34. Data Details #ptw2013
  35. 35. Question template #ptw2013
  36. 36. Build a custom qualification #ptw2013
  37. 37. Post HITs.... #ptw2013
  38. 38. Success. #ptw2013
  39. 39. Let the work begin. #ptw2013
  40. 40. To get results... #ptw2013
  41. 41. AWeber Were hiring. aweber.jobs
  42. 42. ....and we have slides. aweberopenhouse.eventbrite.com

×