Looking at a photo and deciding whether the person depicted is happy, angry or sad may seem like a trivial task for anyone to do. However, differing contexts and other subtle factors make it very costly for a computer to do the same. Being able to analyze subjective information automatically is an invaluable tool for small businesses. This data can be used to shape business decisions and drive profits. One way to achieve this goal is through crowdsourcing. In other words, getting a large group of volunteers to participate in a common problem and combining their contirbutions. Actually organizing, funding, and managing a project like this can be daunting and expensive, this is where Amazon's Mechanical Turk comes in. This talk explains how Mechanical Turk works and cover various ways in which it can be leveraged by anyone. We will cover use cases that have been successful, the mechanics of posting, processing and testing tasks, and specific tools for accomplishing these goals. This talk was given by Michael Becker and Kelly O'Brien at the 2013 Philly Tech Week on April 23, 2013.

1. data processing with
mechanical turk
Kelly O'Brien @klm427; github.com/kellyob
Michael Becker @beckerfuffle; github.com/mdbecker
#ptw2013

2. Mechanical Turk
#ptw2013

3. "The Turk"
#ptw2013

4. Lets focus on the crowdsourcing...
Relatively cheap means of getting random
samples of input for small, tedious tasks
"Crowdsourced labor can cost companies less than half as much as typical outsourcing"
-- Panagiotis G. Ipeirotis, an associate professor at NYU's Stern School of Business
#ptw2013

5. "Nothing is a waste of time if you use the experience wisely."
~Auguste Rodin
#ptw2013

6. The business challenge....
#ptw2013

7. The solution....
#ptw2013

8. Let start with the basics
Template
Data Template Template
Requester's Template
Data HITs
Workers (Turkers)
#ptw2013

9. Use cases
● Classification
● Transcription
● Content Generation
● Surveys
#ptw2013

10. Do people actually use this?
#ptw2013

11. AOL
#ptw2013

12. Twitter
#ptw2013

13. CardMunch @LinkedIn
#ptw2013

14. The Sheep Market
#ptw2013

15. Development Tools
● Requester user interface
● Amazon offers four official APIs
○ Ruby, .NET, Perl, and Java
● AWS API
● Boto mturk
○ Python
● Houdini, Clockwork Raven,
Crowdflower, QuikTurKit
#ptw2013

16. Create a HIT
● A title
● A description
● Keywords, used to help Workers find the HITs with a search
● The amount of the reward
● An amount of time in which the Worker must complete the HIT
● An amount of time after which the HIT will no longer be available
to Workers
● The number of Workers needed to submit results for the HIT
before the HIT is considered complete
● Qualification requirements
● All of the information required to answer the question
#ptw2013

17. Process Results
● Assignment id
● Worker id
● HIT id
● Assignment status
● Auto approval time
● Accept time
● Submit time
● Approval time
● Rejection time
● Deadline
● Answer
● Requester feedback
#ptw2013

18. What was the question?
● Question forms
● External questions
● HTML questions
#ptw2013

19. Formatting HITs
● Compact
● Coherent
● Cost-effective
#ptw2013

20. Bad Actors
"Unfortunately, since manually verifying the quality of the submitted results is hard, malicious workers often take
advantage of the verification difficulty and submit answers of low quality." [1]
#ptw2013

21. Quality Control
● Manually spot
check
● Qualifications
● Multiple agreement
● Gold HITs
● Calculate worker
error
#ptw2013

22. Quality Control: Manually Check
Look through the results of some workers and manually
reject/ban those which look bad
#ptw2013

23. Quality Control: Multiple Agreement
1. Submit HITs to multiple turks (3-10)
2. Reject/throw out all HITs below some
agreement threshold
#ptw2013

24. Quality Control: Qualifications
● Pay extra for "superior"
turks
● Build your own custom
qualification
"Thought Masters was just bad for non-blessed workers? It's even worse for requesters [1]"
#ptw2013

25. Quality Control: Gold HITs
1. Give turks HITs which we know the correct answer to
2. Reject/Ban turks with high error rates
This technique is used by CrowdFlower
#ptw2013

26. Quality Control: Calculate Error
Calculate each worker's error rate based solely on their agreement with other
workers. Use an expectation-maximization algorithm as described by Dawid
and Skene.
Lots of math, consider using 3rd party service like Project Troia
#ptw2013

27. Auto-approval
"Quick approval is important, too. Watching that money pile up is a serious
motivator; I’ll sometimes choose a lower-paying task that approves in close to real
time over a higher-paying one that won’t pay out for several days."
-worker[1]
#ptw2013

28. Turkopticon
"Turkopticon lets you REPORT and AVOID shady
employers"
#ptw2013

29. Turkernation
"If you want to make a living on Amazon
Mechanical Turk, this is the forum for you"
#ptw2013

30. Do's and Don'ts
#ptw2013

31. What exactly do I do with this?
#ptw2013

32. A demo in python
#ptw2013

33. Requirements
#ptw2013

34. Data Details
#ptw2013

35. Question template
#ptw2013

36. Build a custom qualification
#ptw2013

37. Post HITs....
#ptw2013

38. Success.
#ptw2013

39. Let the work begin.
#ptw2013

40. To get results...
#ptw2013

41. AWeber
We're hiring.
aweber.jobs

42. ....and we have slides.
aweberopenhouse.eventbrite.com