While technology continues to improve, there are still many things that human beings can do much more effectively than computers, such as performing data deduplication or content moderation. Traditionally, such tasks have been accomplished by hiring a large temporary workforce—which is time consuming, expensive, and difficult to scale—or have gone undone. However, businesses or developers can use Amazon Mechanical Turk (Mechanical Turk) to access thousands of on-demand workers—and then integrate the results of that work directly into their business processes and systems. In this session, learn how enterprises are using Mechanical Turk to scale and automate their human-powered workflow.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Business Process Automation
Using Crowdsourcing
Mark Chien
GM of Requester
Amazon Mechanical Turk
A I M 3 5 2

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introduction to MTurk for business process
automation
Business process automation customers on MTurk
Guest speakers
Using MTurk for business process automation

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is crowdsourcing?
Harnessing the collective power of many individuals

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is Amazon Mechanical Turk?
Amazon Mechanical Turk (MTurk) operates an online crowdsourcing
marketplace for work that requires human intelligence

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Microwork: Typically small, repeatable tasks that give workers
variety and flexibility
Each item
can be sent
to a worker
Or multiple
workers to increase
accuracy
Or through
multiple steps with
different workers
What kind of work is on MTurk?

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Mechanical Turk customers

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is business process automation?
“Business process automation (BPA) is the technology-
enabled automation of complex business processes”
- Wikipedia

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Choosing the right business processes with MTurk
• Processes that can be broken down into smaller, simpler tasks
• Tasks require minimal context and can handle high turnover
• Handle non-confidential/critical data

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common business process automation use cases on
MTurk
• Data collection/gathering
• Content moderation
• Image/Text/Audio/Video transcription
• Categorization
• Data validation

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Advantages of MTurk for business process automation
Cost savings
• Avoid the overhead and fixed costs associated with hiring and managing a
temp workforce
• Free up time and resources from your team
Elastic scale
• Use as much or as little as you need based on your demand
• There are no contracts or minimum required spend
• 24/7 global workforce to complete your work whenever needed
Diversity
• Access to 500,000 workers across 190 countries, across various demographics

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Finding opportunities to use MTurk
• Large teams of temporary or contract workers
• Processes that have already been outsourced
• Teams doing repetitive, low-context tasks
• Teams challenged with handling large deviations in volume of work
• Need for 24x7 manual processing

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content moderation and classification
Company: Online travel marketplace
Problem: Need help to moderate and categorize hotel
images uploaded by users
Approach: User-generated images are sent to MTurk in
real-time for review
MTurk task: Ensure images meet guidelines and
categorize them
Result: Effectively moderate and categorize these
photos before they are posted publicly, so that the most
relevant photos are surfaced on the hotel listings

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Validating sales accounts
Company: Online job search portal
Problem: Keeping information about customers
up to date in Salesforce
Approach: Extract customer websites from
Salesforce records and submit to MTurk
MTurk task: Confirm that the address is still valid,
find out if the company has been acquired, collect
updated data points about the company
Result: Accurate sales data and more-efficient sales
teams

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Validating catalog information
Company: Consumer travel information service
Problem: Need to confirm that recommended
restaurants are still open
Approach: Have MTurk workers visit restaurant
websites to confirm hours of operation
MTurk task: Visit the website and confirm that
it is still open
Result: Users get most up-to-date information
about restaurants on their travels

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Assessing damage after disasters
Company: Disaster assessment
Problem: Need to get building damage
assessments completed quickly to take action
Approach: Have MTurk workers review photos of
the damage
MTurk task: Assess and categorize the damage
Result: Faster response times and more-accurate
decisions

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Food Genius / US Foods Data Science
David Falck
Executive Director
dave@getfoodgenius.com

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
US Foods + Food Genius
• Chicago-based start-up, acquired by US Foods
in 2016 (2nd largest food distribution
company in the US)
• Support sellers’ conversations with customers
in the field with data
• Provide valuable, actionable
recommendations based on accurate and
fresh restaurant data
• Goal: Make positive impact in our sellers’ day-
to-day routines (and their wallets)

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Background
• Small, independent team that needed to scale quickly
• Familiarity with crowdsourcing – including code development and data
science projects
• Experimented with MTurk for research
• Early wins spurred continued investment in MTurk
• Shared case studies and knowledge with rest of company
• Expanding MTurk use cases to other US Foods business units

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Mechanical Turk?
• We use Mechanical Turk to find up-to-date, real data to help our customers
• Once we have enough external data, coupling it with our internal data makes it MUCH
more valuable…

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mechanical Turk workflows
Example 1: Find the “real” website and menus of a restaurant
(not delivery, Facebook menus)

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mechanical Turk workflows
Example 2: Multi-step information extraction from images
OCR

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Initial process

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key Learning #1
Feedback from Workers:
• We didn’t take into consideration fair payment
• We didn’t make our HITs’ instructions clear enough
Solution:
• Remember, even though you can access their services via APIs,
Workers are real people!
• Make the instructions super simple, even fun, if you can.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key Learning #2
Feedback from Workers:
• Manual verification took longer than expected, QA of 4,500 URLs took 6 days
• Coverage was low too – only about 30%
Solution:
• Build a cadre of Qualified Workers and use them!
• Use Workers to help with QA work
• Provide examples, better list of prohibited domains, empty answers will be
rejected
• QA the work closely and reject work that doesn’t meet our criteria

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results
• Our manual work went from 100% to 50% to 25% as we
made refinements to our process.
• Coverage went from 30%-63% for homepage URLs
• Workers are happier – many reach out and inquire when
we’re doing a batch of work next
• Modifying the original process lets us pivot to other asks
quickly and efficiently, such as collecting reviews, social
media data, etc. about restaurants

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary
Make crowdsourcing work for your business – You can make the
process efficient, cost-effective and scalable by putting yourself in
the shoes of a Worker and continuously improving your processes for
them
Remember:
• Listen to the Workers
• Develop effective tasks
• Measure and enhance your processes

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Indeed
Andy Hilal
Product Manager
Indeed Job Spotter

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Job Spotter: Getting local jobs onto Indeed
Using MTurk for content moderation
Results and impact
Making your MTurk program work

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OCR and photo analysis
“CioSO
eli
cioso Super Rico Pollo
PERUVIAN STYLE
CHARCOAL
CHICKEN
GRAND
ESPECIAL Pollo entero
PENING$99
con
SOPA DE
THUR
CARNE ASADA
PUPUSAS
LOMO SALTADO
TACOS
POLLO FRITO
N
CHARCOAL CHICKEN”

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine learning (with humans teaching)
What is the business being shown?
Is it a hiring sign?
• Indeed keeps high standards for job content
• No business opportunities
• No training programs
• No volunteer positions
• No elected offices
• No sexist language
• … and many more

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User-generated content risks
Low quality
• Low lighting, blurry
Objectionable content
• Faces of passersby on the street
• Worse
Fraud
• Photos of newspaper want ads
• Falsified or duplicated job sign submissions

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
From local vendor to MTurk
Nearshore Vendor Amazon MTurk
Speed of Completion 2841 minutes 34 minutes
Quality 9.73% variance 9.93% variance
Cost 50 cents / task 7 cents / task

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our biggest wins
Assume goodwill
Build clear, fast tools
Provide contextual help content
Low HIT expiry time
Golden questions and clear warnings
Price HITs correctly
Qualify a closed set of Workers
Talk to your Workers :)

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MTurk concepts primer

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Steps for using MTurk for business process automation

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Define workflow
• Break complex tasks down to simple, discrete tasks
• Determine how you want the task flow to work

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Design tasks
Use HTML/JS/CSS to design your task
Provide clear instructions and assume no context
Complete the task yourself:
• To validate instructions
• Determine average completion time to set appropriate reward
• To confirm that the task UI works
Use the sandbox to test

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Set workforce
Use Qualifications to determine who gets to do your tasks
MTurk system Qualifications
• Experience-based (# of approved HITs, approval rate)
• Locale (US, states)
• Demographics (adult, gender, degree)
Define custom Qualifications
• Inclusion Qualifications (such as whitelist)
• Exclusion Qualifications (such as blacklist)

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Load tasks
Use the API to load dataset programmaticallyUse the UI to load dataset via CSV
Or

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Review results
Use the API to download results and SNS
notifications when tasks are completed
Use the UI to view progress and
download results via CSV
Or

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Refine process
• Analyze results to assess for quality
• Review feedback provided by workers
• Update instructions and/or task design
• Update qualification membership or requirements
• Update reward for desired quality and throughput

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Get started at www.mturk.com

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related Sessions
Wednesday, Nov 28
Session | AIM351 – Harness the Power of Crowdsourcing with Amazon
Mechanical Turk
12:15 – 1:15 p.m. | Aria East, Plaza Level, Orovada 2
Wednesday, Nov 28
Workshop | AIM362 – Crowdsourcing Data Collection with
Amazon Mechanical Turk
3:15 – 5:30 p.m. | Venetian, Level 4, Lando 4205
Wednesday, Nov 28
Chalk Talk | AIM358 - Human-in-the-Loop for Machine Learning
7 – 8 p.m. | Aria West, Level 3, Starvine 7

51. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mark Chien
mchien@amazon.com

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.