This document summarizes a talk on human computation and crowdsourcing from an industrial perspective. It discusses how crowdsourcing can provide large amounts of cheap labeled data through platforms like Mechanical Turk but that ensuring high quality labels requires careful task design, payments, quality control methods and addressing issues like worker experience and content. Current trends include algorithms for optimizing human-machine workflows and routing tasks between crowds based on their expertise.

1. Human computation, crowdsourcing
and social: An industrial perspective
Omar Alonso
Microsoft
12 November 2014

2. Disclaimer
The views, opinions, positions, or strategies expressed in
this talk are mine and do not necessarily reflect the
official policy or position of Microsoft.

3. Introduction
• Crowdsourcing is hot
• Lots of interest in the research community
– Articles showing good results
– Journals special issues (IR, IEEE Internet Computing, etc.)
– Workshops and tutorials (SIGIR, NACL, WSDM, WWW, CHI,
RecSys, VLDB, etc.)
– HCOMP
– CrowdConf
• Large companies leveraging crowdsourcing
• Big data
• Start-ups
• Venture capital investment

4. Crowdsourcing
• Crowdsourcing is the act of taking a
job traditionally performed by a
designated agent (usually an
employee) and outsourcing it to an
undefined, generally large group of
people in the form of an open call.
• The application of Open Source
principles to fields outside of
software.
• Most successful story: Wikipedia

7. HUMAN COMPUTATION

8. Human computation
• Not a new idea
• Computers before
computers
• You are a human
computer

9. Some definitions
• Human computation is a computation
that is performed by a human
• Human computation system is a system
that organizes human efforts to carry
out computation
• Crowdsourcing is a tool that a human
computation system can use to
distribute tasks.
Edith Law and Luis von Ahn. Human Computation.Morgan & Claypool Publishers, 2011.

10. More examples
• ESP game
• Captcha: 200M every day
• ReCaptcha: 750M to date

11. Data is king
• Massive free Web data
changed how we train
learning systems
• Crowds provide new access
to cheap & labeled big data
• But quality also matters
M. Banko and E. Brill. “Scaling to Very Very Large Corpora for Natural Language Disambiguation”, ACL 2001.
A. Halevy, P. Norvig, and F. Pereira. “The Unreasonable Effectiveness of Data”, IEEE Intelligent Systems 2009.

12. Traditional Data Collection
• Setup data collection software / harness
• Recruit participants / annotators / assessors
• Pay a flat fee for experiment or hourly wage
• Characteristics
– Slow
– Expensive
– Difficult and/or Tedious
– Sample Bias…

13. Natural Language Processing
• MTurk annotation for 5 NLP tasks
• 22K labels for US $26
• High agreement between consensus labels and
gold-standard labels
• Workers as good as experts
R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng. “Cheap and Fast But is it Good? Evaluating Non-Expert
Annotations for Natural Language Tasks”. EMNLP-2008.

14. Machine Translation
• Manual evaluation
on translation quality
is slow and expensive
• High agreement
between non-experts
and experts
• $0.10 to translate a
sentence
C. Callison-Burch. “Fast, Cheap, and Creative: Evaluating Translation Quality
Using Amazon’s Mechanical Turk”, EMNLP 2009.

15. Soylent
M. Bernstein et al. “Soylent: A Word Processor with a Crowd Inside”, UIST 2010

16. Mechanical Turk
• Amazon Mechanical Turk
(AMT, MTurk,
www.mturk.com)
• Crowdsourcing platform
• On-demand workforce
• “Artificial artificial
intelligence”: get humans to
do hard part
• Named after faux automaton
of 18th C.

17. • Multiple Channels
• Gold-based tests
• Only pay for
“trusted” judgments

18. HIT example

19. HIT example

20. {where to go on vacation}
• MTurk: 50 answers,
$1.80
• Quora: 2 answers
• Y! Answers: 2
answers
• FB: 1 answer
• Tons of results
• Read title + snippet +
URL
• Explore a few pages in
detail

21. {where to go on vacation}
Countries
Cities

22. Flip a coin
• Please flip a coin and report the results
• Two questions
1. Coin type?
2. Head or tails?
• Results
Row Labels Counts
head 57
tail 43
Grand Total 100
Row Labels Count
Dollar 56
Euro 11
Other 30
(blank) 3
Grand Total 100

23. Why is this interesting?
• Easy to prototype and test new experiments
• Cheap and fast
• No need to setup infrastructure
• Introduce experimentation early in the cycle
• For new ideas, this is very helpful

24. Caveats and clarifications
• Trust and reliability
• Wisdom of the crowd re-visit
• Adjust expectations
• Crowdsourcing is another data point for your
analysis
• Complementary to other experiments

25. Why now?
• The Web
• Use humans as processors in a distributed
system
• Address problems that computers aren’t good
• Scale
• Reach

26. Who are
the workers?
• A. Baio, November 2008. The Faces of Mechanical Turk.
• P. Ipeirotis. March 2010. The New Demographics of Mechanical Turk
• J. Ross, et al. Who are the Crowdworkers? CHI 2010.

27. Issues

28. ASSESSMENTS AND LABELS

29. Relevance assessments
Is this document relevant to the query?

31. Careful with That Axe Data, Eugene
• In the area of big data and machine learning:
– labels -> features -> predictive model -> optimization
• Labeling/experimentation perceived as boring
• Don’t rush labeling
– Human and machine
• Label quality is very important
– Don’t outsource it
– Own it end to end
– Large scale

32. More on label quality
• Data gathering is not a free lunch
• Labels for the machine != labels for humans
• Emphasis on algorithms,
models/optimizations and mining from labels
• Not so much on algorithms for ensuring high
quality labels
• Training sets

33. The importance of labels – IR context

34. INFORMATION RETRIEVAL AND
CROWDSOURCING

36. Motivating Example: Relevance Judging
• Relevance of search results is difficult to judge
– Highly subjective
– Expensive to measure
• Professional editors commonly used
• Potential benefits of crowdsourcing
– Scalability (time and cost)
– Diversity of judgments

38. Started with a joke …

39. Results for {idiot}
February 2011: 5/7 (R), 2/7 (NR)
Relevant
1. Most of the time those TV reality stars have absolutely no talent. They do whatever they
can to make a quick dollar. Most of the time the reality tv stars don not have a mind of
their own. R
2. Most are just celebrity wannabees. Many have little or no talent, they just want fame. R
3. Have you seen the knuckledraggers on reality television? They should be required to change
their names to idiot after appearing on the show. You could put numbers after the word
idiot so we can tell them apart. R
4. Although I have not followed too many of these shows, those that I have encountered have
for a great part a very common property. That property is that most of the participants
involved exhibit a shallow self-serving personality that borders on social pathological
behavior. To perform or act in such an abysmal way could only be an act of an idiot. R
5. I can see this one going both ways. A particular sort of reality star comes to mind,
though, one who was voted off Survivor because he chose not to use his immunity necklace.
Sometimes the label fits, but sometimes it might be unfair. R
Not Relevant
1. Just because someone else thinks they are an "idiot", doesn't mean that is what the word
means. I don't like to think that any one person's photo would be used to describe a
certain term. NR
2. While some reality-television stars are genuinely stupid (or cultivate an image of
stupidity), that does not mean they can or should be classified as "idiots." Some simply
act that way to increase their TV exposure and potential earnings. Other reality-television
stars are really intelligent people, and may be considered as idiots by people who don't
like them or agree with them. It is too subjective an issue to be a good result for a
search engine. NR

40. You have a new idea
• Novel IR technique
• Don’t have access to click data
• Can’t hire editors
• How to test new ideas?

41. Crowdsourcing and relevance evaluation
• Subject pool access: no need to come into the
lab
• Diversity
• Low cost
• Agile

42. Pedal to the metal
• You read the papers
• You tell your boss (or advisor) that
crowdsourcing is the way to go
• You now need to produce hundreds of
thousands of labels per month
• Easy, right?

43. Ask the right questions
• Instructions are key
• Workers are not IR experts so don’t assume
the same understanding in terms of
terminology
• Show examples
• Hire a technical writer
• Prepare to iterate

44. How not to do things
• Lot of work for a few cents
• Go here, go there, copy, enter, count …

45. UX design
• Time to apply all those usability concepts
• Need to grab attention
• Generic tips
– Experiment should be self-contained.
– Keep it short and simple.
– Be very clear with the task.
– Engage with the worker. Avoid boring stuff.
– Always ask for feedback (open-ended question) in an
input box.
• Localization

46. Payments
• How much is a HIT?
• Delicate balance
– Too little, no interest
– Too much, attract spammers
• Heuristics
– Start with something and wait to see if there is
interest or feedback (“I’ll do this for X amount”)
– Payment based on user effort. Example: $0.04 (2 cents
to answer a yes/no question, 2 cents if you provide
feedback that is not mandatory)
• Bonus

47. Managing crowds

48. Quality control
• Extremely important part of the experiment
• Approach it as “overall” quality – not just for
workers
• Bi-directional channel
– You may think the worker is doing a bad job.
– The same worker may think you are a lousy
requester.
• Test with a gold standard

49. When to assess work quality?
• Beforehand (prior to main task activity)
– How: “qualification tests” or similar mechanism
– Purpose: screening, selection, recruiting, training
• During
– How: assess labels as worker produces them
– Like random checks on a manufacturing line
– Purpose: calibrate, reward/penalize, weight
• After
– How: compute accuracy metrics post-hoc
– Purpose: filter, calibrate, weight, retain

50. How do we measure work quality?
• Compare worker’s label vs.
– Known (correct, trusted) label
– Other workers’ labels
– Model predictions of workers and labels
• Verify worker’s label
– Yourself
– Tiered approach (e.g. Find-Fix-Verify)

51. Methods for measuring agreement
• Inter-agreement level
– Agreement between judges
– Agreement between judges and the gold set
• Some statistics
– Cohen’s kappa (2 raters)
– Fleiss’ kappa (any number of raters)
– Krippendorff’s alpha
• Gray areas
– 2 workers say “relevant” and 3 say “not relevant”
– 2-tier system

52. Content quality
• People like to work on things that they like
• Content and judgments according to modern
times
– TREC data set: airport security docs are pre 9/11
• Document length
• Randomize content
• Avoid worker fatigue
– Judging 100 documents on the same subject can be
tiring, leading to decreasing quality

53. Was the task difficult?
Ask workers to rate difficulty of a search topic
50 topics; 5 workers, $0.01 per task

54. So far …
• One may say “this is all good but looks like a
ton of work”
• The original goal: data is king
• Data quality and experimental designs are
preconditions to make sure we get the right
stuff
• Don’t cut corners

55. Pause
• Crowdsourcing works
– Fast turnaround, easy to experiment, few dollars to test
– But: you have to design experiments carefully, quality,
platform limitations
• Crowdsourcing in production
– Large scale data sets
– Continuous execution
– Difficult to debug
• How do you know the experiment is working
• Goal: framework for ensuring reliability on
crowdsourcing tasks
O. Alonso, C. Marshall and M. Najork. “Crowdsourcing a subjective labeling task: A human centered framework to ensure reliable
results” http://research.microsoft.com/apps/pubs/default.aspx?id=219755.

56. Labeling tweets – an example of a task
• Is this tweet interesting?
• Subjective activity
• Not focused on specific events
• Findings
– Difficult problem, low inter-rater agreement
– Tested many designs, number of workers, platforms
(MTurk and others)
• Multiple contingent factors
– Worker performance
– Work
– Task design
O. Alonso, C. Marshall & M. Najork. “Are some tweets more interesting than others? #hardquestion. HCIR 2013.

57. Designs that include in-task CAPTCHA
• Borrowed idea from reCAPTCHA -> use of
control term
• HIDDEN
• Adapt your labeling task
• 2 more questions as control
– 1 algorithmic
– 1 semantic

58. Production example #1
Q1 (k = 0.91, alpha = 0.91)
Q2 (k = 0.771, alpha = 0.771)
Q3 (k = 0.033, alpha = 0.035)
Tweet de-branded
In-task captcha
The main
question

59. Production example #2
Q1 (k = 0.907, alpha = 0.907)
Q2 (k = 0.728, alpha = 0.728)
• Q3 Worthless (alpha = 0.033)
• Q3 Trivial (alpha = 0.043)
• Q3 Funny (alpha = -0.016)
• Q3 Makes me curious (alpha = 0.026)
• Q3 Contains useful info (alpha = 0.048)
• Q3 Important news (alpha = 0.207)
Tweet de-branded
In-task captcha
Breakdown by
categories to
get better signal

60. Once we get here
• High quality labels
• Data will be later be used for rankers, ML
models, evaluations, etc.
• Training sets
• Scalability and repeatability

61. CURRENT TRENDS

62. Algorithms
• Bandit problems; explore-exploit
• Optimizing amount of work by workers
– Humans have limited throughput
– Harder to scale than machines
• Selecting the right crowds
• Stopping rule

63. Humans in the loop
• Computation loops that mix humans and
machines
• Kind of active learning
• Double goal:
– Human checking on the machine
– Machine checking on humans
• Example: classifiers for social data

64. Routing
• Expertise detection and routing
• Social load balancing
• When to switch between machines and
humans
• CrowdSTAR
B. Nushi, O. Alonso, M. Hentschel, V. Kandylas. “CrowdSTAR: A Social Task Routing
Framework for Online Communities”, 2014. http://arxiv.org/abs/1407.6714

65. Social Task Routing
Task A? Task B?
C1 C2 Crowd Summaries
Crowd 1 (Twitter) Crowd 2 (Quora)
Routing across
crowds
Routing within a crowd

66. Question Posting – Twitter Examples

67. Conclusions
• Crowdsourcing at scale works but requires a solid
framework
• Fast turnaround, easy to experiment, few dollars to test
• But you have to design the experiments carefully
• Usability considerations
• Lots of opportunities to improve current platforms
• Three aspects that need attention: workers, work and
task design
• Labeling social data is hard

68. Conclusions – II
• Important to know your limitations and be
ready to collaborate
• Lots of different skills and expertise required
– Social/behavioral science
– Human factors
– Algorithms
– Economics
– Distributed systems
– Statistics

69. Thank you - @elunca