Strata 2012: Humans, Machines, and the Dimensions of Microwork
Upcoming SlideShare
Loading in...5
×
 

Strata 2012: Humans, Machines, and the Dimensions of Microwork

on

  • 3,823 views

Presentation from O'Reilly Strata 2012 on Big Data...

Presentation from O'Reilly Strata 2012 on Big Data

Humans, Machines, and the Dimensions of Microwork
Daniel Tunkelang (LinkedIn)
Claire Hunsaker (Samasource)

The advent of crowdsourcing has wildly expanded the ways we think of incorporating human judgments into computational workflows. Computer scientists, economists, and sociologists have explored how to effectively and efficiently distribute microwork tasks to crowds and use their work as inputs to create or improve data products. Simultaneously, crowdsourcing providers are exploring the bounds of mechanical QA flows, worker interfaces, and workforce management systems.

But what tasks should be performed by humans rather than algorithms? And what makes a set of human judgments robust? Quantity? Consensus? Quality or trustworthiness of the workers? Moreover, the robustness of judgments depends not only on the workers, but on the task design. Effective crowdsourcing is a cooperative endeavor.

In this talk, we will analyze various dimensions of microwork that characterize applications, tasks, and crowds. Drawing on our experience at companies that have pioneered the use of microwork (Samasource) and data science (LinkedIn), we will offer practical advice to help you design crowdsourcing workflows to meet your data product needs.

Statistics

Views

Total Views
3,823
Views on SlideShare
1,375
Embed Views
2,448

Actions

Likes
0
Downloads
26
Comments
0

11 Embeds 2,448

http://thenoisychannel.com 2392
http://smartdatacollective.com 20
http://recsys.acm.org 13
http://www.twylah.com 6
http://www.linkedin.com 5
http://feeds2.feedburner.com 3
http://translate.googleusercontent.com 3
http://www.newsblur.com 3
http://feeds.feedburner.com 1
http://recsys.hosting.acm.org 1
http://newsblur.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Strata 2012: Humans, Machines, and the Dimensions of Microwork Strata 2012: Humans, Machines, and the Dimensions of Microwork Presentation Transcript

  • Humans, Machines, and theDimensions of MicroworkDaniel Tunkelang, LinkedInClaire Hunsaker, Samasource Recruiting Solutions 1
  • What is LinkedIn? Identity Rolodex, Resume, Connect, find and be found Business Card LinkedIn Profile, Address Book, Search Insights Newspapers, Be great at what you do Trade Magazines, Events Homepage, LinkedIn Today, Groups Everywhere Work wherever our members work Desktop Mobile, APIs, Plug-Ins 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • How do Microworkers help Data Scientists? Data Collection Human Judgments as Training Data Evaluation 8
  • Quality Assurance 9
  • Foreign University Research Project 10
  • Stay Objective vs. 11
  • At What Price Independence? Independent judgments enable statistical reasoning. – Can increase accuracy by requiring agreement of independent workers on the same task. 12
  • At What Price Independence? Independent judgments enable statistical reasoning. – Can increase accuracy by requiring agreement of independent workers on the same task. But independent workers can’t help each other out. – No benefit from collaboration = less accurate workers. – Number of workers becomes bottleneck, and workers may be incented to create fake alter egos. 13
  • Add slide of workers helping each other out at delivery center 14
  • At What Price Independence? Independent judgments enable statistical reasoning. – Can increase accuracy by requiring agreement of independent workers on the same task. But independent workers can’t help each other out. – No benefit from collaboration = less accurate workers. – Number of workers becomes bottleneck, and workers may be incented to create fake alter egos. Market for lemons (Akerlof, 1970) 15
  • 16
  • Keep It Simple Avoid unnecessary difficulty. – Provide clear instructions with examples. – Be transparent about what you’re trying to achieve. – Check an early sample of the work closely. – Set expectations on quality and accuracy and manage to those. 17
  • Keep It Simple Avoid unnecessary difficulty. – Provide clear instructions with examples – Be transparent about what you’re trying to achieve – Check an early sample of the work closely – Set expectations on quality and accuracy and manage to those Trade-offs between task value and difficulty. – Easier to select from options than answer open-ended questions. – Even easier if there are only two options. – But open-ended questions leverage more intelligence. 18
  • Keep It Simple Avoid unnecessary difficulty. – Provide clear instructions with examples – Be transparent about what you’re trying to achieve – Check an early sample of the work closely – Set expectations on quality and accuracy and manage to those Trade-offs between task value and difficulty. – Easier to select from options than answer open-ended questions. – Even easier if there are only two options. – But open-ended questions leverage more intelligence. Watch out for systematic bias. – Even independent judges may make the same mistakes. – Especially if they use the same tools. 19
  • Take Aways Independent judgments are nice for some tasks, but not always worth the cost. Keep crowdsourcing tasks as simple as possible. Manage the trade-off between task value and difficulty. Watch out for systematic bias. 20
  • Thank You! Questions? Contact: Daniel: dtunkelang@linkedin.com Claire: sales@samasource.org We’re (both) hiring (a lot)! 21