Your SlideShare is downloading. ×
The Future of Crowd Work
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

The Future of Crowd Work


Published on

Paper at …

Paper at

Paid crowd work offers remarkable opportunities for
improving productivity, social mobility, and the global
economy by engaging a geographically distributed
workforce to complete complex tasks on demand and at
scale. But it is also possible that crowd work will fail to
achieve its potential, focusing on assembly-line piecework.
Can we foresee a future crowd workplace in which we
would want our children to participate? This paper frames
the major challenges that stand in the way of this goal.
Drawing on theory from organizational behavior and
distributed computing, as well as direct feedback from
workers, we outline a framework that will enable crowd
work that is complex, collaborative, and sustainable. The
framework lays out research challenges in twelve major
areas: workflow, task assignment, hierarchy, real-time
response, synchronous collaboration, quality control,
crowds guiding AIs, AIs guiding crowds, platforms, job
design, reputation, and motivation.

Published in: Technology

1 Comment
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. The Future of Crowd Work Aniket Kittur, CMU Elizabeth Gerber, Northwestern Matthew Lease, UT Austin Jeff Nickerson, Stevens Aaron Shaw, Northwestern John Horton, oDesk Michael Bernstein, Stanford John Zimmerman, CMUFriday, March 1, 13even  though  only  michael  and  i  are  up  here,  this  paper  was  wri6en  by  a  crowd  that  includes  jeff,  etc.and  what  were  trying  to  do  is  to  go  beyond  what  we  currently  think  of  as  crowd-­‐SOURCING  to  envision  a  bigger  future  for  crowd  work.there  has  been  a  lot  of  concern  lately  about  online  labor  and  the  risks  of  it  becoming  exploitaHve,  low  pay,  and  monotonous.
  • 2. What would it take for us to be proud of our children growing up to be crowd workers?Friday, March 1, 13aIer  becoming  a  father  a  year  and  a  half  ago  (and  many  of  you  with  children,  nieces  or  nephews  may  relate),  i  started  to  really  worry  about  what  would  the  world  look  like  for  my  daughter  ashima  when  she  grows  up.    would  i  want  her  to  be  entering  a  workforce  that  looked  like  that?so  this  paper  is  framed  around  the  quesHon:  what  would  it  take  for  us  to  be  PROUD  of  our  children  growing  up  to  be  crowd  workers?
  • 3. cognitive psych, management, computer science, design, sociology, economicsFriday, March 1, 13thats  a  pre6y  ambiHous  goal,  and  one  "I"  couldnt  answer  on  my  own,  so  we  brought  together  a  bunch  of  researchers  in  fields  ranging  from  psychology  to  ...  as  well  as  geSng  input  from  crowd  workers,  and  when  you  get  that  many  people  together...  you  end  up  wriHng  the  longest  paper  in  cscw  history.    but  what  we  also  did  is  to  try  to  envision  a  future  for  crowd  work  that  we  would  WANT  our  children  to  parHcipate  in.
  • 4. Crowd work Tasks completed online by a distributed, elastic workforce in exchange for pay.Friday, March 1, 13By crowd work, we mean any work that happens online, and is completed by a distributed,dynamically sized workforce in exchange for pay. So that means not just Mechanical Turk,but potentially any work that could be sent down a wire.
  • 5. estimated future volume [Blinder 2006, Horton 2013] $454,000,000,000 per year 91,000,000,000 hours per year 45,000,000 full-time workersFriday, March 1, 13And in fact, Blinder argues that about 20% of current American jobs could be sent down awire. These include tasks like programming, accounting, marketing, and even machineoperators. Recent evidence for crowd work in particular suggests that its volume will beroughly 454 billion dollars per year. That’s 91 billion hours per year, employing about 45million fulltime workers.What might this mean? Think of current workers having the ability to become fulltimecontractors, able to control their jobs and their career as they desire. On the other end of thespectrum, we get far more flexibility in time, like a stay-at-home dad who uses his skillswhile the baby is sleeping.
  • 6. Friday, March 1, 13However, I think many of us most strongly associate the concept with the online labormarketplace Amazon Mechanical Turk. Mechanical Turk has framed most of our discussionsabout paid crowdsourcing.
  • 7. Crowd work: PresentFriday, March 1, 13The problem with that is that Mechanical Turk, and most crowdsourcing discussions, looklike this. The work is small, simple, independent tasks and the workers are undifferentiatedand pseudonymous. Work is automatically aggregated, typically by an algorithm. Imagelabeling, part of speech tagging, and so on.
  • 8. Intellectual framing of low-cost results & exploitative labor Requesters view workers as exchangeable and untrustworthy Workers view requesters as distant and capriciousFriday, March 1, 13This intellectual framing of crowd work, both in academia and industry, has largely led togoals around low cost. We run the risk, as Six Silberman, Lilly Irani, and Ben Bederson havenoted, of falling into this framing permanently.In that framing, requesters will view workers as undifferentiated, discardable and essentiallyshirkers. Likewise, the workers will view the requesters as distant and wielding incrediblepower to deny payment or harm reputations.
  • 9. Crowd work: Future • Much real work is complex, creative, and interdependent • Writing a news story • Programming software • Composing a symphonyFriday, March 1, 13This is an incredibly dangerous framing for a number of reasons, but one major one is thatwork simply doesn’t look like that. The things we really want to help people coordinate toaccomplish are complex, they’re creative, and they’re interdependent. Think about writing,engineering software, or consider: could a crowd compose a symphony?
  • 10. Interdependencies Required expertise Drop out Low quality work Propagation of errors Unpredictability, March 1, 13Let’s say we tried to write a symphony with the crowd using the simple approach. Obviouslyeveryone generating their own stanza and putting them together wouldn’t work, though I’mtold that this is how they write Justin Bieber’s music.Early bad decisions would propagate throughout the work, people would drop out midwaythrough writing, and it would be impossible to get a global view. [switch]
  • 11. distributed distributed organizations computing, March 1, 13So  how  can  we  get  there?    Well,  we  need  some  kind  of  roadmap  or  framework  for  helping  us  think  about  where  we  need  to  go  and  what  are  the  challenges  we  need  to  overcome  to  get  there.    To  develop  this  framework  we  draw  on  ideas  from  distributed  organizaHons  on  the  one  hand  and  distributed  compuHng  on  the  other.    The  idea  is  that  both  are  facing  the  same  fundamental  challenges,First,  of  decomposing  tasks...
  • 12. WORK WORKERS Decompose task Assemble teams WORKFLOW Execute workflow OUTPUTFriday, March 1, 13the first challenge is to decompose the task:if you were planning a conference, you might split up finding a venue from reviewing papersif you were google, you might split up different parts of the web for different machines to processneed to assemble the right teams of people:if you were conference chair you need to find respectable academics or coercable friends to be on the committeeif you were google: assign different machines play different roles, like a master node coordinating a mapreduceprocessfinally, you have to execute workflows, which may have multiple stages and decision processesfor a conference we have multistage review processesin computing, we have algorithms: for example, the output of one mapreduce process may get passed to another
  • 13. Future Model of Crowd Work WORK WORKERS platform motivation task decomposition job design hierarchy reputation task assignment WORKFLOW collaboration real-time work workflow quality assurance crowds AI OUTPUTFriday, March 1, 13Synthesizing  these  we  propose  a  model  of  the  future  crowd  work  in  which  we  might  use  crowd  work  to  accomplish  complex  and  creaHve  tasks,  and  idenHfy  12  key  research  challenges  we  need  to  solve  in  order  to  get  to  a  posiHve  future  for  crowd  work.  Now  I  dont  expect  you  to  read  all  of  these;  what  well  do  in  the  rest  of  the  prez  is  to  give  you  a  brief  flavor…
  • 14. Outline • Brief flavor of three core areas: • Crowd Workers • Crowd Work • Crowd ComputationFriday, March 1, 13We’ll discuss research challenges associated with three core areas and provide some highlevel goals as grand challenges for the community. for more details, including a review of thecurrent state of the field in each of these areas and calls for action, see the paper.
  • 15. CROWD WORKERS: credentials and educationFriday, March 1, 13We’re going to start with opportunities for crowd workers. I’m going to focus on questions ofeducation and credentials.
  • 16. “I graduated from Stanford.” “I work for Google.”, March 1, 13Today there are important status markers that we use to establish our reputation. I can saythings like...
  • 17. Friday, March 1, 13But if you look at crowd work platforms, those kinds of mechanisms are totally absent. LikeNetflix and eBay, there’s a collapse toward high ratings and people signal their expertise likeon Craigslist. [examples]
  • 18. “I have 4.2 stars out of 5 as a composer!”Friday, March 1, 13The future of reputation is not going to be saying [quote]
  • 19. Reputation Goal: mechanisms as straightforward to deploy and interpret as today’s affiliations and degreesFriday, March 1, 13We need credentialing, because reputation is critical to setting up meaningful paths tosuccess.Our goal here should be to develop a reputation mechanism that’s as easy to deploy andinterpret as affiliations and degrees.
  • 20. Education“hello  world!”, March 1, 13To get to credentialing, you need to think about education. The internet has been a real boonto self-driven learning, with people engaging in massive online courses and learning-by-doing. Crowd work has the opportunity to drastically empower this process.We want to think about how to help someone like this, who’s hacking together their firstArduino prototype, to go from their “Hello World” program
  • 21. Education“hello  world!” expert, March 1, becoming an expert maker and hacker. Can we facilitate this process and deliver worksuited to the person’s expertise, all the way along that process?
  • 22. Education Goal: crowd work = education, March 1, 13crowd work drives the demand for the next wave of education, and education relies on crowdwork for learning opportunities and credentialing. The job she wants to do involves Cprogramming on an arduino. So she turns to MOOCs to learn C, and the class uses simplerwork opportunities as project assignments. It keeps pushing her with work at the edge of herabilities until she can tackle the expert requirements. Then, she can use her credentials fromthat class to take on new, higher-paying and more interesting jobs.
  • 23. CROWD WORKFriday, March 1, 13When  you  think  of  work  in  tradiHonal  organizaHons  one  of  the  first  things  you  think  of  is  hierarchy  and  management.    
  • 24. CROWD WORK hierarchyFriday, March 1, 13But  in  crowd  work  we  currently  see  very  li6le  of  this.    And  so  we  propose  the  grand  challenge  of  what  if  crowd  work  could  be  as  good  as  the  best  managers.
  • 25. Hierarchy goal: crowd work as good as the best managersFriday, March 1, 13what  if  crowd  work  could  be  as  good  as  the  best  managers.and when we think of good managers, you might think of this guy
  • 26. Tim Cook CEO, Apple $378,000,000Friday, March 1, 13Here’s tim cook, ceo of apple, who many think is one of the best managers in the world. Timgot paid $378M last year, which is probably more than most crowd workers. And he does alot of things for Apple, ranging from hiring and firing to streamlining Apple’s supply chainand improving apple’s workflow.
  • 27. John Doe CEO, Crowd $378,000,000Friday, March 1, 13The question is, could we have a crowd-CEO who would be worth as much as Tim Cook? Andwhat would it take to make that possible?
  • 28. {John, Jane, Joe, ...} Doe CEOs, Crowd $378,000,000Friday, March 1, 13If  that  doesnt  seem  that  plausible,  just  think  about  how  leadership  can  be  distributed.  Different  people  can  bring  different  leadership  skills  to  the  table,  with  some  providing  feedback,  some  direcHon,  some  rewards  and  sancHons.  Can  we  draw  on  what  we  know  from  shared  leadership  in  online  communiHes  like  wikipedia  or  open  source  soIware  to  develop  dynamic  and  distributed  leadership  teams  that  together  would  be  as  valuable  as  a  Hm  cook.
  • 29. CROWD WORK: hierarchy workflowFriday, March 1, 13
  • 30. Workflow CrowdWeaver [Kittur et al. 2012] Find-Fix-Verify [Bernstein et al. 2010] TurKit [Little et al. 2009] CrowdForge [Kittur et al. 2011]Friday, March 1, 13crowd work today looks like this. as researchers we’re trying to make more complexworkflows, but it still only looks like this.
  • 31. goal: as complex and high quality but faster and more dynamic than traditional workflows, March 1, 13This is a real workflow from how a tv broadcasting station generates their can we get to something like this?to do that we need crowdflows to be more robust, modular, and reusable
  • 32., March 1, 13ultimately, we might want to move towards shared libraries or design patterns so we canbuild up more and more complex workflows instead of every person creating workflows fromscratch
  • 33. CROWD COMPUTATION: platformsFriday, March 1, 13
  • 34. Goal: crowd platforms replace firmsFriday, March 1, 13An audacious goal here would be for crowd platforms to replace firms as the default methodfor organizing work and workers. So could you create a flash startup that only exists for anafternoon, or a team of expert crowd workers who consult for thousands of jobs over aperiod of years.
  • 35. Platforms determine workFriday, March 1, 13How the platform is designed has a huge influence on the kind of work that eventually getsdone. Given how Mechanical Turk was designed, it was almost inevitable that it would feelisolating, focus on small tasks, and lean toward low pay. But alternative platforms will createalternate outcomes.
  • 36. Platforms • Create platforms that enable new kinds of work and new kinds of careers • There are problems that only a platform can solve: e.g., reliability of long-term employment, setting and achieving wage goalsFriday, March 1, 13Graduate students can do it: think about Google
  • 37. CROWD WORKERS CROWD WORK CROWD COMPUTATIONFriday, March 1, 13In order to make this happen, we need to make advances in...
  • 38. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Friday, March 1, 13and we need to do it quickly. this picture is of my daughter when we started this paper. it’snow been a year ... and I don’t have too many of those left before she might join theworkforce. There’s a LOT to be done between now and then, certainly more than I can do,certainly more than we can do.
  • 39. WORKERS education & credentials motivation & rewards job design WORK hierarchy workflow task assignment realtime, synchronous work quality control COMPUTATION platform crowds guiding AIs AIs guiding crowdsFriday, March 1, 13and in fact in this talk we only covered a quarter of the challenges we raise in the paper, and there are many moreranging from motivation to quality control and the interaction between crowds and artificial intelligence. But wewant to turn this over to you and ask:First, what’s missing here? What should we be aiming for, but we haven’t recognized yet? What would it take toget there?Second, what can YOU do to help? For any expertise that you bring to the table, there are lots of problems herethat you can help with. We challenge you to pick one and have that impact by solving it. If you know qualitativemethods well, you can help us think about motivation. If you’re technical, you can help us build this future.To close: crowd work does not have to be simple low-cost tasks. It can be the way that work is done in the future.But we need all of us to make that future one that our children will be proud to grow up in. Thanks.