• Like
Analytic Methods for Optimizing Realtime Crowdsourcing
Upcoming SlideShare
Loading in...5
×

Analytic Methods for Optimizing Realtime Crowdsourcing

  • 301 views
Uploaded on

Collective Intelligence 2012 …

Collective Intelligence 2012

Realtime crowdsourcing research has demonstrated that it is possible to recruit paid crowds within seconds by managing a small, fast-reacting worker pool. Realtime crowds enable crowd-powered systems that respond at interactive speeds: for example, cameras, robots and instant opinion polls. So far, these techniques have mainly been proof-of-concept prototypes: research has not yet attempted to understand how they might work at large scale or optimize their cost/performance trade-offs. In this paper, we use queueing theory to analyze the retainer model for realtime crowdsourcing, in particular its expected wait time and cost to requesters. We provide an algorithm that allows requesters to minimize their cost subject to performance requirements. We then propose and analyze three techniques to improve performance: push notifications, shared retainer pools, and precruitment, which involves recalling retainer workers before a task actually arrives. An experimental validation finds that precruited workers begin a task 500 milliseconds after it is posted, delivering results below the one-second cognitive threshold for an end-user to stay in flow.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
301
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
7
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Analytic Methods for Optimizing Realtime Crowdsourcing Michael Bernstein, David Karger, Rob Miller, and Joel Brandt MIT CSAIL and Adobe Systems MIT HUMAN-COMPUTER INTERACTIONTuesday, May 8, 12
  • 2. Use queueing theory to understand and optimize performance of a paid, realtime crowdsourcing platform. • Relationship between crowd size and response time • Algorithm for optimizing crowd size & cost vs. response time • Improvements to the platform: 500 millisecond feedback MIT HUMAN-COMPUTER INTERACTIONTuesday, May 8, 12
  • 3. Realtime Crowds Answering visual questions for blind users [Bigham et al. 2010]Tuesday, May 8, 12
  • 4. Realtime Crowds Answering visual questions for blind users [Bigham et al. 2010] Crowd-assisted photography [Bernstein et al. 2011]Tuesday, May 8, 12
  • 5. Realtime Crowds Answering visual questions for blind users [Bigham et al. 2010] Crowd-assisted photography [Bernstein et al. 2011]Tuesday, May 8, 12
  • 6. Paid Crowdsourcing Pay small amounts of money for short tasks Amazon Mechanical Turk: Roughly five million tasks completed per year at 1-5¢ each [Ipeirotis 2010] Label an image Transcribe short audio clip Requester: Matt C. Requester: Gordon L. Reward: $0.01 Reward: $0.04Tuesday, May 8, 12
  • 7. Retainer Recruitment Workers sign up in advance ½¢ per minute to remain on call Alert when the task is ready Wait at most: 5 minutes Task: Click on the verbs He leapt the fence and dashed toward the door. in the paragraph [Bernstein et al. 2011]Tuesday, May 8, 12
  • 8. Retainer Recruitment Workers sign up in advance ½¢ per minute to remain on call Alert when the task is ready Wait at most: alert() 5 minutes Start now! OK Task: Click on the verbs He leapt the fence and dashed toward the door. in the paragraph [Bernstein et al. 2011]Tuesday, May 8, 12
  • 9. Retainer Recruitment Workers sign up in advance ½¢ per minute to remain on call Alert when the task is ready 50% of workers return in two seconds, and 75% of workers return in three seconds. [Bernstein et al. 2011]Tuesday, May 8, 12
  • 10. State of the Literature Realtime Crowds • Recruit crowds in two seconds, execute traditional tasks (e.g., votes) in five seconds • Maintain continuous control of remote interfaces • Opportunities in deployable, intelligently reactive software[Bigham et al. 2010, Bernstein et al. 2011, Lasecki et al. 2011]Tuesday, May 8, 12
  • 11. The Challenge Running Out of Retainer WorkersTuesday, May 8, 12
  • 12. The Challenge Running Out of Retainer WorkersTuesday, May 8, 12
  • 13. The Challenge Running Out of Retainer WorkersTuesday, May 8, 12
  • 14. The Challenge Running Out of Retainer WorkersTuesday, May 8, 12
  • 15. The Challenge Running Out of Retainer WorkersTuesday, May 8, 12
  • 16. The Challenge Running Out of Retainer WorkersTuesday, May 8, 12
  • 17. The Challenge Running Out of Retainer Workers Loss Non-realtime responseTuesday, May 8, 12
  • 18. The Tradeoff Missed tasks, Extra retainer workers, non-realtime results extra costTuesday, May 8, 12
  • 19. The Goal Optimize the tradeoff between recruiting too many workers and dropping too many tasks.Tuesday, May 8, 12
  • 20. The Goal Optimize the tradeoff between recruiting too many workers and dropping too many tasks. Budget-optimal crowdsourcing is possible in non-realtime scenarios [Dai, Mausam and Weld 2010; Kamar, Hacker and Horvitz 2012; Karger, Oh, and Shah 2011]Tuesday, May 8, 12
  • 21. 1 ModelOutline 2 Optimization 3 PlatformTuesday, May 8, 12
  • 22. Queueing Theory • Formal framework for stochastic arrival and service processes • Basic idea: random task arrivals and random processing times for workers • Quantify how long tasks will need to wait in line Model Outline Optimize Queueing theory for completion times: [Ipeirotis 2010] PlatformTuesday, May 8, 12
  • 23. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 24. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 25. Queueing Theory M/M/1 queue Task Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 26. Queueing Theory M/M/1 queue Task Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 27. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 28. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 29. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 30. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 31. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 32. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 33. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 34. Queueing Theory M/M/1 queue Server M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ 1 One serverTuesday, May 8, 12
  • 35. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 36. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 37. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 38. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 39. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 40. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 41. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 42. Queueing Theory M/M/c/c queue All servers busy M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 43. Queueing Theory M/M/c/c queue M Markovian (Poisson process) task arrivals, rate M Markovian (Poisson process) server work time, rate µ c c servers c c max tasks in servers and queueTuesday, May 8, 12
  • 44. Modeling Retainer RecruitmentTuesday, May 8, 12
  • 45. Retainer Queue M/M/c/c queue Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 46. Retainer Queue M/M/c/c queue Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 47. Retainer Queue M/M/c/c queue Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 48. Retainer Queue M/M/c/c queue Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 49. Retainer Queue M/M/c/c queue Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 50. Retainer Queue M/M/c/c queue Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 51. Retainer Queue Loss Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 52. Retainer Queue Loss Crowd c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 53. Retainer Queue Loss Crowd All servers busy c workers, no waiting queue Task arrivals: Poisson process, rate Worker recruitment time: Poisson process, rate µTuesday, May 8, 12
  • 54. Retainer Queue Loss Crowd All servers busyTuesday, May 8, 12
  • 55. Retainer Queue Loss Crowd All servers busyTuesday, May 8, 12
  • 56. Retainer Queue Loss P (i servers busy) = ⇡(i) Crowd P (all servers busy) = ⇡(c) All servers busyTuesday, May 8, 12
  • 57. Retainer Queue Loss P (i servers busy) = ⇡(i) Crowd P (all servers busy) = ⇡(c) All servers busyTuesday, May 8, 12
  • 58. Retainer Queue Loss P (i servers busy) = ⇡(i) Crowd P (all servers busy) = ⇡(c) All servers busyTuesday, May 8, 12
  • 59. Retainer Queue Loss P (i servers busy) = ⇡(i) Crowd P (all servers busy) = ⇡(c) All servers busyTuesday, May 8, 12
  • 60. Model Predictions 1. Probability that all workers are busy: ⇡(c) → the task has to wait for expected time 1/µ 2. Cost of keeping a retainer pool of size c → cost depends on number of idle serversTuesday, May 8, 12
  • 61. Probability of Loss • Draw on Erlang’s Loss Formula from queueing theory: probability of a rejected request in an M/M/c/c queue • Let ⇢ be the traffic intensity: ⇢ = /µ (roughly, the number of new tasks that will arrive in the time it takes to recruit a worker)Tuesday, May 8, 12
  • 62. Probability of Loss Erlang’s Loss Formula says: ⇡(c) = P (c servers busy) c ⇢ /c! = Pc i=0 ⇢i /i! Remarkably, this result makes no assumptions about the arrival distribution.Tuesday, May 8, 12
  • 63. f retainer Probability of Loss (b) Probability of waiting (c)ainer (b) Probability of waiting (c) Exp 1e+02 1e−07 1e−04 1e−01 1e+02 ● ● ● ● ● ● ● ● ● 1e−01 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−10 Expected wait time (seconds) ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● 1e−01 ● ● ● ● ● ● ● ● ● ●● 1e−01● ● ● ● ● ● ● ●● ● ● Expected wait time (seconds)● ● ● ● ● ● ● ●●● ● ● ●● Probability of waiting ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● Probability of waiting 1e−04 ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● 1e−04 ● ● 1e−04 ● ● ● ●● ● ● ● ● ● ● ● ● ● Traffic intensity ρ ● Traffic intensity ρ ● Traffic inten ● ●● ● ●● ●raffic intensity0.1 ρ Traffic intensity0.1 ●ρ Traffic intensity0ρ 1e−07 ● 1e−07 ● ● ● ● 0.5 0.5 0 1e−07 ● 0.1 0.1 0.1 1 1 ● ● ● 1 0.5 0.5 ● ● 0.5 1 5 1 5 ● 1 5 1e−10 1e−10 ● 5 10 5 10 ● ● 5 1 1e−10 10 10 ● ● 10 6 8 10 2 4 6 ● 8 10 2 8 10 2 4 6 8 ● 10 2 4ainer pool Size of retainer ● poolpool Size of retainer pool Size ● ● Tuesday, May 8, 12
  • 64. Expected Waiting Timeers⇥ ⇥ (expected P (expected recruitment time) recruitment time) (c servers busy) recruitment time) y) busy) ⇥ (expected 1 1 1 = ⇡(c) = ⇡(c) = ⇡(c) µ µ µ c c ⇢ /c! ⇢ 1 c /c! = Pc 1 ⇢ /c! 1 = Pc = Pc ⇢i /i! µ i=0 ⇢i /i! µ ⇢i /i! µ i=0 i=0 Tuesday, May 8, 12
  • 65. Expected Cost How much do we pay in steady-state? Depends on how many workers are usually waiting on retainer.Tuesday, May 8, 12
  • 66. Expected Cost Probability of i busy servers in an M/M/c/c queue is a more general version of Erlang’s Loss Formula: i ⇢ /i! ⇡(i) = Pc i=0 ⇢i /i! Derive the expected number of busy workers: E[i] = ⇢[1 ⇡(c)]Tuesday, May 8, 12
  • 67. Expected Cost Probability of i busy servers in an M/M/c/c queue is a more general version of Erlang’s Loss Formula: i ⇢ /i! ⇡(i) = Pc i=0 ⇢i /i! Derive the expected number of busy workers: E[i] = ⇢[1 ⇡(c)] Total cost is the number of idle workers: c ⇢[1 ⇡(c)]Tuesday, May 8, 12
  • 68. etainer Expected Cost (b) Probability of waiting (c) Ex (a) Cost of retainer (b) Prob 1e−04 1e−01 1e+02 ● ● ● ● ● ● ● ● ● 1e−01 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Expected wait time (seconds) ● ● ● ● ● ● ● ● ● 1e−01 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Payment units per unit time ● ● ● ● 5.0 ● Probability of waiting ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Probability of waiting ● ● ● ● ● 1e−04 ● ● ● ● ● ● ● ● ● ● ● 0.5 1.0 2.0 1e−04 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Traffic intensity ρ Traffic intensity ρ ● ● ● Traffic intensity ● ● ● ● Traffic intensity ρ ● ● Traffic intensity ρ ● 1e−07 0.1 ● ●0.1 0.1 1e−10 1e−07 ● 1e−07 ● 0.5 ● 0.5 ● ● ● 0.1 0.1 0.5 ● 0.5 0.5 1 ●1 1 ● 5 5 ● 1 15 0.1 0.2 1e−10 ● 10 ● 10 5 ● 5 10 1e−10 ● 10 10 ● 8 10 2 4 6 8 10 2 4 2 4 6 8● 10 2 4r pool Size of retainer pool Siz Size of retainer pool Size Cost goes down when c < ● ⇢, but performance suffers. Tuesday, May 8, 12
  • 69. etainer Expected Cost (b) Probability of waiting (c) Ex (a) Cost ofof retainer (a) Cost retainer (b) Probability ofof waiting (b) Probability waiting 1e+02 ● ● ● ● ● ● ● ● ● 1e−01 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Expected wait time (seconds) ● ● 1e−01 1e−01 ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−01 ● ● ● ● ● ● ● ● ● ● ● ● Payment units per unit time Payment units per unit time ● 5.0 5.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● Probability of waiting ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Probability of waiting Probability of waiting ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−04 ● ● ● ● ● ● ● 0.5 1.0 2.0 0.5 1.0 2.0 1e−04 1e−04 ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−04 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Traffic intensity ρ ● ● ● ● ● ● ● Traffic intensity ρ ρ Traffic intensity ρ Traffic intensity Traffic intensity ρ● ρ Traffic intensity ● Traffic intensity ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−07 0.1 0.1 0.1 1e−07 1e−07 ● ● 0.1 0.1 0.1 0.1 1e−07 ● ● ● ● ● ● 0.5 ● ● ● ● 0.5 0.5 0.5 ● 0.5 0.5 ● ● ● ● 0.5 1 11 1 1 1 0.1 0.2 0.1 0.2 ● ● ● ● ● 5 ● ● 55 5 ● 5 5 ● ● 5 1e−10 1e−10 1e−10 1e−10 10● ● 1010 10 1010 ● 10 ● ● ● ● ● 2 2 4 4 6 6 8 8 1010 2 2 4 4 6 6 8 8 1010 8 10 2 4 6 8 10 2 4 ● ● Size of of retainer pool Size retainer pool ● Size of of retainer pool Size retainer poolr pool Size of retainer pool Siz Cost goes down when c < ⇢, ● ● ● but performance suffers. Tuesday, May 8, 12
  • 70. Optimal Retainer Size • Size of retainer pool is typically the only value that requesters can manipulate • Minimize costs by keeping the retainer pool small while keeping ⇡(c) low Model Outline Optimize PlatformTuesday, May 8, 12
  • 71. Optimal Retainer Size Based on Maximum Miss Probability Cost vs. probability of waiting Given a maximum ● ●●● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● Probability of waiting ● desired probability ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−04 ● ● of a miss pmax : Traffic intensity ρ 0.1 ● ● ● ● ● ● ● ● 0.5 Minimize c subject 1 5 ● ● ● ● to ⇡(c)  pmax ● 1e−10 10 ● 0 2 4 6 8 ● Expected payments per unit time ●Tuesday, May 8, 12
  • 72. Optimal Retainer Size Based on Maximum Miss Probability Cost vs. probability of waiting Given a maximum ● ●●● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● Probability of waiting ● desired probability ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−04 ● ● of a miss pmax : Traffic intensity ρ 0.1 ● ● ● ● ● ● ● ● 0.5 Minimize c subject 1 5 ● ● ● ● to ⇡(c)  pmax ● 1e−10 10 ● 0 2 4 6 8 ● Expected payments per unit time ●Tuesday, May 8, 12
  • 73. Optimal Retainer Size Based on Joint Cost Combined cost of retainer and missed task Total cost of task and retainer 500 ● Cost of a missed task If the “pizza delivery” ● 1 100 10 property holds: we ● ● 100 20 ● 1000 can quantify the ● ● ● ● ● ● ● ● ● cost of loss ● 5 ● ● ● ● ● ● ● ● ● 1 2 4 6 8 10 Size of retainer poolTuesday, May 8, 12
  • 74. Improving the Retainer Model 1 Subscriptions 2 Shared Pools 3 Predictive Recruitment Model Outline Optimize PlatformTuesday, May 8, 12
  • 75. Retainer Subscriptions • Proposal: increase µ by allowing workers to subscribe to realtime tasks • Instead of posting to the global task list, the platform sends a message to subscribers • Change crowdsourcing from a pull model to a push modelTuesday, May 8, 12
  • 76. Global Retainer Pools • Sharing one global retainer pool across requesters improves performance • Intuition: Most workers are padding for unlikely runs of arrivals) Time Task 1Tuesday, May 8, 12
  • 77. Global Retainer Pools • Sharing one global retainer pool across requesters improves performance • Intuition: Most workers are padding for unlikely runs of arrivals) Time Task 1 Task 2Tuesday, May 8, 12
  • 78. Global Retainer Pools • Sharing one global retainer pool across requesters improves performance • Intuition: Most workers are padding for unlikely runs of arrivals) Time CombinedTuesday, May 8, 12
  • 79. Global Retainer Pools • Sharing one global retainer pool across requesters improves performance • Intuition: Most workers are padding for unlikely runs of arrivals) Time CombinedTuesday, May 8, 12
  • 80. Global Retainer Pools • Through approximation, individual pools: p ⇢ c ⇡(c) ⇡ p 2⇡c e (e⇢/c) c ⇢ 2⇡c e (e⇢/c) p across k requesters: c k • Shared pools p ⇢ 2⇡kc e (e⇢/c) c k ⇢ ⇡(c) ⇡ 2⇡kc e (e⇢/c) • Loss rate declines exponentially with the number of bundled retainer poolsTuesday, May 8, 12
  • 81. Global Retainer Pools • Through approximation, individual pools: p ⇢ c ⇡(c) ⇡ p 2⇡c e (e⇢/c) c ⇢ 2⇡c e (e⇢/c) p across k requesters: c k • Shared pools p ⇢ 2⇡kc e (e⇢/c) c k ⇢ ⇡(c) ⇡ 2⇡kc e (e⇢/c) • Loss rate declines exponentially with the number of bundled retainer poolsTuesday, May 8, 12
  • 82. Global Retainer Pools Cost dramatically decreases as you combine retainers: k dollars to log(k) dollarsTuesday, May 8, 12
  • 83. Global Retainer Routing • Not every worker in a global retainer pool is good at every task • If we assigned each worker to any task they could do, some tasks would starveTuesday, May 8, 12
  • 84. Global Retainer Routing • We want to maintain a buffer of workers to respond to all kinds of tasks • A linear programming technique can balance the traffic intensities across all tasksTuesday, May 8, 12
  • 85. Precruitment • Predictive Recruitment: notify workers before the task arrives • Recall workers in expectation of having a task by the time they arrive 2–3 seconds laterTuesday, May 8, 12
  • 86. Precruitment Formative Study, N=373 tasks • 3¢ for 3-minute retainer task: whack-a-mole • ‘Loading...’ screen for randomly-selected time [0, 20] seconds after worker returns • Click on randomly-placed moleTuesday, May 8, 12
  • 87. Precruitment Formative Study, N=373 tasks • 3¢ for 3-minute retainer task: whack-a-mole • ‘Loading...’ screen for randomly-selected time [0, 20] seconds after worker returns • Click on randomly-placed moleTuesday, May 8, 12
  • 88. Precruitment Results • Median time to mouse move: 0.50 seconds ● ● 10.0 ● to move mouse (sec) ● ● 5.0 ● ● ● ● ● ● ● ● ● ● ● ● ● Response time ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● • ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 5 10 15 20 Time waiting for task (sec) • Standard retainer model (start timer @ alert): median mouse move in 1.36 secondsTuesday, May 8, 12
  • 89. Discussion • Empirics: Can deployed crowdsourcing platforms support lots of realtime tasks? • Theory: Crowds as queueing systems • Reputation: median response time, overall response rateTuesday, May 8, 12
  • 90. Use queueing theory to understand and optimize performance of a paid, realtime crowdsourcing platform. • Relationship between crowd size and response time • Algorithm for optimizing crowd size vs. response time • Improvements to the platform: 500 millisecond feedback MIT HUMAN-COMPUTER INTERACTIONTuesday, May 8, 12
  • 91. Analytic Methods for Optimizing Realtime Crowdsourcing Michael Bernstein, David Karger, Rob Miller, and Joel Brandt MIT CSAIL and Adobe Systems MIT HUMAN-COMPUTER INTERACTIONTuesday, May 8, 12