Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing

317 views

Published on

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing
Ting-Hao K. Huang, Jeffrey P. Bigham.
In Proceedings of The fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017), 2017, Quebec City, Canada.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing

  1. 1. A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Live Feedback & Questions: http://tinyurl.com/20171026IGNITION Ting-Hao (Kenneth) Huang Jeffrey P. Bigham Carnegie Mellon University Image: https://pixabay.com/photo-743920/
  2. 2. 2 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Chorus: A Crowd-powered Conversation Assistant "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang et. al., HCOMP 2016
  3. 3. 3 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Constraints of Deploying Chorus • Small user traffic • Avg. 2.97 conversations per day • Fast worker response time • Response time to a message ~= 32 sec (student participant)* • Response time to a message ~= 105 sec (startup participant)* • Need multiple workers to work together • Limited budget • 24-7 service * IM waiting: timing and responsiveness in semi-synchronous communication. Avrahami et al. CSCW’08. …Similar to many academic projects !
  4. 4. 4 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] How to recruit workers quickly on-demand? On-demand (QuikTurkit, Bigham, et al., ‘10) On-call (Retainer Model, Bernstein, et al.,‘11) Ignition Method Post HITs + SEO Retainer Hybrid Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds How to Hold Workers Using old tasks Waiting Page Waiting Page Recruiting Cost Per Task Old tasks + SEO (mturk fee) All Workers’ Waiting Time 55% of Workers’ Waiting Time Recruiting Cost For Small Deployments Very Low ( k* #tasks ) High ( k* duration ) Low ( k* #tasks +) …Can we have something inexpensive and fast?
  5. 5. 5 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Engines Generate Power On-demand (QuikTurkit, Bigham, et al., ‘10) On-call (Retainer Model, Bernstein, et al.,‘11) Ignition Method Post HITs + SEO Retainer Hybrid Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds How to Hold Workers Using old tasks Waiting Page Waiting Page Recruiting Cost Per Task Old tasks + SEO (mturk fee) All Workers’ Waiting Time 55% of Workers’ Waiting Time Recruiting Cost For Small Deployments Very Low ( k* #tasks ) High ( k* duration ) Low ( k* #tasks +) • Crowd-powered System Why not both? Re-Start the Engine Each Time Start the Engine Once, and Hold the Power Image: https://commons.wikimedia.org/wiki/File:3791_cc_Holden_3800_V6_engine_(2015-08-29)_03.jpg
  6. 6. 6 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Ignition Model 1. Hire slightly more workers than we need 2. Hold the extra workers in the retainer 3. Always call workers from retainer first, and then post HITs
  7. 7. 7 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Ignition = Retainer + Market • When you want to recruit 8 workers each time… • A Chorus task need at most 5 workers #Workers Recruited from Retainer #Workers Recruited from Mturk Market ( Posting a HIT w/ Cmarket Assignments ) Posting More Assignments -> Gets Workers Faster Cretainer + Cmarket = 8
  8. 8. 8 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Waiting Page & Point System http://TinyUrl.com/IgnitionDemo
  9. 9. 9 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Transition Graph for Workers in Ignition
  10. 10. 10 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Transition Graph for Workers in Ignition
  11. 11. 11 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Transition Graph for Workers in Ignition
  12. 12. 12 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] 1. The task occurs instantly [ Instant ] 2. The task occurs later [ Retained ] 3. No tasks at all [ No Task ] Transition Graph for Workers in Ignition
  13. 13. 13 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Ignition Deployment • From July 1st, 2016 to April 27th, 2017 (~10 months) • 6,823 assignments, 648 workers • $0.25 for a 30-minute waiting time • Mturk Qualifications Used • HIT Approval Rate > 90%, #Approved HITs> 200, Adult Content Qualification Assignments Tasks (Conversation) #workers in retainer 50% of tasks occurred when 1+ workers in the retainer
  14. 14. 14 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Posting More Assignments -> Gets Workers Faster Simply Posting HITs Getting 1+ Workers Getting 3+ Workers X-Axis: Time (x seconds) Y-Axis: Prob(Getting N+ workers within x seconds) Cmarket = 3 Cmarket = 8 Cmarket = 8 Cmarket = 3
  15. 15. 15 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Simply Posting HITs v.s. Ignition Getting 1+ Workers Getting 3+ Workers Cmarket = 8 Cmarket = 8 Cmarket = 8 Cmarket = 8
  16. 16. 16 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Does a Deployed Retainer Work? • Yes, and (surprisingly) stable. • Response rate = Proportion of workers in the retainer who responded within 20 secs when being called Response rate is stable over time. Month of a Year Day of a Week Retainer Time
  17. 17. 17 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] What do workers think? • Worker Survey • Collect opinions and self-reported behavior • Ask 156 workers who completed 10+ Ignition HITs • $2 compensation (~10mins) • 101 workers finished the survey (Response Rate = 64.74%)
  18. 18. 18 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] General HITs v.s. Ignition HITs "I like doing HITs on MTurk in general." "I like doing Retainer HITs." • 5-point Likert scale • Strongly Disagree to Strongly Agree
  19. 19. 19 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] How to recruit workers quickly on-demand? On-demand (QuikTurkit, Bigham, et al., ‘10) On-call (Retainer Model, Bernstein, et al.,‘11) Ignition Method Post HITs + SEO Retainer Hybrid Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds How to Hold Workers Using old tasks Waiting Page Waiting Page Recruiting Cost Per Task Old tasks + SEO (mturk fee) All Workers’ Waiting Time 55% of Workers’ Waiting Time Recruiting Cost For Small Deployment Very Low ( k* #tasks ) High ( k* duration ) Low ( k* #tasks +)
  20. 20. 20 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Conclusion & Future Work • Ignition Model • Posting HIT: Cheap, but Slow • Retainer: Fast, but Costly -> Ignition balances both • 10-Month-Long Deployment Study • Adding a light-weight retainer speeds up recruiting • Deploying a low-latency crowd-powered system is feasible • What’s Next? • Improving Ignition to address workers’ feedback • Code Release Soon!
  21. 21. 21 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Thank you! @windx0303 KennethHuang.cc

×