Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A 10-Month-Long Deployment Study of
On-Demand Recruiting for
Low-Latency Crowdsourcing
Live Feedback & Questions: http://t...
2
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Chorus: A Crowd-powered
Conversation Assistant
"Is th...
3
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Constraints of Deploying Chorus
• Small user traffic
...
4
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
How to recruit workers
quickly on-demand?
On-demand
(...
5
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Engines Generate Power
On-demand
(QuikTurkit,
Bigham,...
6
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Ignition Model
1. Hire slightly more workers than we ...
7
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Ignition = Retainer + Market
• When you want to recru...
8
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Waiting Page & Point System
http://TinyUrl.com/Igniti...
9
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Transition Graph for Workers in Ignition
10
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Transition Graph for Workers in Ignition
11
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Transition Graph for Workers in Ignition
12
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
1. The task occurs instantly [ Instant ]
2. The task...
13
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Ignition Deployment
• From July 1st, 2016 to April 2...
14
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Posting More Assignments -> Gets Workers Faster
Simp...
15
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Simply Posting HITs v.s. Ignition
Getting 1+ Workers...
16
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Does a Deployed Retainer Work?
• Yes, and (surprisin...
17
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
What do workers think?
• Worker Survey
• Collect opi...
18
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
General HITs v.s. Ignition HITs
"I like doing HITs o...
19
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
How to recruit workers
quickly on-demand?
On-demand
...
20
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Conclusion & Future Work
• Ignition Model
• Posting ...
21
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Thank you!
@windx0303 KennethHuang.cc
Upcoming SlideShare
Loading in …5
×

of

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 1 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 2 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 3 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 4 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 5 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 6 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 7 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 8 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 9 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 10 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 11 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 12 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 13 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 14 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 15 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 16 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 17 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 18 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 19 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 20 A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Slide 21
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing

Download to read offline

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing
Ting-Hao K. Huang, Jeffrey P. Bigham.
In Proceedings of The fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017), 2017, Quebec City, Canada.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing

  1. 1. A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Live Feedback & Questions: http://tinyurl.com/20171026IGNITION Ting-Hao (Kenneth) Huang Jeffrey P. Bigham Carnegie Mellon University Image: https://pixabay.com/photo-743920/
  2. 2. 2 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Chorus: A Crowd-powered Conversation Assistant "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang et. al., HCOMP 2016
  3. 3. 3 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Constraints of Deploying Chorus • Small user traffic • Avg. 2.97 conversations per day • Fast worker response time • Response time to a message ~= 32 sec (student participant)* • Response time to a message ~= 105 sec (startup participant)* • Need multiple workers to work together • Limited budget • 24-7 service * IM waiting: timing and responsiveness in semi-synchronous communication. Avrahami et al. CSCW’08. …Similar to many academic projects !
  4. 4. 4 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] How to recruit workers quickly on-demand? On-demand (QuikTurkit, Bigham, et al., ‘10) On-call (Retainer Model, Bernstein, et al.,‘11) Ignition Method Post HITs + SEO Retainer Hybrid Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds How to Hold Workers Using old tasks Waiting Page Waiting Page Recruiting Cost Per Task Old tasks + SEO (mturk fee) All Workers’ Waiting Time 55% of Workers’ Waiting Time Recruiting Cost For Small Deployments Very Low ( k* #tasks ) High ( k* duration ) Low ( k* #tasks +) …Can we have something inexpensive and fast?
  5. 5. 5 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Engines Generate Power On-demand (QuikTurkit, Bigham, et al., ‘10) On-call (Retainer Model, Bernstein, et al.,‘11) Ignition Method Post HITs + SEO Retainer Hybrid Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds How to Hold Workers Using old tasks Waiting Page Waiting Page Recruiting Cost Per Task Old tasks + SEO (mturk fee) All Workers’ Waiting Time 55% of Workers’ Waiting Time Recruiting Cost For Small Deployments Very Low ( k* #tasks ) High ( k* duration ) Low ( k* #tasks +) • Crowd-powered System Why not both? Re-Start the Engine Each Time Start the Engine Once, and Hold the Power Image: https://commons.wikimedia.org/wiki/File:3791_cc_Holden_3800_V6_engine_(2015-08-29)_03.jpg
  6. 6. 6 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Ignition Model 1. Hire slightly more workers than we need 2. Hold the extra workers in the retainer 3. Always call workers from retainer first, and then post HITs
  7. 7. 7 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Ignition = Retainer + Market • When you want to recruit 8 workers each time… • A Chorus task need at most 5 workers #Workers Recruited from Retainer #Workers Recruited from Mturk Market ( Posting a HIT w/ Cmarket Assignments ) Posting More Assignments -> Gets Workers Faster Cretainer + Cmarket = 8
  8. 8. 8 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Waiting Page & Point System http://TinyUrl.com/IgnitionDemo
  9. 9. 9 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Transition Graph for Workers in Ignition
  10. 10. 10 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Transition Graph for Workers in Ignition
  11. 11. 11 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Transition Graph for Workers in Ignition
  12. 12. 12 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] 1. The task occurs instantly [ Instant ] 2. The task occurs later [ Retained ] 3. No tasks at all [ No Task ] Transition Graph for Workers in Ignition
  13. 13. 13 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Ignition Deployment • From July 1st, 2016 to April 27th, 2017 (~10 months) • 6,823 assignments, 648 workers • $0.25 for a 30-minute waiting time • Mturk Qualifications Used • HIT Approval Rate > 90%, #Approved HITs> 200, Adult Content Qualification Assignments Tasks (Conversation) #workers in retainer 50% of tasks occurred when 1+ workers in the retainer
  14. 14. 14 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Posting More Assignments -> Gets Workers Faster Simply Posting HITs Getting 1+ Workers Getting 3+ Workers X-Axis: Time (x seconds) Y-Axis: Prob(Getting N+ workers within x seconds) Cmarket = 3 Cmarket = 8 Cmarket = 8 Cmarket = 3
  15. 15. 15 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Simply Posting HITs v.s. Ignition Getting 1+ Workers Getting 3+ Workers Cmarket = 8 Cmarket = 8 Cmarket = 8 Cmarket = 8
  16. 16. 16 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Does a Deployed Retainer Work? • Yes, and (surprisingly) stable. • Response rate = Proportion of workers in the retainer who responded within 20 secs when being called Response rate is stable over time. Month of a Year Day of a Week Retainer Time
  17. 17. 17 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] What do workers think? • Worker Survey • Collect opinions and self-reported behavior • Ask 156 workers who completed 10+ Ignition HITs • $2 compensation (~10mins) • 101 workers finished the survey (Response Rate = 64.74%)
  18. 18. 18 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] General HITs v.s. Ignition HITs "I like doing HITs on MTurk in general." "I like doing Retainer HITs." • 5-point Likert scale • Strongly Disagree to Strongly Agree
  19. 19. 19 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] How to recruit workers quickly on-demand? On-demand (QuikTurkit, Bigham, et al., ‘10) On-call (Retainer Model, Bernstein, et al.,‘11) Ignition Method Post HITs + SEO Retainer Hybrid Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds How to Hold Workers Using old tasks Waiting Page Waiting Page Recruiting Cost Per Task Old tasks + SEO (mturk fee) All Workers’ Waiting Time 55% of Workers’ Waiting Time Recruiting Cost For Small Deployment Very Low ( k* #tasks ) High ( k* duration ) Low ( k* #tasks +)
  20. 20. 20 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Conclusion & Future Work • Ignition Model • Posting HIT: Cheap, but Slow • Retainer: Fast, but Costly -> Ignition balances both • 10-Month-Long Deployment Study • Adding a light-weight retainer speeds up recruiting • Deploying a low-latency crowd-powered system is feasible • What’s Next? • Improving Ignition to address workers’ feedback • Code Release Soon!
  21. 21. 21 [ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ] Thank you! @windx0303 KennethHuang.cc

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Ting-Hao K. Huang, Jeffrey P. Bigham. In Proceedings of The fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017), 2017, Quebec City, Canada.

Views

Total views

639

On Slideshare

0

From embeds

0

Number of embeds

3

Actions

Downloads

6

Shares

0

Comments

0

Likes

0

×