1. A 10-Month-Long Deployment Study of
On-Demand Recruiting for
Low-Latency Crowdsourcing
Live Feedback & Questions: http://tinyurl.com/20171026IGNITION
Ting-Hao (Kenneth) Huang
Jeffrey P. Bigham
Carnegie Mellon University
Image: https://pixabay.com/photo-743920/

2. 2
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Chorus: A Crowd-powered
Conversation Assistant
"Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered
Conversational Agent. Huang et. al., HCOMP 2016

3. 3
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Constraints of Deploying Chorus
• Small user traffic
• Avg. 2.97 conversations per day
• Fast worker response time
• Response time to a message ~= 32 sec (student participant)*
• Response time to a message ~= 105 sec (startup participant)*
• Need multiple workers to work together
• Limited budget
• 24-7 service
* IM waiting: timing and responsiveness in semi-synchronous communication. Avrahami et al. CSCW’08.
…Similar to many academic projects !

4. 4
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
How to recruit workers
quickly on-demand?
On-demand
(QuikTurkit,
Bigham, et al., ‘10)
On-call
(Retainer Model,
Bernstein, et al.,‘11)
Ignition
Method Post HITs + SEO Retainer Hybrid
Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds
How to Hold
Workers
Using old tasks Waiting Page Waiting Page
Recruiting Cost
Per Task
Old tasks
+ SEO (mturk fee)
All Workers’
Waiting Time
55% of Workers’
Waiting Time
Recruiting Cost
For Small
Deployments
Very Low
( k* #tasks )
High
( k* duration )
Low
( k* #tasks +)
…Can we have
something
inexpensive
and fast?

5. 5
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Engines Generate Power
On-demand
(QuikTurkit,
Bigham, et al., ‘10)
On-call
(Retainer Model,
Bernstein, et al.,‘11)
Ignition
Method Post HITs + SEO Retainer Hybrid
Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds
How to Hold
Workers
Using old tasks Waiting Page Waiting Page
Recruiting Cost
Per Task
Old tasks
+ SEO (mturk fee)
All Workers’
Waiting Time
55% of Workers’
Waiting Time
Recruiting Cost
For Small
Deployments
Very Low
( k* #tasks )
High
( k* duration )
Low
( k* #tasks +)
• Crowd-powered System
Why not
both?
Re-Start
the
Engine
Each Time
Start the
Engine
Once, and
Hold the
Power
Image: https://commons.wikimedia.org/wiki/File:3791_cc_Holden_3800_V6_engine_(2015-08-29)_03.jpg

6. 6
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Ignition Model
1. Hire slightly more workers than we need
2. Hold the extra workers in the retainer
3. Always call workers from retainer first, and then post HITs

7. 7
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Ignition = Retainer + Market
• When you want to recruit 8 workers each time…
• A Chorus task need at most 5 workers
#Workers
Recruited from
Retainer
#Workers Recruited from
Mturk Market
( Posting a HIT w/
Cmarket Assignments )
Posting More Assignments -> Gets Workers Faster
Cretainer + Cmarket = 8

8. 8
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Waiting Page & Point System
http://TinyUrl.com/IgnitionDemo

9. 9
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Transition Graph for Workers in Ignition

10. 10
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Transition Graph for Workers in Ignition

11. 11
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Transition Graph for Workers in Ignition

12. 12
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
1. The task occurs instantly [ Instant ]
2. The task occurs later [ Retained ]
3. No tasks at all [ No Task ]
Transition Graph for Workers in Ignition

13. 13
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Ignition Deployment
• From July 1st, 2016 to April 27th, 2017 (~10 months)
• 6,823 assignments, 648 workers
• $0.25 for a 30-minute waiting time
• Mturk Qualifications Used
• HIT Approval Rate > 90%, #Approved HITs> 200, Adult Content Qualification
Assignments Tasks (Conversation)
#workers in retainer
50% of tasks occurred when
1+ workers in the retainer

14. 14
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Posting More Assignments -> Gets Workers Faster
Simply Posting HITs
Getting 1+ Workers Getting 3+ Workers
X-Axis: Time (x seconds)
Y-Axis: Prob(Getting N+ workers within x seconds)
Cmarket = 3
Cmarket
= 8
Cmarket = 8
Cmarket = 3

15. 15
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Simply Posting HITs v.s. Ignition
Getting 1+ Workers Getting 3+ Workers
Cmarket
= 8
Cmarket = 8
Cmarket = 8
Cmarket = 8

16. 16
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Does a Deployed Retainer Work?
• Yes, and (surprisingly) stable.
• Response rate = Proportion of workers in the retainer
who responded within 20 secs when being called
Response rate is stable over time.
Month of a Year Day of a Week Retainer Time

17. 17
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
What do workers think?
• Worker Survey
• Collect opinions and self-reported behavior
• Ask 156 workers who completed 10+ Ignition HITs
• $2 compensation (~10mins)
• 101 workers finished the survey (Response Rate = 64.74%)

18. 18
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
General HITs v.s. Ignition HITs
"I like doing HITs on MTurk in general." "I like doing Retainer HITs."
• 5-point Likert scale
• Strongly Disagree to Strongly Agree

19. 19
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
How to recruit workers
quickly on-demand?
On-demand
(QuikTurkit,
Bigham, et al., ‘10)
On-call
(Retainer Model,
Bernstein, et al.,‘11)
Ignition
Method Post HITs + SEO Retainer Hybrid
Latency 60 - 120 seconds < 10 seconds 20 – 90 seconds
How to Hold
Workers
Using old tasks Waiting Page Waiting Page
Recruiting Cost
Per Task
Old tasks
+ SEO (mturk fee)
All Workers’
Waiting Time
55% of Workers’
Waiting Time
Recruiting Cost
For Small
Deployment
Very Low
( k* #tasks )
High
( k* duration )
Low
( k* #tasks +)

20. 20
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Conclusion & Future Work
• Ignition Model
• Posting HIT: Cheap, but Slow
• Retainer: Fast, but Costly
-> Ignition balances both
• 10-Month-Long Deployment Study
• Adding a light-weight retainer speeds up recruiting
• Deploying a low-latency crowd-powered system is feasible
• What’s Next?
• Improving Ignition to address workers’ feedback
• Code Release Soon!

21. 21
[ Live Feedback & Questions: http://tinyurl.com/20171026Ignition ]
Thank you!
@windx0303 KennethHuang.cc