0
Pattern-Based Specification of
Crowdsourcing Applications
Alessandro Bozzon (TU Delft)
Marco Brambilla (Politecnico di Mila...
Crowdsourcing and Human Computation
It works like magic!
Endless Applications
Endless Success Stories
2008 Olympics Openin...
Actually…
Often a Try&HopeError process
Task Design Matters
Crowd can be unreliable
($) Incentives Matter
Quality Control ...
Setting Nº1
Global Annotations with
simple counting
Setting Nº2
Local Annotations
with Bounding Boxes
Setting Nº3
Local Annotations with Verified Bounding Boxes
Setting Nº3
Local Annotations with Verified Bounding Boxes
Setting Nº3
Local Annotations with Verified Bounding Boxes
Ok, so what?
#Workers
#Useful
Workers
#Executions Cost $
Time
(hours)
Precision
Setting
Nº1
732
44
(6%)
488 72 ~40 ~67%
Se...
Our study
Our study
Our Contribution
GOAL
!
Simplify and systematize the design, deploy, and
monitoring of applications (including experiments)
Contributions
An Abstract
Model of
Crowdsourcing
Activities
A Composition
model for
Crowdsourcing
Activities
A Library of
...
Models
DEMO VIDEO
Crowd Task
[T operation types]
(intra-task patterns)
Object Type
block size
min #obj
(cons)
input buffer
batch flow (on clo...
Case Study: Movie Scene Analysis
Scenario 1: Scene Positioning
Spoiler Alert!
Order Scenes
Scene in Beg/Mid/End
Scenario 2...
Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spoiler Scenes
[Like]
(Static Agr...
Patterns
Intra-Task Auxiliary Workflow
Intra-Task
Pre-Processing
Post-
Processing
Task
Consensus
Splitting
Assignment
Aggregation
microTaskmicroTaskmicroTask
mic...
Auxiliary Intra-Task
Pre-Processing
Post-
Processing
Task
Consensus
Splitting
Assignment
Aggregation
microTaskmicroTaskmic...
Workflow
• Set of heterogeneous
tasks
Create Decide
Improve
Compare
/ Verify
Find Fix
(a)
(b)
(c)
Auxiliary
Task
Create/Dec...
Experiments
1700 Executions
39$
September 2013
Streaming Vs. Batch Execution
Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spo...
Intra-Task Consensus Vs. Workflow Decision
A4 A5 A6
Precision
 0.99 0.95 0.89
Recall 0.90 1 0.96
F-Score 0.93 0.97 0.90
Fin...
Take-Home Message
Engineering approaches should be
applied to people management too
In crowdsourcing, control and
monitori...
Questions?
Pattern-Based Specification of Crowdsourcing Applications
Pattern-Based Specification of Crowdsourcing Applications
Pattern-Based Specification of Crowdsourcing Applications
Pattern-Based Specification of Crowdsourcing Applications
Upcoming SlideShare
Loading in...5
×

Pattern-Based Specification of Crowdsourcing Applications

3,386

Published on

Transcript of "Pattern-Based Specification of Crowdsourcing Applications"

  1. 1. Pattern-Based Specification of Crowdsourcing Applications Alessandro Bozzon (TU Delft) Marco Brambilla (Politecnico di Milano) Stefano Ceri (Politecnico di Milano) Andrea Mauri (Politecnico di Milano) Riccardo Volonterio (Politecnico di Milano)
  2. 2. Crowdsourcing and Human Computation It works like magic! Endless Applications Endless Success Stories 2008 Olympics Opening Ceremony
  3. 3. Actually… Often a Try&HopeError process Task Design Matters Crowd can be unreliable ($) Incentives Matter Quality Control Matters Platform of Execution Matters
  4. 4. Setting Nº1 Global Annotations with simple counting
  5. 5. Setting Nº2 Local Annotations with Bounding Boxes
  6. 6. Setting Nº3 Local Annotations with Verified Bounding Boxes
  7. 7. Setting Nº3 Local Annotations with Verified Bounding Boxes
  8. 8. Setting Nº3 Local Annotations with Verified Bounding Boxes
  9. 9. Ok, so what? #Workers #Useful Workers #Executions Cost $ Time (hours) Precision Setting Nº1 732 44 (6%) 488 72 ~40 ~67% Setting Nº2 498 25 (5%) 547 48 ~169 ~67% Setting Nº3 1420 464 (32%) 3387 83 ~184 ~75% Total 2152 508 4422 203 ~16 days
  10. 10. Our study
  11. 11. Our study Our Contribution
  12. 12. GOAL ! Simplify and systematize the design, deploy, and monitoring of applications (including experiments)
  13. 13. Contributions An Abstract Model of Crowdsourcing Activities A Composition model for Crowdsourcing Activities A Library of crowdsourcing Patterns • A conceptual framework • A specification paradigm • A reactive execution control environment
  14. 14. Models
  15. 15. DEMO VIDEO
  16. 16. Crowd Task [T operation types] (intra-task patterns) Object Type block size min #obj (cons) input buffer batch flow (on closed task) stream flow (on closed object) MicroTask [MT operation types] r data manipulator
  17. 17. Case Study: Movie Scene Analysis Scenario 1: Scene Positioning Spoiler Alert! Order Scenes Scene in Beg/Mid/End Scenario 2: Actor Identification Find Actor Validate Actor
  18. 18. Position Scenes [Classify] (Static Agreement@3) MicroTask [Classify] Scene block 1 min 1 Spoiler Scenes [Like] (Static Agreement@3) MicroTask [Like] Scene block 1 min 1 5 Order Scenes [Order] (SortByLiking) MicroTask [Like] Scene block 2 min 2 [Class=E] [Class=B OR M] Example of Scenario 1 Model
  19. 19. Patterns Intra-Task Auxiliary Workflow
  20. 20. Intra-Task Pre-Processing Post- Processing Task Consensus Splitting Assignment Aggregation microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask • Consensus • Join • Sort • Grouping • Performer Control • Planning • Assignment • Aggregation • Quality & Performer
  21. 21. Auxiliary Intra-Task Pre-Processing Post- Processing Task Consensus Splitting Assignment Aggregation microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask • Pruning • Tie Breaking • Operations before or after the execution
  22. 22. Workflow • Set of heterogeneous tasks Create Decide Improve Compare / Verify Find Fix (a) (b) (c) Auxiliary Task Create/Decide Improve/Compare Find/Fix/Verify
  23. 23. Experiments 1700 Executions 39$ September 2013
  24. 24. Streaming Vs. Batch Execution Position Scenes [Classify] (Static Agreement@3) MicroTask [Classify] Scene block 1 min 1 Spoiler Scenes [Like] (Static Agreement@3) MicroTask [Like] Scene block 1 min 1 7 Order Scenes [Order] (SortByLiking) MicroTask [Like] Scene block 2 min 2 [Class=E] [Class=B OR M] (P1) 5 3 Position Scenes [Classify] (Static Agreement@3) MicroTask [Classify] Scene block 1 min 1 Spoiler Scenes [Like] (Static Agreement@3) MicroTask [Like] Scene block 1 min 1 7 Order Scenes [Order] (SortByLiking) MicroTask [Like] Scene block 2 min 2 Cons. [Class=E] [Class=B OR M] (P2) 5 3 P.Beg P.Mid P.End P1 0.5 1 0.11 P2 0.5 0.8 0.33 Spear.Beg Spear. Mid .MidP1 0.5 0.54 P2 0.9 0.51 Position Order P1 P b) Elapsed Tim #ClosedObjects 1 10 20 30 40 50 60 70 80 5 60 120 180 240 300 360 5 Position Order P1 Position Order P2 b) Elapsed Time (Mins) #ClosedObjects 1 10 20 30 40 50 60 70 80 5 60 120 180 240 300 360 5 60 120 180 240 300 360 Position Order P1 P b) Elapsed Tim #ClosedObjects 1 10 20 30 40 50 60 70 80 5 60 120 180 240 300 360 5
  25. 25. Intra-Task Consensus Vs. Workflow Decision A4 A5 A6 Precision 0.99 0.95 0.89 Recall 0.90 1 0.96 F-Score 0.93 0.97 0.90 Find Actors [Tag] (Static Agreement@3) MicroTask [Tag] Scene block 1 min 1 Validate Actors [Like] MicroTask [Like] Scene+Actor block All min 1 5 (A4) Find Actors [Tag] MicroTask [Tag] Scene block 1 min 1 Validate Actors [Like] (Majority Voting@2) MicroTask [Like] Scene+Actor block All min 1 5 3 (A5) (A6) Find Actors [Tag] (Static Agreement@3) MicroTask [Tag] Scene block 1 min 1 Validate Actors [Like] (Majority Voting@2) MicroTask [Like] Scene+Actor block All min 1 5 3 count(Actor.Like)<=1 Actor Validate A3 Actor Validate A4 A5 a) Elapsed Time ( #ClosedObjects 0 10 20 30 40 50 60 5 30 60 90 120 160 5 30 60 90 120 160 5 Actor Validate Actor Validate A4 Actor Validate A5 a) Elapsed Time (Mins) 30 60 90 120 160 5 30 60 90 120 160 5 60 120 180 240 60 Actor Validate Actor Validate A5 a) Elapsed Time (Mins) Actor Validate A6 30 60 90 120 160 5 60 120 180 240 60 300 540 780 Actor Validate A3 A4 #ClosedObjects 0 10 20 30 40 50 60 5 30 60 90 120 160 5 Actor Validate A3 A4 #ClosedObjects 0 10 20 30 40 50 60 5 30 60 90 120 160 5 Actor Validate A3 A4 #ClosedObjects 0 10 20 30 40 50 60 5 30 60 90 120 160 5
  26. 26. Take-Home Message Engineering approaches should be applied to people management too In crowdsourcing, control and monitoring are key for successful outcomes
  27. 27. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×