Reactive crowdsourcing

1. REACTIVECROWDSOURCINGAlessandro BozzonabMarco BrambillaaStefano CeriaAndrea MauriaaPolitecnico di MilanoDipartimento di Elettronica, Informazione e BioIngegneriabDelft University Of TechnologyDepartment of Software And Computer Technology

2. Crowd Control is tough…• There are several aspects that makes crowdengineering complicated• Task design, planning, assignment• Workers discovery, assessment, engagementWednesday, May 15 Reactive Crowdsourcing 2http://xkcd.com/1060/

3. Crowd Control is tough…Wednesday, May 15 Reactive Crowdsourcing 3• There are several aspects that makes crowdengineering complicated• Task design, planning, assignment• Workers discovery, assessment, engagement• Goal: taming the crowd• Cost• Time• Quality

4. Crowd Control is tough…Wednesday, May 15 Reactive Crowdsourcing 4• There are several aspects that makes crowdengineering complicated• Task design, planning, assignment• Workers discovery, assessment, engagement• Goal: taming the crowd• Cost• Time• Quality• Motivation!• Need for higher level abstractions and tools• CONTROL as first-class citizen

5. Reactive Crowdsourcing• A conceptual framework for modeling crowdsourcingcomputations and control requirements• Task Design• Reactive Control Design• Active Rule programming framework• Declarative rule language• A reactive execution environment for requirementenforcement and reactive execution• Based on the CrowdSearcher approachWednesday, May 15 Reactive Crowdsourcing 5

6. Why Active Rules?• Crowdsourcing control typically focuses on task data• Execution results, agreement on truth value, workers performance• An active rule approach can provide• Ease of Use: control is easily expressible• Simple control data structures• Familiar formalism• Power: support to arbitrarily complex controls• Extensibility mechanisms• Automation: most active rules can be system-generated• Well-defined semantic• Flexibility: simple control variants have localized impact on therules set• Control isolationWednesday, May 15 Reactive Crowdsourcing 6

7. The CrowdSearcher Approach• Human-Enhanced data management with social networksand Q&A systems as crowdsourcing platforms• Example: search task (WWW2012)Wednesday, May 15 Reactive Crowdsourcing 7Human Interaction ManagementSocialNetworksHumanComputationPlatformsQ&ASearch ExecutionEngineractionentQuery InterfaceSocialNetworksery AnswerSearch ExecutionEngineanInteractionanagementHumanQuery InterfaceLocalSocialNetworksQ&AQuery AnswerSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSocialNetworksQ&ACrowd-sourceplatformsQuery AnswerSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSocialNetworksQ&ACrowd-sourceplatformsQuery AnswerData Management SystemHuman AccessInterfaceRemote DataAccessLocal DataAccessSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSocialNetworksQ&ACrowd-sourceplatformsQuery AnswerSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSociaNetworQ&ACrowdsourceplatformQuery AnswerTaskHuman-EnhancedDataQueryResults

8. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessWednesday, May 15 Reactive Crowdsourcing 8I O

9. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessTaskSpecificationTask PlanningControlSpecificationWednesday, May 15 Reactive Crowdsourcing 9• Task Spec: task operations, objects, and performers Dimension Tables

10. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessTaskSpecificationTask PlanningControlSpecificationWednesday, May 15 Reactive Crowdsourcing 10• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution  Execution Table for task monitoring

11. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessTaskSpecificationTask PlanningControlSpecificationWednesday, May 15 Reactive Crowdsourcing 11• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution  Execution Table for task monitoring• Control Specification: task control policies  Control Mart

12. Task Specification_1/3Wednesday, May 15 Reactive Crowdsourcing 12• Operation Types: Choice, Like, Score, Tag, Classify, Order, …• Operation Parameters: e.g. classification classesTasktIDopTypecategoriesTask Specification Task Planning Control SpecificationTask Configurationt1Classify Rep/Dem 

13. Task Specification_2/3Wednesday, May 15 Reactive Crowdsourcing 13PoliticianclassifiedPartylastNamephotooID• Input Objects Schema: typed attributes• Output Attributes (according to task type)TasktIDopTypecategoriesTask Specification Task Planning Control SpecificationTask ConfigurationObjectSpecification o1 Obama http://…. ?????

14. Task Specification_3/3Wednesday, May 15 Reactive Crowdsourcing 14PoliticianclassifiedPartylastNamephotooIDTask ConfigurationObjectSpecificationPerformerSpecification• Execution platform(s)• Qualifications, etc.TasktIDopTypecategories PerformernamepIDplatformTask Specification Task Planning Control Specification p1 Alessandro Facebook

15. Task Planning_1/2• Organize the task in MicroTasks, and allocate input objects• μTaskObjectExecution  Designed for execution monitoring• Track performers responseclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDWednesday, May 15 Reactive Crowdsourcing 15PoliticianclassifiedPartylastNamephotooIDTasktIDopTypecategories PerformernamepIDplatformSplittingTask Specification Task Planning Control Specification mt1 O1 … …… … Facebook

16. Task Planning_2/2• Assign performers to MicroTasks on platforms• Pull: dynamic assignment (First come - First served / Choice of theperformer)• Push: static assignment (Performers’ priority / Performer matching)classifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDWednesday, May 15 Reactive Crowdsourcing 16PoliticianclassifiedPartylastNamephotooIDTasktIDopTypecategories PerformernamepIDplatformSplitting AssignmentTask Specification Task Planning Control Specification mt1 O1 P1 Republican00:00:01 00:00:10 Facebook

17. Control Specification_1/4Wednesday, May 15 Reactive Crowdsourcing 17Task Specification Task Planning Control Specification• Status Variable: tracking task and performers statusclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatus  Trusted/SpammerCreated/Planned/Closed 

18. Control Specification_2/4Wednesday, May 15 Reactive Crowdsourcing 18Task Specification Task Planning Control Specification• Object : tracking objects statusclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatusObjectControl #demoID#eval#rep#curAnswer

19. Control Specification_3/4Wednesday, May 15 Reactive Crowdsourcing 19Task Specification Task Planning Control Specification• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)PerformerControl #rightpID#eval#wrongclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatusObjectControl #demoID#eval#rep#curAnswer

20. Control Specification_4/4• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)• Task: tracking task status: closing @completion, re-planWednesday, May 15 Reactive Crowdsourcing 20TaskControl#compObjtID#compExecPerformerControl #rightpID#eval#wrongclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatusObjectControl #demoID#eval#rep#curAnswerTask Specification Task Planning Control Specification

21. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigmWednesday, May 15 Reactive Crowdsourcing 21

22. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigm• Events: data updates / timer• ROW-level granularity• OLD  before state of a row• NEW  after state of a rowWednesday, May 15 Reactive Crowdsourcing 22e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]

23. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigm• Events: data updates / timer• ROW-level granularity• OLD  before state of a row• NEW  after state of a row• Condition: a predicate that must be satisfied (e.g. conditions oncontrol mart attributes)Wednesday, May 15 Reactive Crowdsourcing 23e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]c: NEW.ClassifiedParty == ’Republican’

24. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigm• Events: data updates / timer• ROW-level granularity• OLD  before state of a row• NEW  after state of a row• Condition: a predicate that must be satisfied (e.g. conditions oncontrol mart attributes)• Actions: updates on data structures (e.g. change attributevalue, create new instances), special functions (e.g. replan)Wednesday, May 15 Reactive Crowdsourcing 24e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]c: NEW.ClassifiedParty == ’Republican’a: SET ObjectControl[oID == NEW.oID].#Eval+= 1

25. Wednesday, May 15 Reactive Crowdsourcing 25e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDEvent

26. Wednesday, May 15 Reactive Crowdsourcing 26e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDCondition

27. Wednesday, May 15 Reactive Crowdsourcing 27e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDAction

28. Wednesday, May 15 Reactive Crowdsourcing 28e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDAction

29. Rule Programming Best Practice• We define three classes of rulesWednesday, May 15 Reactive Crowdsourcing 29μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task

30. Rule Programming Best PracticeWednesday, May 15 Reactive Crowdsourcing 30• We define three classes of rules• Control rules: modifying the control tables;μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task

31. Rule Programming Best PracticeWednesday, May 15 Reactive Crowdsourcing 31• We define three classes of rules• Control rules: modifying the control tables;• Result rules: modifying the dimension tables (object, performer, task);μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task

32. Rule Programming Best PracticeWednesday, May 15 Reactive Crowdsourcing 32• Top-to-bottom, left-to-right, evaluation• Guaranteed termination• We define three classes of rules• Control rules: modifying the control tables;• Result rules: modifying the dimension tables (object, performer, task);μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task

33. Rule Programming Best Practice• We define three classes of rules• Control rules: modifying the control tables;• Result rules: modifying the dimension tables (object, performer, task);• Execution rules: modifying the execution table, either directly or through re-planningWednesday, May 15 Reactive Crowdsourcing 33μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task• Termination must be proven (Rule precedence graph has cycles)

34. Experimental Evaluation• GOAL: demonstrate the flexibility and expressive powerof reactive crowdsourcing• 3 experiments, focused on Italian politicians• Parties: Human Computation  affiliation classification• Law: Game With a Purpose  guess the convicted politician• Order: Pure Game  hot or not• 1 week (November 2012)• 284 distinct performers• Recruited through public mailing lists and social networksannouncements• 3500 Micro TasksWednesday, May 15 Reactive Crowdsourcing 34

35. Politician Affiliation• Given the picture and name of a politician, specify his/her politicalaffiliation• No time limit• Performers are encouraged to look up online• 2 set of rules• Majority Evaluation• Spammer DetectionWednesday, May 15 Reactive Crowdsourcing 35

36. Results – Majority Evaluation_1/3Wednesday, May 15 Reactive Crowdsourcing 3630 object; object redundancy = 9;Final object classification as simple majority after 7 evaluations

37. Results - Majority Evaluation_2/3Wednesday, May 15 Reactive Crowdsourcing 37Majority @7Early Majority @3 R @7-27% executions-18% precision%ofCompl.Objects00.10.20.30.40.50.60.70.80.91.0Precision00.10.20.30.40.50.60.70.8#Executions0 10 20 30 40 50 60 70 80 90Final object classification as total majority after 3 evaluationsOtherwise, re-plan of 4 additional evaluations. Then simple majority at 7

38. Results - Majority Evaluation_3/3Wednesday, May 15 Reactive Crowdsourcing 38Majority @7Early Majority @3 R @7Majority @3 R @5 R @7-23% executions+26% precision+50% precision%ofCompl.Objects00.10.20.30.40.50.60.70.80.91.0Precision00.10.20.30.40.50.60.70.8#Executions0 10 20 30 40 50 60 70 80 90Final object classification as total majority after 3 evaluationsOtherwise, simple majority at 5 or at 7 (with replan)

39. Results – Spammer Detection_1/2Wednesday, May 15 Reactive Crowdsourcing 39New rule for spammer detection without ground truthPerformer correctness on final majority. Spammer if > 50% wrong classificationsMajority @3 R @5 R @7Majority @3 R @5 R @7 - Spammer Detection+46% executions+1.5% precision%ofCompl.Objects00.10.20.30.40.50.60.70.80.91.0Precision00.10.20.30.40.50.60.70.8#Executions0 10 20 30 40 50 60 70 80 90

40. A Short CrowdSearcher DemoWednesday, May 15 Reactive Crowdsourcing 40

41. Summary• Results• An integrated framework for crowdsourcing task design and control• Well-structured control rules with some guarantees of termination• Support for cross-platform crowd interoperability• A working prototype  crowdsearcher.search-computing.org• Forthcoming• Exploitation of interoperability• Expertise finding• Dynamic planning• Integration with other social-networks and human computationplatformsWednesday, May 15 Reactive Crowdsourcing 41

42. QUESTIONS?Wednesday, May 15 Reactive Crowdsourcing 42