Reactive crowdsourcing

8,274 views

Published on

Slides of the presentation given at the 22nd International Conference on the World Wide Web.

URL: http://www2013.org/program/561-reactive-crowdsourcing/

More information on the Crowdsearcher project available at

crowdsearcher.search-computing.com

Published in: Technology, Business
1 Comment
9 Likes
Statistics
Notes
  • Really interesting Alessandro, thanks ​!

    Given your interest, I think that you would be really interested in some recent research that I have come across that theorizes about crowds and similar phenomena.​ ​

    It’s called “The Theory of Crowd Capital” and you can download it here if you’re interested: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2193115

    In my view it provides a powerful, yet simple model, getting to the heart of the matter. Enjoy!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
8,274
On SlideShare
0
From Embeds
0
Number of Embeds
5,714
Actions
Shares
0
Downloads
46
Comments
1
Likes
9
Embeds 0
No embeds

No notes for slide
  • Good afternoonI’m very happy to be here today and discuss with you our work, named reactive crowdsourcing
  • But first allow me to contextualize a bit its scope. I’m sure we are all very familiar with crowdsourcing and human computation, which is a based on a very neat idea: organize humans to accomplish tasks. Unfortunately, or luckily for us , such organization is a complex matter. When designing systems and experiments, there are several things to consider: how do you split your tasks in microtask, which are the best performers for it, how do you scout them, etc.
  • It turns out, our ultimate goal is controlling the crowd. We would like to drive people behavior in order satisfy some constraints, which are typically monetary, qualitative, or temporal. Indeed, a lot of work has been devoted to devise algorithms able to minimize one of those costs, and a lot of effort has been recently put in the creation of frameworks and algorithm for crowd management.However, we observed that existing solutions (e.g. crowddb, deco, and others) provide limited and veryspecific control features. We advocate crowd control to be at the very center of task design. And we also advocate the need for methods for the systematicdesign of complex control strategies based on the state of tasks, results and performers.
  • It turns out, our ultimate goal is controlling the crowd. We would like to drive people behavior in order satisfy some constraints, which are typically monetary, qualitative, or temporal. Indeed, a lot of work has been devoted to devise algorithms able to minimize one of those costs, and a lot of effort has been recently put in the creation of frameworks and algorithm for crowd management.However, we observed that existing solutions (e.g. crowddb, deco, and others) provide limited and veryspecific control features. We advocate crowd control to be at the very center of task design. And we also advocate the need for methods for the systematicdesign of complex control strategies based on the state of tasks, results and performers.
  • This paper wepropose a conceptual framework and a reactive execution environment for modelling and controlling crowdsourcing computationsWit the ultimate goal of minimizing the effort required for control specification, we propose:a simple task design processA rule specification language,whose properties (e.g., termination) can be easily proved in the context of a well-organized computational framework
  • But why we decided to go for an approach based on active rules?The choice stemmed from the observation that crowdsourcing control is typically driven by data, like the status of the HIT executions, the worker performance, the current agreement on the truth value of some object. Therefore, it came almost natural for us to turn on a data-driven approach, that proven very effective for the definition of control in several contexts, including database systemsActive rules are actually relatively easy to use, when expressed on well-define data structures. They allow the definition of arbitrary complex control logic, most of which can be easily automated thanks to a well-defined syntax and semantic. Also, they allow for a great flexibilty, since changes in the control logic of the application can be well-isolated into localized changes of the rule set
  • This work capitalizes on the results of our www2012 paper, where we an approach for data management that integrates SN, QA and traditional HC platform for the execution of human computation and crowdsourcing tasks.This work specifically addressed crowd-enhanced search, a now very popular trend. However, the approach we proposed can be very easily generalized to any kind of human-enhanced data management system, and we believe this crowd-interoperability can be exploited in very interesting ways. LOCAL SOURCE: sorgentidatilocalisfruttate dal Search Execution Engine, magariaccedutedallo Human Interaction Management per configurare / gestirei task. La suaesistenza e’ accessoriarispettoaglialtri, e codificainformazioni applicative specificheICONE DI DX, DALL’ALTO a SX(social networks) Facebook, Twitter, Google +(Q&A systems) StackOverlflow, Yahoo Answers, Quora(HC Platforms) Freebase, Amazon Mechanical Turk, ODesk
  • The design process we propose is based on a very simple data-driven abstract model for task execution: data in, workers do stuff, data outCoherently with this vision, the task design and deployment process is based upon specific data structures, created in an automated way trough model driven transformations. Unfortunately we don’t have time to go too deep in this matter, so I refer you to the paper for further details.
  • The process composes of three simple steps. The first, task specification, is devoted to the definition of basic task information, such as the data-driven operations to perform (e.g. classify a picture, add new instances of a given object, etc.), the shape of the actual objects to evaluate, and the charactersticis of the targeted set of performers. All those aspect are encoded in what we call Dimension Tables
  • Then, the second step is devoted to task planning, i.e. specify how works should be partitioned and distributed among performers. Those planning aspects are encoded into an Execution Table, specifically devoted to task monitoring
  • Finally, the developer must specify the control logic for the task, and it does so by defining active rules upon control-specific data structures contained in what we call the Control Mart.
  • To clarify the process, let us show visually how the supporting data structure are created in a simple case study: a picture classification task, where the picture contain picture of politicians, and the task is to specify to which party they belong. We will use color codes to identify data structures types, and we highlight in bold those attributes which depend on the specifically designed tasks. All the other are, somehow, standard. The first dimension table is the Task table, which contains info about the performed operations (a classification, in this case), and the categories to which politicians can be assigned (e.g. republican and democtrats)
  • Then, we have a dimension table for the actual objects of the task, in this case, politicians. Notice that the classifiedPartyattribute is actually produced by the crowd at the end of the task
  • Finally, a Performer dimension table represents the population of performer currently available for task execution. The table can be pre-filled, if the performers are known in advance, or empty, in case of openly available tasks.
  • The second activity of the process deals with the planning of the task and it also composes of two phases. In the first phase, suited algorithms will decide such things as how which objects should appear in each MicroTask? (e.g. ground truth assessment), how many objects in each MicroTask, how often an object should appear in MicroTasks, etc. TheμTaskObjectExecution table keeps track of this organization, and it contains one tuple for each object/microTask association. A performer, when executing a microTask, will fill in the data value required by the specified operation (e.g. the party of a given politician)
  • The second phase of task planning deals with the assignment of microTasks to performers. This can be done according to several policies (e.g. pull or push). In the example, the assignment is performed in a pull fashion, and attributes are given value on performer arrival
  • The third part of the process is devoted to the specification of control. Please notice here that I added to an addition color code to denote attributes in the Dimension Tables which serve task control purposes. So for instance, a performer can be classified as trusted or spammer. A task as “created/planned/closed” and so on
  • Bin addition to status variable, control is defined upon the Control Mart, which composes of three data structures. The first one, called ObjectControl contains control variable related to the object of the task, like the number of evaluation it received, the number of classifications it received for each class, and the current truth value.
  • The second one, called PerformerControl contains control variable related to the performer of the task, like the number of evaluation she performerd, and the number of times she provided a right or wrong answer (assuming the existance of a ground truth)
  • Finally, the thirdone, called TaskControl contains control variable related to the task, like the number total number of objects currently evaluated, or the number of executed hits.
  • The previous data structure provide in a very simple yet complete way the control variables that are needed to define the task control policies. But how can control be specified? We rely on a language based on the classic ECA paradigm
  • Where events are updated on the data structure values. We decided for a row-level update granularity, so to easily track the before and after states of rows. In the example, the rule triggers when the ClassifiedParty attribute of a tuple in the μTaskObjectExecution table changes
  • Conditions are expressed as conditions on data attributes (e.g., the value specified by the performer)
  • And actions are updated performed on the same, or other data structures. Such updates can be done directly or trough special functions, devoted to such operations as replanning’Of course there is no time to show the syntax of the language, but you can find more on the paper.
  • The second rule is a bit more complex, and it is used to assess the truth value of an object trough majority voting. For instance, here we assume that as soon as a Politician gets 2 evaluation as Rep or Dem, the object can be deemed as completed. triggers when the ObjectControl table updates
  • It is quite known that active rule programming can be rather subtle and unstable, as the behavior of a set of rules may change dramatically as a consequence of small changes in the rules To simply, and better control rule programming we devise three classes of rules which, as I will show soon, have interesting properties
  • The first class are named control rules, and are meant to modify control tables. Arrows represents rules triggering on a table (the source of the arrow) and affecting another table (the destination). As you can see, not all the possible source-target couple are adimissible, and I will explain soon why.
  • The second class of rules modify the dimension tables, and are the one devoted to changing the status of the main task entities (e.g. setting a perforomer as spammer when she makes too many bad classifications)
  • Note that, since we suggest a very well-defined top-to-bottom, left-to-right semantic, no cycles are allowed, and therefore rules are guardanteed to terminate. Those cycles in the object control – performer control etc. still bases on precise rules also on the attributes of the table
  • Finally, execution rules are responsivble for the modification of the execution table, and are therefore responsible for modifying the set?distribution/assignment of the currently defined microtasks. Those rules introduce cycles and, therefore, might cause unconvergence (and termination must be verified)
  • To demonstrate the flexibily and expressive power of reactive crowdsourcing we advised the experiments, conducted during one week of November 2012. We developed three very different scenario, all programmed with our approach. Unfortunately we don’t have enough time to describe them all, so we focus on just one. A classification task similar to the one used as example in the presentation
  • Control result precisionspammer detection
  • Reactive crowdsourcing

    1. 1. REACTIVECROWDSOURCINGAlessandro BozzonabMarco BrambillaaStefano CeriaAndrea MauriaaPolitecnico di MilanoDipartimento di Elettronica, Informazione e BioIngegneriabDelft University Of TechnologyDepartment of Software And Computer Technology
    2. 2. Crowd Control is tough…• There are several aspects that makes crowdengineering complicated• Task design, planning, assignment• Workers discovery, assessment, engagementWednesday, May 15 Reactive Crowdsourcing 2http://xkcd.com/1060/
    3. 3. Crowd Control is tough…Wednesday, May 15 Reactive Crowdsourcing 3• There are several aspects that makes crowdengineering complicated• Task design, planning, assignment• Workers discovery, assessment, engagement• Goal: taming the crowd• Cost• Time• Quality
    4. 4. Crowd Control is tough…Wednesday, May 15 Reactive Crowdsourcing 4• There are several aspects that makes crowdengineering complicated• Task design, planning, assignment• Workers discovery, assessment, engagement• Goal: taming the crowd• Cost• Time• Quality• Motivation!• Need for higher level abstractions and tools• CONTROL as first-class citizen
    5. 5. Reactive Crowdsourcing• A conceptual framework for modeling crowdsourcingcomputations and control requirements• Task Design• Reactive Control Design• Active Rule programming framework• Declarative rule language• A reactive execution environment for requirementenforcement and reactive execution• Based on the CrowdSearcher approachWednesday, May 15 Reactive Crowdsourcing 5
    6. 6. Why Active Rules?• Crowdsourcing control typically focuses on task data• Execution results, agreement on truth value, workers performance• An active rule approach can provide• Ease of Use: control is easily expressible• Simple control data structures• Familiar formalism• Power: support to arbitrarily complex controls• Extensibility mechanisms• Automation: most active rules can be system-generated• Well-defined semantic• Flexibility: simple control variants have localized impact on therules set• Control isolationWednesday, May 15 Reactive Crowdsourcing 6
    7. 7. The CrowdSearcher Approach• Human-Enhanced data management with social networksand Q&A systems as crowdsourcing platforms• Example: search task (WWW2012)Wednesday, May 15 Reactive Crowdsourcing 7Human Interaction ManagementSocialNetworksHumanComputationPlatformsQ&ASearch ExecutionEngineractionentQuery InterfaceSocialNetworksery AnswerSearch ExecutionEngineanInteractionanagementHumanQuery InterfaceLocalSocialNetworksQ&AQuery AnswerSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSocialNetworksQ&ACrowd-sourceplatformsQuery AnswerSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSocialNetworksQ&ACrowd-sourceplatformsQuery AnswerData Management SystemHuman AccessInterfaceRemote DataAccessLocal DataAccessSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSocialNetworksQ&ACrowd-sourceplatformsQuery AnswerSearch ExecutionEngineHumanInteractionManagementSE AccessInterfaceHumanAccessInterfaceQuery InterfaceLocalSourceAccessInterfaceSociaNetworQ&ACrowdsourceplatformQuery AnswerTaskHuman-EnhancedDataQueryResults
    8. 8. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessWednesday, May 15 Reactive Crowdsourcing 8I O
    9. 9. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessTaskSpecificationTask PlanningControlSpecificationWednesday, May 15 Reactive Crowdsourcing 9• Task Spec: task operations, objects, and performers Dimension Tables
    10. 10. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessTaskSpecificationTask PlanningControlSpecificationWednesday, May 15 Reactive Crowdsourcing 10• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution  Execution Table for task monitoring
    11. 11. • A simple abstract model• A task receives a list of input objects• Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects• A simple task design and deployment process, based on specific datastructures• created using model-driven transformations• driven by the task specificationThe Design ProcessTaskSpecificationTask PlanningControlSpecificationWednesday, May 15 Reactive Crowdsourcing 11• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution  Execution Table for task monitoring• Control Specification: task control policies  Control Mart
    12. 12. Task Specification_1/3Wednesday, May 15 Reactive Crowdsourcing 12• Operation Types: Choice, Like, Score, Tag, Classify, Order, …• Operation Parameters: e.g. classification classesTasktIDopTypecategoriesTask Specification Task Planning Control SpecificationTask Configurationt1Classify Rep/Dem 
    13. 13. Task Specification_2/3Wednesday, May 15 Reactive Crowdsourcing 13PoliticianclassifiedPartylastNamephotooID• Input Objects Schema: typed attributes• Output Attributes (according to task type)TasktIDopTypecategoriesTask Specification Task Planning Control SpecificationTask ConfigurationObjectSpecification o1 Obama http://…. ?????
    14. 14. Task Specification_3/3Wednesday, May 15 Reactive Crowdsourcing 14PoliticianclassifiedPartylastNamephotooIDTask ConfigurationObjectSpecificationPerformerSpecification• Execution platform(s)• Qualifications, etc.TasktIDopTypecategories PerformernamepIDplatformTask Specification Task Planning Control Specification p1 Alessandro Facebook
    15. 15. Task Planning_1/2• Organize the task in MicroTasks, and allocate input objects• μTaskObjectExecution  Designed for execution monitoring• Track performers responseclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDWednesday, May 15 Reactive Crowdsourcing 15PoliticianclassifiedPartylastNamephotooIDTasktIDopTypecategories PerformernamepIDplatformSplittingTask Specification Task Planning Control Specification mt1 O1 … …… … Facebook
    16. 16. Task Planning_2/2• Assign performers to MicroTasks on platforms• Pull: dynamic assignment (First come - First served / Choice of theperformer)• Push: static assignment (Performers’ priority / Performer matching)classifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDWednesday, May 15 Reactive Crowdsourcing 16PoliticianclassifiedPartylastNamephotooIDTasktIDopTypecategories PerformernamepIDplatformSplitting AssignmentTask Specification Task Planning Control Specification mt1 O1 P1 Republican00:00:01 00:00:10 Facebook
    17. 17. Control Specification_1/4Wednesday, May 15 Reactive Crowdsourcing 17Task Specification Task Planning Control Specification• Status Variable: tracking task and performers statusclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatus  Trusted/SpammerCreated/Planned/Closed 
    18. 18. Control Specification_2/4Wednesday, May 15 Reactive Crowdsourcing 18Task Specification Task Planning Control Specification• Object : tracking objects statusclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatusObjectControl #demoID#eval#rep#curAnswer
    19. 19. Control Specification_3/4Wednesday, May 15 Reactive Crowdsourcing 19Task Specification Task Planning Control Specification• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)PerformerControl #rightpID#eval#wrongclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatusObjectControl #demoID#eval#rep#curAnswer
    20. 20. Control Specification_4/4• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)• Task: tracking task status: closing @completion, re-planWednesday, May 15 Reactive Crowdsourcing 20TaskControl#compObjtID#compExecPerformerControl #rightpID#eval#wrongclassifiedPartyplatformμTaskObjectExecutionμtIDstartTsendTsoIDpIDPoliticianclassifiedPartylastNamephotooIDPerformernamepIDstatusplatformTasktIDopTypecategoriesstatusObjectControl #demoID#eval#rep#curAnswerTask Specification Task Planning Control Specification
    21. 21. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigmWednesday, May 15 Reactive Crowdsourcing 21
    22. 22. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigm• Events: data updates / timer• ROW-level granularity• OLD  before state of a row• NEW  after state of a rowWednesday, May 15 Reactive Crowdsourcing 22e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
    23. 23. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigm• Events: data updates / timer• ROW-level granularity• OLD  before state of a row• NEW  after state of a row• Condition: a predicate that must be satisfied (e.g. conditions oncontrol mart attributes)Wednesday, May 15 Reactive Crowdsourcing 23e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]c: NEW.ClassifiedParty == ’Republican’
    24. 24. Active Rules Language• Active rules are expressed on the previous datastructures• Event-Condition-Action paradigm• Events: data updates / timer• ROW-level granularity• OLD  before state of a row• NEW  after state of a row• Condition: a predicate that must be satisfied (e.g. conditions oncontrol mart attributes)• Actions: updates on data structures (e.g. change attributevalue, create new instances), special functions (e.g. replan)Wednesday, May 15 Reactive Crowdsourcing 24e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]c: NEW.ClassifiedParty == ’Republican’a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
    25. 25. Wednesday, May 15 Reactive Crowdsourcing 25e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDEvent
    26. 26. Wednesday, May 15 Reactive Crowdsourcing 26e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDCondition
    27. 27. Wednesday, May 15 Reactive Crowdsourcing 27e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDAction
    28. 28. Wednesday, May 15 Reactive Crowdsourcing 28e: UPDATE FOR ObjectControlc: (NEW.Rep== 2) or (NEW.Dem == 2)a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,SET TaskControl[tID==NEW.tID].compObj += 1Rule ExampleTaskControl#compObjtID PerformerControlμTaskObjectExecutionPolitician classifiedPartyoIDPerformerTaskObjectControl #demoID#rep#evaltIDAction
    29. 29. Rule Programming Best Practice• We define three classes of rulesWednesday, May 15 Reactive Crowdsourcing 29μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task
    30. 30. Rule Programming Best PracticeWednesday, May 15 Reactive Crowdsourcing 30• We define three classes of rules• Control rules: modifying the control tables;μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task
    31. 31. Rule Programming Best PracticeWednesday, May 15 Reactive Crowdsourcing 31• We define three classes of rules• Control rules: modifying the control tables;• Result rules: modifying the dimension tables (object, performer, task);μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task
    32. 32. Rule Programming Best PracticeWednesday, May 15 Reactive Crowdsourcing 32• Top-to-bottom, left-to-right, evaluation• Guaranteed termination• We define three classes of rules• Control rules: modifying the control tables;• Result rules: modifying the dimension tables (object, performer, task);μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task
    33. 33. Rule Programming Best Practice• We define three classes of rules• Control rules: modifying the control tables;• Result rules: modifying the dimension tables (object, performer, task);• Execution rules: modifying the execution table, either directly or through re-planningWednesday, May 15 Reactive Crowdsourcing 33μTaskObjectExecutionPerformerControlObjectControlTaskControlPolitician Performer Task• Termination must be proven (Rule precedence graph has cycles)
    34. 34. Experimental Evaluation• GOAL: demonstrate the flexibility and expressive powerof reactive crowdsourcing• 3 experiments, focused on Italian politicians• Parties: Human Computation  affiliation classification• Law: Game With a Purpose  guess the convicted politician• Order: Pure Game  hot or not• 1 week (November 2012)• 284 distinct performers• Recruited through public mailing lists and social networksannouncements• 3500 Micro TasksWednesday, May 15 Reactive Crowdsourcing 34
    35. 35. Politician Affiliation• Given the picture and name of a politician, specify his/her politicalaffiliation• No time limit• Performers are encouraged to look up online• 2 set of rules• Majority Evaluation• Spammer DetectionWednesday, May 15 Reactive Crowdsourcing 35
    36. 36. Results – Majority Evaluation_1/3Wednesday, May 15 Reactive Crowdsourcing 3630 object; object redundancy = 9;Final object classification as simple majority after 7 evaluations
    37. 37. Results - Majority Evaluation_2/3Wednesday, May 15 Reactive Crowdsourcing 37Majority @7Early Majority @3 R @7-27% executions-18% precision%ofCompl.Objects00.10.20.30.40.50.60.70.80.91.0Precision00.10.20.30.40.50.60.70.8#Executions0 10 20 30 40 50 60 70 80 90Final object classification as total majority after 3 evaluationsOtherwise, re-plan of 4 additional evaluations. Then simple majority at 7
    38. 38. Results - Majority Evaluation_3/3Wednesday, May 15 Reactive Crowdsourcing 38Majority @7Early Majority @3 R @7Majority @3 R @5 R @7-23% executions+26% precision+50% precision%ofCompl.Objects00.10.20.30.40.50.60.70.80.91.0Precision00.10.20.30.40.50.60.70.8#Executions0 10 20 30 40 50 60 70 80 90Final object classification as total majority after 3 evaluationsOtherwise, simple majority at 5 or at 7 (with replan)
    39. 39. Results – Spammer Detection_1/2Wednesday, May 15 Reactive Crowdsourcing 39New rule for spammer detection without ground truthPerformer correctness on final majority. Spammer if > 50% wrong classificationsMajority @3 R @5 R @7Majority @3 R @5 R @7 - Spammer Detection+46% executions+1.5% precision%ofCompl.Objects00.10.20.30.40.50.60.70.80.91.0Precision00.10.20.30.40.50.60.70.8#Executions0 10 20 30 40 50 60 70 80 90
    40. 40. A Short CrowdSearcher DemoWednesday, May 15 Reactive Crowdsourcing 40
    41. 41. Summary• Results• An integrated framework for crowdsourcing task design and control• Well-structured control rules with some guarantees of termination• Support for cross-platform crowd interoperability• A working prototype  crowdsearcher.search-computing.org• Forthcoming• Exploitation of interoperability• Expertise finding• Dynamic planning• Integration with other social-networks and human computationplatformsWednesday, May 15 Reactive Crowdsourcing 41
    42. 42. QUESTIONS?Wednesday, May 15 Reactive Crowdsourcing 42

    ×