Reactive crowdsourcing

1. REACTIVE CROWDSOURCING Alessandro Bozzonab Marco Brambillaa Stefano Ceria Andrea Mauria aPolitecnico di Milano Dipartimento di Elettronica, Informazione e BioIngegneria bDelft University Of Technology Department of Software And Computer Technology
2. Crowd Control is tough… • There are several aspects that makes crowd engineering complicated • Task design, planning, assignment • Workers discovery, assessment, engagement Wednesday, May 15 Reactive Crowdsourcing 2 http://xkcd.com/1060/
3. Crowd Control is tough… Wednesday, May 15 Reactive Crowdsourcing 3 • There are several aspects that makes crowd engineering complicated • Task design, planning, assignment • Workers discovery, assessment, engagement • Goal: taming the crowd • Cost • Time • Quality
4. Crowd Control is tough… Wednesday, May 15 Reactive Crowdsourcing 4 • There are several aspects that makes crowd engineering complicated • Task design, planning, assignment • Workers discovery, assessment, engagement • Goal: taming the crowd • Cost • Time • Quality • Motivation! • Need for higher level abstractions and tools • CONTROL as first-class citizen
5. Reactive Crowdsourcing • A conceptual framework for modeling crowdsourcing computations and control requirements • Task Design • Reactive Control Design • Active Rule programming framework • Declarative rule language • A reactive execution environment for requirement enforcement and reactive execution • Based on the CrowdSearcher approach Wednesday, May 15 Reactive Crowdsourcing 5
6. Why Active Rules? • Crowdsourcing control typically focuses on task data • Execution results, agreement on truth value, workers performance • An active rule approach can provide • Ease of Use: control is easily expressible • Simple control data structures • Familiar formalism • Power: support to arbitrarily complex controls • Extensibility mechanisms • Automation: most active rules can be system-generated • Well-defined semantic • Flexibility: simple control variants have localized impact on the rules set • Control isolation Wednesday, May 15 Reactive Crowdsourcing 6
7. The CrowdSearcher Approach • Human-Enhanced data management with social networks and Q&A systems as crowdsourcing platforms • Example: search task (WWW2012) Wednesday, May 15 Reactive Crowdsourcing 7 Human Interaction Management Social Networks Human Computation Platforms Q&A Search Execution Engine raction ent Query Interface Social Networks ery Answer Search Execution Engine anInteraction anagement Human Query Interface Local Social Networks Q&A Query Answer Search Execution Engine HumanInteraction Management SE Access Interface Human Access Interface Query Interface Local Source Access Interface Social Networks Q&A Crowd- source platforms Query Answer Search Execution Engine HumanInteraction Management SE Access Interface Human Access Interface Query Interface Local Source Access Interface Social Networks Q&A Crowd- source platforms Query Answer Data Management System Human Access Interface Remote Data Access Local Data Access Search Execution Engine HumanInteraction Management SE Access Interface Human Access Interface Query Interface Local Source Access Interface Social Networks Q&A Crowd- source platforms Query Answer Search Execution Engine HumanInteraction Management SE Access Interface Human Access Interface Query Interface Local Source Access Interface Socia Networ Q&A Crowd source platform Query Answer Task Human-Enhanced Data Query Results
8. • A simple abstract model • A task receives a list of input objects • Performers execute one or more operations upon them • The task produces a list of crowd-manipulated objects • A simple task design and deployment process, based on specific data structures • created using model-driven transformations • driven by the task specification The Design Process Wednesday, May 15 Reactive Crowdsourcing 8 I O
9. • A simple abstract model • A task receives a list of input objects • Performers execute one or more operations upon them • The task produces a list of crowd-manipulated objects • A simple task design and deployment process, based on specific data structures • created using model-driven transformations • driven by the task specification The Design Process Task Specification Task Planning Control Specification Wednesday, May 15 Reactive Crowdsourcing 9 • Task Spec: task operations, objects, and performers Dimension Tables
10. • A simple abstract model • A task receives a list of input objects • Performers execute one or more operations upon them • The task produces a list of crowd-manipulated objects • A simple task design and deployment process, based on specific data structures • created using model-driven transformations • driven by the task specification The Design Process Task Specification Task Planning Control Specification Wednesday, May 15 Reactive Crowdsourcing 10 • Task Spec: task operations, objects, and performers Dimension Tables • Task Planning: work distribution  Execution Table for task monitoring
11. • A simple abstract model • A task receives a list of input objects • Performers execute one or more operations upon them • The task produces a list of crowd-manipulated objects • A simple task design and deployment process, based on specific data structures • created using model-driven transformations • driven by the task specification The Design Process Task Specification Task Planning Control Specification Wednesday, May 15 Reactive Crowdsourcing 11 • Task Spec: task operations, objects, and performers Dimension Tables • Task Planning: work distribution  Execution Table for task monitoring • Control Specification: task control policies  Control Mart
12. Task Specification_1/3 Wednesday, May 15 Reactive Crowdsourcing 12 • Operation Types: Choice, Like, Score, Tag, Classify, Order, … • Operation Parameters: e.g. classification classes Task tID opType categories Task Specification Task Planning Control Specification Task Configuration t1 Classify  Rep/Dem 
13. Task Specification_2/3 Wednesday, May 15 Reactive Crowdsourcing 13 Politician classifiedParty lastName photo oID • Input Objects Schema: typed attributes • Output Attributes (according to task type) Task tID opType categories Task Specification Task Planning Control Specification Task Configuration Object Specification  o1  Obama  http://….  ?????
14. Task Specification_3/3 Wednesday, May 15 Reactive Crowdsourcing 14 Politician classifiedParty lastName photo oID Task Configuration Object Specification Performer Specification • Execution platform(s) • Qualifications, etc. Task tID opType categories Performer name pID platform Task Specification Task Planning Control Specification  p1  Alessandro  Facebook
15. Task Planning_1/2 • Organize the task in MicroTasks, and allocate input objects • μTaskObjectExecution  Designed for execution monitoring • Track performers response classifiedPartyplatform μTaskObject Execution μtID startTs endTs oID pID Wednesday, May 15 Reactive Crowdsourcing 15 Politician classifiedParty lastName photo oID Task tID opType categories Performer name pID platform Splitting Task Specification Task Planning Control Specification  mt1  O1  …  … …  …  Facebook
16. Task Planning_2/2 • Assign performers to MicroTasks on platforms • Pull: dynamic assignment (First come - First served / Choice of the performer) • Push: static assignment (Performers’ priority / Performer matching) classifiedPartyplatform μTaskObject Execution μtID startTs endTs oID pID Wednesday, May 15 Reactive Crowdsourcing 16 Politician classifiedParty lastName photo oID Task tID opType categories Performer name pID platform Splitting Assignment Task Specification Task Planning Control Specification  mt1  O1  P1  Republican 00:00:01  00:00:10  Facebook
17. Control Specification_1/4 Wednesday, May 15 Reactive Crowdsourcing 17 Task Specification Task Planning Control Specification • Status Variable: tracking task and performers status classifiedPartyplatform μTaskObject Execution μtID startTs endTs oID pID Politician classifiedParty lastName photo oID Performer name pID status platformTask tID opType categories status  Trusted/SpammerCreated/Planned/Closed 
18. Control Specification_2/4 Wednesday, May 15 Reactive Crowdsourcing 18 Task Specification Task Planning Control Specification • Object : tracking objects status classifiedPartyplatform μTaskObject Execution μtID startTs endTs oID pID Politician classifiedParty lastName photo oID Performer name pID status platformTask tID opType categories status Object Control #dem oID #eval #rep #curAnswer
19. Control Specification_3/4 Wednesday, May 15 Reactive Crowdsourcing 19 Task Specification Task Planning Control Specification • Object : tracking object responses • Performer: tracking performer behavior (e.g. spammers) Performer Control #right pID #eval #wrong classifiedPartyplatform μTaskObject Execution μtID startTs endTs oID pID Politician classifiedParty lastName photo oID Performer name pID status platformTask tID opType categories status Object Control #dem oID #eval #rep #curAnswer
20. Control Specification_4/4 • Object : tracking object responses • Performer: tracking performer behavior (e.g. spammers) • Task: tracking task status: closing @completion, re-plan Wednesday, May 15 Reactive Crowdsourcing 20 Task Control#compObj tID #compExec Performer Control #right pID #eval #wrong classifiedPartyplatform μTaskObject Execution μtID startTs endTs oID pID Politician classifiedParty lastName photo oID Performer name pID status platformTask tID opType categories status Object Control #dem oID #eval #rep #curAnswer Task Specification Task Planning Control Specification
21. Active Rules Language • Active rules are expressed on the previous data structures • Event-Condition-Action paradigm Wednesday, May 15 Reactive Crowdsourcing 21
22. Active Rules Language • Active rules are expressed on the previous data structures • Event-Condition-Action paradigm • Events: data updates / timer • ROW-level granularity • OLD  before state of a row • NEW  after state of a row Wednesday, May 15 Reactive Crowdsourcing 22 e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
23. Active Rules Language • Active rules are expressed on the previous data structures • Event-Condition-Action paradigm • Events: data updates / timer • ROW-level granularity • OLD  before state of a row • NEW  after state of a row • Condition: a predicate that must be satisfied (e.g. conditions on control mart attributes) Wednesday, May 15 Reactive Crowdsourcing 23 e: UPDATE FOR μTaskObjectExecution[ClassifiedParty] c: NEW.ClassifiedParty == ’Republican’
24. Active Rules Language • Active rules are expressed on the previous data structures • Event-Condition-Action paradigm • Events: data updates / timer • ROW-level granularity • OLD  before state of a row • NEW  after state of a row • Condition: a predicate that must be satisfied (e.g. conditions on control mart attributes) • Actions: updates on data structures (e.g. change attribute value, create new instances), special functions (e.g. replan) Wednesday, May 15 Reactive Crowdsourcing 24 e: UPDATE FOR μTaskObjectExecution[ClassifiedParty] c: NEW.ClassifiedParty == ’Republican’ a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
25. Wednesday, May 15 Reactive Crowdsourcing 25 e: UPDATE FOR ObjectControl c: (NEW.Rep== 2) or (NEW.Dem == 2) a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer, SET TaskControl[tID==NEW.tID].compObj += 1 Rule Example Task Control#compObj tID Performer Control μTaskObject Execution Politician classifiedParty oID PerformerTask Object Control #dem oID #rep #eval tIDEvent
26. Wednesday, May 15 Reactive Crowdsourcing 26 e: UPDATE FOR ObjectControl c: (NEW.Rep== 2) or (NEW.Dem == 2) a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer, SET TaskControl[tID==NEW.tID].compObj += 1 Rule Example Task Control#compObj tID Performer Control μTaskObject Execution Politician classifiedParty oID PerformerTask Object Control #dem oID #rep #eval tID Condition
27. Wednesday, May 15 Reactive Crowdsourcing 27 e: UPDATE FOR ObjectControl c: (NEW.Rep== 2) or (NEW.Dem == 2) a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer, SET TaskControl[tID==NEW.tID].compObj += 1 Rule Example Task Control#compObj tID Performer Control μTaskObject Execution Politician classifiedParty oID PerformerTask Object Control #dem oID #rep #eval tID Action
28. Wednesday, May 15 Reactive Crowdsourcing 28 e: UPDATE FOR ObjectControl c: (NEW.Rep== 2) or (NEW.Dem == 2) a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer, SET TaskControl[tID==NEW.tID].compObj += 1 Rule Example Task Control#compObj tID Performer Control μTaskObject Execution Politician classifiedParty oID PerformerTask Object Control #dem oID #rep #eval tID Action
29. Rule Programming Best Practice • We define three classes of rules Wednesday, May 15 Reactive Crowdsourcing 29 μTaskObject Execution Performer Control Object Control Task Control Politician Performer Task
30. Rule Programming Best Practice Wednesday, May 15 Reactive Crowdsourcing 30 • We define three classes of rules • Control rules: modifying the control tables; μTaskObject Execution Performer Control Object Control Task Control Politician Performer Task
31. Rule Programming Best Practice Wednesday, May 15 Reactive Crowdsourcing 31 • We define three classes of rules • Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task); μTaskObject Execution Performer Control Object Control Task Control Politician Performer Task
32. Rule Programming Best Practice Wednesday, May 15 Reactive Crowdsourcing 32 • Top-to-bottom, left-to-right, evaluation • Guaranteed termination • We define three classes of rules • Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task); μTaskObject Execution Performer Control Object Control Task Control Politician Performer Task
33. Rule Programming Best Practice • We define three classes of rules • Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task); • Execution rules: modifying the execution table, either directly or through re-planning Wednesday, May 15 Reactive Crowdsourcing 33 μTaskObject Execution Performer Control Object Control Task Control Politician Performer Task • Termination must be proven (Rule precedence graph has cycles)
34. Experimental Evaluation • GOAL: demonstrate the flexibility and expressive power of reactive crowdsourcing • 3 experiments, focused on Italian politicians • Parties: Human Computation  affiliation classification • Law: Game With a Purpose  guess the convicted politician • Order: Pure Game  hot or not • 1 week (November 2012) • 284 distinct performers • Recruited through public mailing lists and social networks announcements • 3500 Micro Tasks Wednesday, May 15 Reactive Crowdsourcing 34
35. Politician Affiliation • Given the picture and name of a politician, specify his/her political affiliation • No time limit • Performers are encouraged to look up online • 2 set of rules • Majority Evaluation • Spammer Detection Wednesday, May 15 Reactive Crowdsourcing 35
36. Results – Majority Evaluation_1/3 Wednesday, May 15 Reactive Crowdsourcing 36 30 object; object redundancy = 9; Final object classification as simple majority after 7 evaluations
37. Results - Majority Evaluation_2/3 Wednesday, May 15 Reactive Crowdsourcing 37 Majority @7 Early Majority @3 R @7 -27% executions -18% precision %ofCompl.Objects 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Precision 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 #Executions 0 10 20 30 40 50 60 70 80 90 Final object classification as total majority after 3 evaluations Otherwise, re-plan of 4 additional evaluations. Then simple majority at 7
38. Results - Majority Evaluation_3/3 Wednesday, May 15 Reactive Crowdsourcing 38 Majority @7 Early Majority @3 R @7 Majority @3 R @5 R @7 -23% executions +26% precision +50% precision %ofCompl.Objects 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Precision 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 #Executions 0 10 20 30 40 50 60 70 80 90 Final object classification as total majority after 3 evaluations Otherwise, simple majority at 5 or at 7 (with replan)
39. Results – Spammer Detection_1/2 Wednesday, May 15 Reactive Crowdsourcing 39 New rule for spammer detection without ground truth Performer correctness on final majority. Spammer if > 50% wrong classifications Majority @3 R @5 R @7 Majority @3 R @5 R @7 - Spammer Detection +46% executions +1.5% precision %ofCompl.Objects 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Precision 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 #Executions 0 10 20 30 40 50 60 70 80 90
40. A Short CrowdSearcher Demo Wednesday, May 15 Reactive Crowdsourcing 40
41. Summary • Results • An integrated framework for crowdsourcing task design and control • Well-structured control rules with some guarantees of termination • Support for cross-platform crowd interoperability • A working prototype  crowdsearcher.search-computing.org • Forthcoming • Exploitation of interoperability • Expertise finding • Dynamic planning • Integration with other social-networks and human computation platforms Wednesday, May 15 Reactive Crowdsourcing 41
42. QUESTIONS? Wednesday, May 15 Reactive Crowdsourcing 42

Editor's Notes

Good afternoonI’m very happy to be here today and discuss with you our work, named reactive crowdsourcing

But first allow me to contextualize a bit its scope. I’m sure we are all very familiar with crowdsourcing and human computation, which is a based on a very neat idea: organize humans to accomplish tasks. Unfortunately, or luckily for us , such organization is a complex matter. When designing systems and experiments, there are several things to consider: how do you split your tasks in microtask, which are the best performers for it, how do you scout them, etc.

It turns out, our ultimate goal is controlling the crowd. We would like to drive people behavior in order satisfy some constraints, which are typically monetary, qualitative, or temporal. Indeed, a lot of work has been devoted to devise algorithms able to minimize one of those costs, and a lot of effort has been recently put in the creation of frameworks and algorithm for crowd management.However, we observed that existing solutions (e.g. crowddb, deco, and others) provide limited and veryspecific control features. We advocate crowd control to be at the very center of task design. And we also advocate the need for methods for the systematicdesign of complex control strategies based on the state of tasks, results and performers.

This paper wepropose a conceptual framework and a reactive execution environment for modelling and controlling crowdsourcing computationsWit the ultimate goal of minimizing the effort required for control specification, we propose:a simple task design processA rule specification language,whose properties (e.g., termination) can be easily proved in the context of a well-organized computational framework

But why we decided to go for an approach based on active rules?The choice stemmed from the observation that crowdsourcing control is typically driven by data, like the status of the HIT executions, the worker performance, the current agreement on the truth value of some object. Therefore, it came almost natural for us to turn on a data-driven approach, that proven very effective for the definition of control in several contexts, including database systemsActive rules are actually relatively easy to use, when expressed on well-define data structures. They allow the definition of arbitrary complex control logic, most of which can be easily automated thanks to a well-defined syntax and semantic. Also, they allow for a great flexibilty, since changes in the control logic of the application can be well-isolated into localized changes of the rule set

This work capitalizes on the results of our www2012 paper, where we an approach for data management that integrates SN, QA and traditional HC platform for the execution of human computation and crowdsourcing tasks.This work specifically addressed crowd-enhanced search, a now very popular trend. However, the approach we proposed can be very easily generalized to any kind of human-enhanced data management system, and we believe this crowd-interoperability can be exploited in very interesting ways. LOCAL SOURCE: sorgentidatilocalisfruttate dal Search Execution Engine, magariaccedutedallo Human Interaction Management per configurare / gestirei task. La suaesistenza e’ accessoriarispettoaglialtri, e codificainformazioni applicative specificheICONE DI DX, DALL’ALTO a SX(social networks) Facebook, Twitter, Google +(Q&A systems) StackOverlflow, Yahoo Answers, Quora(HC Platforms) Freebase, Amazon Mechanical Turk, ODesk

The design process we propose is based on a very simple data-driven abstract model for task execution: data in, workers do stuff, data outCoherently with this vision, the task design and deployment process is based upon specific data structures, created in an automated way trough model driven transformations. Unfortunately we don’t have time to go too deep in this matter, so I refer you to the paper for further details.

The process composes of three simple steps. The first, task specification, is devoted to the definition of basic task information, such as the data-driven operations to perform (e.g. classify a picture, add new instances of a given object, etc.), the shape of the actual objects to evaluate, and the charactersticis of the targeted set of performers. All those aspect are encoded in what we call Dimension Tables

Then, the second step is devoted to task planning, i.e. specify how works should be partitioned and distributed among performers. Those planning aspects are encoded into an Execution Table, specifically devoted to task monitoring

Finally, the developer must specify the control logic for the task, and it does so by defining active rules upon control-specific data structures contained in what we call the Control Mart.

To clarify the process, let us show visually how the supporting data structure are created in a simple case study: a picture classification task, where the picture contain picture of politicians, and the task is to specify to which party they belong. We will use color codes to identify data structures types, and we highlight in bold those attributes which depend on the specifically designed tasks. All the other are, somehow, standard. The first dimension table is the Task table, which contains info about the performed operations (a classification, in this case), and the categories to which politicians can be assigned (e.g. republican and democtrats)

Then, we have a dimension table for the actual objects of the task, in this case, politicians. Notice that the classifiedPartyattribute is actually produced by the crowd at the end of the task

Finally, a Performer dimension table represents the population of performer currently available for task execution. The table can be pre-filled, if the performers are known in advance, or empty, in case of openly available tasks.

The second activity of the process deals with the planning of the task and it also composes of two phases. In the first phase, suited algorithms will decide such things as how which objects should appear in each MicroTask? (e.g. ground truth assessment), how many objects in each MicroTask, how often an object should appear in MicroTasks, etc. TheμTaskObjectExecution table keeps track of this organization, and it contains one tuple for each object/microTask association. A performer, when executing a microTask, will fill in the data value required by the specified operation (e.g. the party of a given politician)

The second phase of task planning deals with the assignment of microTasks to performers. This can be done according to several policies (e.g. pull or push). In the example, the assignment is performed in a pull fashion, and attributes are given value on performer arrival

The third part of the process is devoted to the specification of control. Please notice here that I added to an addition color code to denote attributes in the Dimension Tables which serve task control purposes. So for instance, a performer can be classified as trusted or spammer. A task as “created/planned/closed” and so on

Bin addition to status variable, control is defined upon the Control Mart, which composes of three data structures. The first one, called ObjectControl contains control variable related to the object of the task, like the number of evaluation it received, the number of classifications it received for each class, and the current truth value.

The second one, called PerformerControl contains control variable related to the performer of the task, like the number of evaluation she performerd, and the number of times she provided a right or wrong answer (assuming the existance of a ground truth)

Finally, the thirdone, called TaskControl contains control variable related to the task, like the number total number of objects currently evaluated, or the number of executed hits.

The previous data structure provide in a very simple yet complete way the control variables that are needed to define the task control policies. But how can control be specified? We rely on a language based on the classic ECA paradigm

Where events are updated on the data structure values. We decided for a row-level update granularity, so to easily track the before and after states of rows. In the example, the rule triggers when the ClassifiedParty attribute of a tuple in the μTaskObjectExecution table changes

Conditions are expressed as conditions on data attributes (e.g., the value specified by the performer)

And actions are updated performed on the same, or other data structures. Such updates can be done directly or trough special functions, devoted to such operations as replanning’Of course there is no time to show the syntax of the language, but you can find more on the paper.

The second rule is a bit more complex, and it is used to assess the truth value of an object trough majority voting. For instance, here we assume that as soon as a Politician gets 2 evaluation as Rep or Dem, the object can be deemed as completed. triggers when the ObjectControl table updates

It is quite known that active rule programming can be rather subtle and unstable, as the behavior of a set of rules may change dramatically as a consequence of small changes in the rules To simply, and better control rule programming we devise three classes of rules which, as I will show soon, have interesting properties

The first class are named control rules, and are meant to modify control tables. Arrows represents rules triggering on a table (the source of the arrow) and affecting another table (the destination). As you can see, not all the possible source-target couple are adimissible, and I will explain soon why.

The second class of rules modify the dimension tables, and are the one devoted to changing the status of the main task entities (e.g. setting a perforomer as spammer when she makes too many bad classifications)

Note that, since we suggest a very well-defined top-to-bottom, left-to-right semantic, no cycles are allowed, and therefore rules are guardanteed to terminate. Those cycles in the object control – performer control etc. still bases on precise rules also on the attributes of the table

Finally, execution rules are responsivble for the modification of the execution table, and are therefore responsible for modifying the set?distribution/assignment of the currently defined microtasks. Those rules introduce cycles and, therefore, might cause unconvergence (and termination must be verified)

To demonstrate the flexibily and expressive power of reactive crowdsourcing we advised the experiments, conducted during one week of November 2012. We developed three very different scenario, all programmed with our approach. Unfortunately we don’t have enough time to describe them all, so we focus on just one. A classification task similar to the one used as example in the presentation

Control result precisionspammer detection

Reactive crowdsourcing

You are reading a preview.

Reactive crowdsourcing

More Related Content

More from Alessandro Bozzon

Featured

Related Books

Related Audiobooks