21st International Conference on Cooperative Information Systems (CoopIS’13)
September 11-13, 2013, Graz, Austria
Ognjen S...
2 CoopIS’13
Outline
 Incentives in Crowdsourcing today and tomorrow
 Problems with evaluating incentives
 Our approach
...
3 CoopIS’13
Incentives & Rewards
• Incentives
Stimulate (motivate) or discourage
certain worker activities before the
actu...
4 CoopIS’13
Evolution of Crowdsourcing
Conventional workflows
• formal description
• structured execution
• predefined rol...
5 CoopIS’13
 Incentive schemes can be built
by composing and customizing
well-known incentive elements.
 Programmable in...
6 CoopIS’13
 Need a systematic approach in designing and evaluating
incentive schemes before deployment on real systems.
...
7 CoopIS’13
1) Mathematical incentive models
(e.g., principal-agent theory, game theory)
2) Empirical evaluation
Existing ...
8 CoopIS’13
3) Experimental evaluation
(e.g., on micro-task platforms, with students, volunteers)
Existing Evaluation Appr...
9 CoopIS’13
 Offer methodology for quickly selecting, composing and
customizing existing incentive mechanisms.
 Roughly ...
10 CoopIS’13
Simulation Model of Incentive Mechanism
Decision-making function fa considers:
1) the statistically or intent...
11 CoopIS’13
 True power of incentives  composition of incentive mechanisms
 Two basic operators on incentive mechanism...
12 CoopIS’13
Simulation Methodology
1) Defining domain-specific meta-model by extending core meta-model
2) Capturing worke...
13 CoopIS’13
Simulation Methodology
1) Defining domain-specific meta-model by extending core meta-model
2) Capturing worke...
14 CoopIS’13
Simulation Methodology
1) Defining domain-specific meta-model by extending core meta-model
2) Capturing worke...
15 CoopIS’13
 Simulation model of a realistic scenario, inspired by
– Citizen-driven traffic reporting (SmartJourney – Ab...
16 CoopIS’13
 Composite Incentive Schemes (CIS) evaluated:
 3 Experiments:
– Exp1: Compare impact of CIS1, CIS2, and CIS...
17 CoopIS’13
 CIS3 most reasonable to use. Can cope well with up to
20% of malicious workers.
Evaluation Results – Example
18 CoopIS’13
 Presented a methodology for modeling and simulating
incentives in crowdsourcing environments.
 Advantages:...
21st International Conference on Cooperative Information Systems (CoopIS’13)
September 11-13, 2013, Graz, Austria
Ognjen S...
Upcoming SlideShare
Loading in …5
×

Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcing Environments

654 views

Published on

Conventional incentive mechanisms were designed for business environments
involving static business processes and a limited number of actors. They
are not easily applicable to crowdsourcing and other social computing platforms,
characterized by dynamic collaboration patterns and high numbers of actors, because
the effects of incentives in these environments are often unforeseen and
more costly than in a well-controlled environment of a traditional company.

In this paper we investigate how to design and calibrate incentive schemes for
crowdsourcing processes by simulating joint effects of a combination of different
participation and incentive mechanisms applied to a working crowd. More
specifically, we present a simulation model of incentive schemes and evaluate it
on a relevant real-world scenario. We show how the model is used to simulate
different compositions of incentive mechanisms and model parameters, and how
these choices influence the costs on the system provider side and the number of
malicious workers.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
654
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcing Environments

  1. 1. 21st International Conference on Cooperative Information Systems (CoopIS’13) September 11-13, 2013, Graz, Austria Ognjen Scekic, Christoph Dorn, Schahram Dustdar Distributed Systems Group Vienna University of Technology http://dsg.tuwien.ac.at Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcing
  2. 2. 2 CoopIS’13 Outline  Incentives in Crowdsourcing today and tomorrow  Problems with evaluating incentives  Our approach – Simulation model – Simulation methodology & tools  Real-world scenario example  Conclusion and Outlook
  3. 3. 3 CoopIS’13 Incentives & Rewards • Incentives Stimulate (motivate) or discourage certain worker activities before the actual execution of those activities. • Rewards Any kind of recompense for worthy services rendered or retribution for wrongdoing exerted upon workers after the completion of activity. • Incentive Mechanism A plan (rule) for assigning rewards.
  4. 4. 4 CoopIS’13 Evolution of Crowdsourcing Conventional workflows • formal description • structured execution • predefined roles and activities • complex tasks Crowdsourcing • simple tasks • anonymous replaceable actors • short, unstructured interactions • No interaction/collaboration among actors + = Socio-technical Collective Adaptive Systems • ad-hoc assembled teams • complex tasks • social orchestration • indirect adaptation
  5. 5. 5 CoopIS’13  Incentive schemes can be built by composing and customizing well-known incentive elements.  Programmable incentive management  Portable, reusable, scalable incentives.  Problem: Composition  evaluation complexity – How to prevent malicious workers? – How to anticipate free riding, multitasking, tragedy of the commons? – How to assess appropriate reward amounts Modeling Incentives – Problems
  6. 6. 6 CoopIS’13  Need a systematic approach in designing and evaluating incentive schemes before deployment on real systems.  How to select, customize and evaluate appropriate atomic incentive mechanisms and how to compose them for a given crowdsourcing scenario?  We present: – Simulation model of incentive mechanism – Modeling and simulation methodology for approximate estimation of the composition of incentive mechanisms. Contributions
  7. 7. 7 CoopIS’13 1) Mathematical incentive models (e.g., principal-agent theory, game theory) 2) Empirical evaluation Existing Evaluation Approaches PRO CON precise and reliable related to particular collaboration patterns, cannot handle unforeseen runtime changes PRO CON good for evaluating simple existing incentives and behavioral responses impossibility to isolate particular mechanisms and their effects, or causes of behavior in complex cases; platform limitations (e.g., communication channels, predefined incentives and metrics);
  8. 8. 8 CoopIS’13 3) Experimental evaluation (e.g., on micro-task platforms, with students, volunteers) Existing Evaluation Approaches PRO CON controlled environment and reproducible setups platform limitations (e.g., communication channels, predefined incentives and metrics); limited monetary funds may derive skewed results; working with people inherently willing or forced to perform work may derive skewed results
  9. 9. 9 CoopIS’13  Offer methodology for quickly selecting, composing and customizing existing incentive mechanisms.  Roughly predicting effects of composition in dynamic crowdsourcing environments.  Model and simulation parameters can be changed dynamically, allowing quick testing of different incentive scheme setups and behavioral responses at low cost.  Modeling of incentives and responses of arbitrary complexity.  We do not devise novel nor optimal incentive mechanisms! Our approach
  10. 10. 10 CoopIS’13 Simulation Model of Incentive Mechanism Decision-making function fa considers: 1) the statistically or intentionally determined personality of the worker St 2) historical records of past actions {S0, … , St-1} 3) authority’s view of worker’s performance Mj 4) performance of other workers {Mk}, k ≠ j 5) promised rewards R Incentive mechanism IM considers: 1) current state of artifact Ki 2) the current performance metrics of worker Mj 3) output from another incentive mechanism returning the same type of reward R′ak
  11. 11. 11 CoopIS’13  True power of incentives  composition of incentive mechanisms  Two basic operators on incentive mechanisms: – addition (+) and functional composition () – operate on common metrics – final metrics’ values advertised to workers represent the promised reward  Major difficulty in designing successful incentive strategies is to properly choose performance metrics, basic incentive mechanisms and the proper composition. Simulating Complex Incentive Strategies
  12. 12. 12 CoopIS’13 Simulation Methodology 1) Defining domain-specific meta-model by extending core meta-model 2) Capturing worker behavioral patterns and reward calculation into executable model 3) Defining scenarios, assumptions, and configurations for individual simulation runs 4) Evaluating and interpreting simulation results 1 2 3 4
  13. 13. 13 CoopIS’13 Simulation Methodology 1) Defining domain-specific meta-model by extending core meta-model 2) Capturing worker behavioral patterns and reward calculation into executable model 3) Defining scenarios, assumptions, and configurations for individual simulation runs 4) Evaluating and interpreting simulation results 1 2 DomainPro Designer *www.quandarypeak.com
  14. 14. 14 CoopIS’13 Simulation Methodology 1) Defining domain-specific meta-model by extending core meta-model 2) Capturing worker behavioral patterns and reward calculation into executable model 3) Defining scenarios, assumptions, and configurations for individual simulation runs 4) Evaluating and interpreting simulation results 3 DomainPro Analyst 4 *www.quandarypeak.com
  15. 15. 15 CoopIS’13  Simulation model of a realistic scenario, inspired by – Citizen-driven traffic reporting (SmartJourney – Aberdeen) – Crowdsourced software testing  Generalized scenario: – Entities: Authority, Workers, Situations, Reports – Activities: Submit, Improve, Rate, Report duplicates – Metrics: Reputation (for trustworthiness), Points (for productivity) – Incentives: Three incentive mechanisms:  IM1 – fixed amounts of points per activity  IM2 – points related with report quality  IM3 – users are assigned reputation based on past activities Evaluation
  16. 16. 16 CoopIS’13  Composite Incentive Schemes (CIS) evaluated:  3 Experiments: – Exp1: Compare impact of CIS1, CIS2, and CIS3 on authority cost. – Exp2: Analyze effects of having too few or too many workers per situation – Exp3: Evaluate effects of malicious workers (0-50%) on cost. Evaluation
  17. 17. 17 CoopIS’13  CIS3 most reasonable to use. Can cope well with up to 20% of malicious workers. Evaluation Results – Example
  18. 18. 18 CoopIS’13  Presented a methodology for modeling and simulating incentives in crowdsourcing environments.  Advantages: – Useful for quick, runtime, approximate evaluations of different compositions of incentive mechanisms.  Drawbacks: – Inconclusive results. See: Advantages  Future Work: – Devise suitable models for more complex socio-technical systems. Conclusion & Outlook
  19. 19. 21st International Conference on Cooperative Information Systems (CoopIS’13) September 11-13, 2013, Graz, Austria Ognjen Scekic, Christoph Dorn, Schahram Dustdar Distributed Systems Group Vienna University of Technology http://dsg.tuwien.ac.at Thank you! Questions? 21st International Conference on Cooperative Information Systems (CoopIS’13) September 11-13, 2013, Graz, Austria

×