Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Methods and techniques-comm (3/6)


Published on

ISWC 2010: Tutoria: ten ways to make your semantic app addictive

  • Be the first to comment

Methods and techniques-comm (3/6)

  1. 1. Methods and techniques to analyze and design incentivized semantic applications Roberta Cuel, University of Trento, IT and Markus Rohde, University of Siegen, DE ISWC 2010
  2. 2. 1/29/2015 2 Why is my app not as successful as Facebook… 1/29/2015 2 • Motivation and incentives – Reciprocity – Reputation – Competition – Altruism – Self-esteem – Fun – Money • This session is about the methods and techniques you can use – To study your scenario prior to application design – To evaluate your incentives strategy and adjust your application
  3. 3. Overview • Basic notions and definitions • Methods and techniques for analysis, design and evaluation • Case studies 1/29/2015 3
  4. 4. BASIC NOTIONS AND DEFINITIONS Basic notions and definitions Methods and techniques for analysis, design and evaluation Case studies 1/29/2015 4
  5. 5. Starting with motivation… • Basic tenets of organizational behavior – Performance : f(ability*motivation) 1/29/2015 5
  6. 6. Motivation and incentives • Intrinsic vs extrinsic motivations • Incentives are ‘rewards’ assigned to performer to make him/her commit – Can be totally uncorrelated to the nature of the task – Need to be compliant with the values and beliefs of the acting person 1/29/2015 61/29/2015 6
  7. 7. Example: DBPedia The DBpedia community collaborates using the following tools – Mailing lists, Facebook group – Blog, quality assurance and bug tracking – Development • “The framework is easily extended and we offer the possibility to do this yourself, increasing the likelihood of your desired data being included in the next DBpedia release.” – DBpedia consulting (money) 1/29/2015 7
  8. 8. 1/29/2015 8 Typology of motivations Motivations Internal (embedded in structure, e.g., task, tools) External (additional to structure, external re-inforcements) Intrinsic (predispositioned in person, e.g., drives, needs, desires ) Fun, joy, gaming, interest, satisfaction, self-actualization, self-re- inforcement Social appreciation, reputation, love, trust, social capital, community support Extrinsic (additional to personal predispositions, extern re- inforcements ) Usability, sociability, Design-for-fun, curiosity, community-building support Material/financial capital, money, rewards, prices, medals, credit points Structure Person Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)
  9. 9. METHODS AND TECHNIQUES FOR ANALYSIS, DESIGN AND EVALUATION Basic notions and definitions Methods and techniques for analysis, design and evaluation Case studies 1/29/2015 9
  10. 10. Overview of methods and techniques • To analyze your scenario and design your application – Game theory, mechanism design – Participatory design and end-user development • To evaluate an incentives strategy in its context – Interviews and questionnaires and field experiments – Heuristic evaluation, guideline reviews – Cognitive and pluralistic walk-throughs – Participatory/User evaluation 1/29/2015 101/29/2015 10
  11. 11. Game theory • Game theory is a formal way to analyze interaction among a number of rational agents who behave strategically – The rational agents: players involved in the situation (best choice) – A number of players: more than one – Rationality/payoffs: what are the players’ preferences over the outcomes of the game – The interactions: one player’s behavior affects another – The rules: who moves when, what do they know, what can they do – The outcomes: what is the outcome of the game (for each move) 1/29/2015 11
  12. 12. Mechanism design • Mechanism design is about how to translate game theory in effective behavior – To design rules such that a desired set of outcomes happens – Alignment of interests between parties and production of maximum social welfare • Relevant variables – Structural and personal motivations – Goal and nature of good produced – Tasks and skills required – Social structure 1/29/2015 12
  13. 13. Analysis matrix • A multidimensional tool that allows for an analysis of a particular task with respect to the variables relevant for mechanism design 1/29/2015 13 Goal Tasks Social structure Nature of good produced Skill variety/level Communication level Variety of Hierarchy- neutral Private good Skilled ability Participation level Specificity of Hierarchical Public good Competence Clearness level Identification with Common resource Club good
  14. 14. 1/29/2015 14 The matrix in practice: semantic annotation • Goal of the annotation or ontology population exercise • Task, or more typically, an ordered collection of tasks into which the annotation exercise can be broken down • Social structure, a stylized and simplified set of social relationships among the subjects participating in the exercise • Nature of good, a stylized description, in game-theoretical terms, of the relationship between what good is produced and who consumes it • Required skills of the agents to complete the annotation task 1/29/2015 14
  15. 15. The matrix in practice: crowd-sourcing of ontology evolution • Goal: communication and participation level – Coordination (free-open) • Interests are already aligned (free riding) • Private and public benefit coincide • Task: variety vs specificity • Social structure: hierarchy neutral (large vs small groups) • Nature of good: public good non-rival (use does not limit use of others), non-exclusive (open access) • Skill variety/level: Skilled ability 1/29/2015 151/29/2015 15
  16. 16. Participatory Design (PD) • To develop software for the participation of end-users, we propose a participatory way of designing these software tools, integrating potential users by participatory design methods • Participatory Design – Improvement of the participation of workers in software development processes – The cooperation between software developers and end-users – Participate in IT development projects as experts of their own work 1/29/2015 16
  17. 17. 1/29/2015 17 Methods for PD • Participatory-design projects combine – Design-by-doing methods – PD workshops – Scenarios – Different forms of prototyping (mockups, rapid prototypes) – Work organization games – Ethnographic methods
  18. 18. 1/29/2015 18 End-User Development (EUD) • EUD is about designing highly flexible systems that enable users to participate during the use of the system by adapting and modifying the tools according to their needs/preferences • Goal: Empower end-users to develop and adapt systems themselves by designing these systems to be easy to – Understand – Learn – Use – Teach
  19. 19. 1/29/2015 19 How to design an incentivized application • Ideally: field  desk  lab  field • Analyze the domain and find yourselves in the matrixes – Find the relevant point of that situation (goal and tasks) – Focus on a small group of individuals (social structure) – Analyze their motivation (internal/external intrinsic/extrinsic) – Analyze the other relevant variables (nature of good being produced, kill variety/level) • Design a simplest possible model that can effectively support contributors • Test and get feedback • Fine-tune the experiment and add other elements
  20. 20. 1/29/2015 20 Methods • Domain observations (second-hand data) • Ethnography or qualitative face-to-face interviews • Questionnaires • Observations with selected individuals • Quantitative analysis (data collections) • Usability-design methods 1/29/2015 20
  21. 21. CASE STUDIES Basic notions and definitions Methods and techniques for analysis, design and evaluation Case studies 1/29/2015 21
  22. 22. Examples from our project • Telefónica Investigación y Desarrollo (TID - Spain): Knowledge Intranet Platform • Pepper’s Ghost (PGP-UK): Virtual Games/Worlds • Seekda! (Austria): Web Services Search Portal 1/29/2015 22
  23. 23. 1/29/2015 23 Field work: TID • Domain analysis – Site visit, semi-structured, qualitative interviews • Communication processes • Existing usage practices and problems • Existing tools/solutions • Semantic annotation solutions – Tape recording, transcription – Data analysis per ex-post categorization • Focus group discussion – Usability lab tests – Expert walkthroughs
  24. 24. 1/29/2015 24 Field work: TID (2) 1/29/2015 24 • Find TID in the matrix – Goal: communication and participation (contribute to the knowledge portal) – Social structure (various structures co-exist) • Strongly hierarchical organization (control is an issue) • working groups and community of experts – Nature of good: public good vs. private, club goods – Skill variety/level: Skilled ability (knowledge workers) – Motivations: fun, visibility, reputation, promotion, money
  25. 25. 1/29/2015 25 Field work: PGP • Domain analysis – Data analysis (Benchmark: Galaxyzoo, Moonzoo) – Usability lab tests and expert walkthroughs – Collect data on users such as logs files • Find PGP in the matrix – Goal: participation level – Task: specific – Social structure: hierarchy neutral – Nature of good: public good non-rival (use does not limit use of others), non-exclusive (open access) – Skill variety/level: non skilled ability
  26. 26. 1/29/2015 26 Field work: SEEKDA • Domain analysis – Interviews and workshops – Replicating Telefonica model • Find SEEKDA in the matrix: – Goal: communication and participation level – Task: variety vs. specificity – Social structure: hierarchy neutral (mainly) – Nature of good: public good non-rival (use does not limit use of others), exclusive, or club good – Skill variety/level: Skilled ability
  27. 27. Thank you Any question? 1/29/2015 27