Methods and techniques (2/6)


Published on

ISWC 2010: TUTORIAL: Ten Ways to Make your Semantic App Addictive

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • 2 is ex ante
    3 is ex post
  • Nash and travelling salesman
  • Methods and techniques (2/6)

    1. 1. Methods and techniques to analyze and design incentivized semantic applications Roberta Cuel, University of Trento, IT and Markus Rohde, University of Siegen, DE ISWC 2010
    2. 2. 1/29/2015 2 Why is my app not as successful as Facebook… 1/29/2015 2 • Motivation and incentives – Reciprocity – Reputation – Competition – Altruism – Self-esteem – Fun – Money • This session is about the methods and techniques you can use – To study your scenario prior to application design – To evaluate your incentives strategy and adjust your application
    3. 3. Overview • Basic notions and definitions • Methods and techniques for analysis, design and evaluation • Case studies 1/29/2015 3
    4. 4. BASIC NOTIONS AND DEFINITIONS Basic notions and definitions Methods and techniques for analysis, design and evaluation Case studies 1/29/2015 4
    5. 5. Starting with motivation… • Basic tenets of organizational behavior – Performance : f(ability*motivation) 1/29/2015 5
    6. 6. Motivation and incentives • Intrinsic vs extrinsic motivations • Incentives are ‘rewards’ assigned to performer to make him/her commit – Can be totally uncorrelated to the nature of the task – Need to be compliant with the values and beliefs of the acting person 1/29/2015 61/29/2015 6
    7. 7. Example: DBPedia The DBpedia community collaborates using the following tools – Mailing lists, Facebook group – Blog, quality assurance and bug tracking – Development • “The framework is easily extended and we offer the possibility to do this yourself, increasing the likelihood of your desired data being included in the next DBpedia release.” – DBpedia consulting (money) 1/29/2015 7
    8. 8. 1/29/2015 8 Typology of motivations Motivations Internal (embedded in structure, e.g., task, tools) External (additional to structure, external re-inforcements) Intrinsic (predispositioned in person, e.g., drives, needs, desires ) Fun, joy, gaming, interest, satisfaction, self-actualization, self-re- inforcement Social appreciation, reputation, love, trust, social capital, community support Extrinsic (additional to personal predispositions, extern re- inforcements ) Usability, sociability, Design-for-fun, curiosity, community-building support Material/financial capital, money, rewards, prices, medals, credit points Structure Person Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)
    9. 9. METHODS AND TECHNIQUES FOR ANALYSIS, DESIGN AND EVALUATION Basic notions and definitions Methods and techniques for analysis, design and evaluation Case studies 1/29/2015 9
    10. 10. Overview of methods and techniques • To analyze your scenario and design your application – Game theory, mechanism design – Participatory design and end-user development • To evaluate an incentives strategy in its context – Interviews and questionnaires and field experiments – Heuristic evaluation, guideline reviews – Cognitive and pluralistic walk-throughs – Participatory/User evaluation 1/29/2015 101/29/2015 10
    11. 11. Game theory • Game theory is a formal way to analyze interaction among a number of rational agents who behave strategically – The rational agents: players involved in the situation (best choice) – A number of players: more than one – Rationality/payoffs: what are the players’ preferences over the outcomes of the game – The interactions: one player’s behavior affects another – The rules: who moves when, what do they know, what can they do – The outcomes: what is the outcome of the game (for each move) 1/29/2015 11
    12. 12. Mechanism design • Mechanism design is about how to translate game theory in effective behavior – To design rules such that a desired set of outcomes happens – Alignment of interests between parties and production of maximum social welfare • Relevant variables – Structural and personal motivations – Goal and nature of good produced – Tasks and skills required – Social structure 1/29/2015 12
    13. 13. Analysis matrix • A multidimensional tool that allows for an analysis of a particular task with respect to the variables relevant for mechanism design 1/29/2015 13 Goal Tasks Social structure Nature of good produced Skill variety/level Communication level Variety of Hierarchy- neutral Private good Skilled ability Participation level Specificity of Hierarchical Public good Competence Clearness level Identification with Common resource Club good
    14. 14. 1/29/2015 14 The matrix in practice: semantic annotation • Goal of the annotation or ontology population exercise • Task, or more typically, an ordered collection of tasks into which the annotation exercise can be broken down • Social structure, a stylized and simplified set of social relationships among the subjects participating in the exercise • Nature of good, a stylized description, in game-theoretical terms, of the relationship between what good is produced and who consumes it • Required skills of the agents to complete the annotation task 1/29/2015 14
    15. 15. The matrix in practice: crowd-sourcing of ontology evolution • Goal: communication and participation level – Coordination (free-open) • Interests are already aligned (free riding) • Private and public benefit coincide • Task: variety vs specificity • Social structure: hierarchy neutral (large vs small groups) • Nature of good: public good non-rival (use does not limit use of others), non-exclusive (open access) • Skill variety/level: Skilled ability 1/29/2015 151/29/2015 15
    16. 16. Participatory Design (PD) • To develop software for the participation of end-users, we propose a participatory way of designing these software tools, integrating potential users by participatory design methods • Participatory Design – Improvement of the participation of workers in software development processes – The cooperation between software developers and end-users – Participate in IT development projects as experts of their own work 1/29/2015 16
    17. 17. 1/29/2015 17 Methods for PD • Participatory-design projects combine – Design-by-doing methods – PD workshops – Scenarios – Different forms of prototyping (mockups, rapid prototypes) – Work organization games – Ethnographic methods
    18. 18. 1/29/2015 18 End-User Development (EUD) • EUD is about designing highly flexible systems that enable users to participate during the use of the system by adapting and modifying the tools according to their needs/preferences • Goal: Empower end-users to develop and adapt systems themselves by designing these systems to be easy to – Understand – Learn – Use – Teach
    19. 19. 1/29/2015 19 How to design an incentivized application • Ideally: field  desk  lab  field • Analyze the domain and find yourselves in the matrixes – Find the relevant point of that situation (goal and tasks) – Focus on a small group of individuals (social structure) – Analyze their motivation (internal/external intrinsic/extrinsic) – Analyze the other relevant variables (nature of good being produced, kill variety/level) • Design a simplest possible model that can effectively support contributors • Test and get feedback • Fine-tune the experiment and add other elements
    20. 20. 1/29/2015 20 Methods • Domain observations (second-hand data) • Ethnography or qualitative face-to-face interviews • Questionnaires • Observations with selected individuals • Quantitative analysis (data collections) • Usability-design methods 1/29/2015 20
    21. 21. CASE STUDIES Basic notions and definitions Methods and techniques for analysis, design and evaluation Case studies 1/29/2015 21
    22. 22. Examples from our project • Telefónica Investigación y Desarrollo (TID - Spain): Knowledge Intranet Platform • Pepper’s Ghost (PGP-UK): Virtual Games/Worlds • Seekda! (Austria): Web Services Search Portal 1/29/2015 22
    23. 23. 1/29/2015 23 Field work: TID • Domain analysis – Site visit, semi-structured, qualitative interviews • Communication processes • Existing usage practices and problems • Existing tools/solutions • Semantic annotation solutions – Tape recording, transcription – Data analysis per ex-post categorization • Focus group discussion – Usability lab tests – Expert walkthroughs
    24. 24. 1/29/2015 24 Field work: TID (2) 1/29/2015 24 • Find TID in the matrix – Goal: communication and participation (contribute to the knowledge portal) – Social structure (various structures co-exist) • Strongly hierarchical organization (control is an issue) • working groups and community of experts – Nature of good: public good vs. private, club goods – Skill variety/level: Skilled ability (knowledge workers) – Motivations: fun, visibility, reputation, promotion, money
    25. 25. 1/29/2015 25 Field work: PGP • Domain analysis – Data analysis (Benchmark: Galaxyzoo, Moonzoo) – Usability lab tests and expert walkthroughs – Collect data on users such as logs files • Find PGP in the matrix – Goal: participation level – Task: specific – Social structure: hierarchy neutral – Nature of good: public good non-rival (use does not limit use of others), non-exclusive (open access) – Skill variety/level: non skilled ability
    26. 26. 1/29/2015 26 Field work: SEEKDA • Domain analysis – Interviews and workshops – Replicating Telefonica model • Find SEEKDA in the matrix: – Goal: communication and participation level – Task: variety vs. specificity – Social structure: hierarchy neutral (mainly) – Nature of good: public good non-rival (use does not limit use of others), exclusive, or club good – Skill variety/level: Skilled ability
    27. 27. Thank you Any question? 1/29/2015 27