Presentation for Doctoral Consortium at UMAP'11

883 views

Published on

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Presentation for Doctoral Consortium at UMAP'11

  1. 1. A reputation system to model expertise in online communities<br />Doctoral Consortium UMAP2011<br />+ some social mechanisms<br />
  2. 2. From reputation to social mechanisms<br />Peer-based learning in online communities<br />Challenges<br />Motivation<br />Quality<br />Initial research direction: reputation<br />
  3. 3. Reputationprinciples<br />Linkedwith trust<br />Reciprocity & Goodbehaviour<br />Someoneelse’s story about me<br />Linkedwithidentity; long-lived<br />A currency, a resource<br />Narrative, dynamic<br />Based on claims, transactions, opinion, rating, endorsements, …<br />Based on indirect information<br />Context of community<br />Contextual<br />Trade-off between trust & privacy<br />Regular assessment of reputationquality<br />Windley, Phillip J., Kevin Tew, and Devlin Daley. A Framework for Building Reputation Systems. <br />In WWW2007.<br />
  4. 4. Main challenge<br />What:<br />Aggregating and interpreting meaningfulinteractions between & among objects* in online communities <br />Why:<br />to give insight in the value on the object level based on user feedback and usage.<br />* Objects are people and information objects<br />
  5. 5. So what did I do?<br />Literature<br />Learning theories, knowledge management<br />Trust and reputation<br />Look at different successful reputation systems/techniques<br />Google PageRank, eBay, StackOverflow, Guru, etc.<br />How? <br />value flow<br />context integration<br />sustainability<br />Hennis, T., Lukosch, S., & Veen, W. (2011). Reputation in peer-based learning environments. In O. C. Santos & J. G. Boticario (Eds.), Educational Recommender Systems and Technologies. IGI.<br />
  6. 6. Value flow<br />Source objects<br />People, organizations, …<br />Target objects<br />Blog posts, articles, …<br />Claims (value statements)<br />Rates, links, recommendations, etc.<br />Implicit, explicit<br />(Farmer & Glass, 2009)<br />
  7. 7. Context integration<br />
  8. 8. Sustainability, i.e. StackOverflow<br />
  9. 9.
  10. 10.
  11. 11. Concept reputation model<br />
  12. 12. Concept reputation model – 1/6claim weight<br />claim weight ~ expressiveness of the claim type <br /> (i.e. rating versus click)<br />
  13. 13. Concept reputation model – 2/6target object weight<br />claim weight ~ expressiveness of the claim type (i.e. rating versus click)<br />target object weight ~ importance of the contribution type <br /> (i.e. article versus comment)<br />
  14. 14. Concept reputation model – 3/6affiliated keyword weight<br />apples (0.8)<br />pears (0.2)<br />claim weight ~ expressiveness of the claim type (i.e. rating versus click)<br />target object weight ~ importance of the contribution type (i.e. article versus comment)<br />affiliated keyword weight ~ expressiveness of tag about the target object<br />
  15. 15. Concept reputation model – 4/6source object weight = authority<br />match keywords with reputation!<br />apples (64)<br />pears (33)<br />kiwis (0)<br />apples (0.6)<br />pears (0.2)<br />kiwis (0.2)<br />claim weight ~ expressiveness of the claim type (i.e. rating versus click)<br />target object weight ~ importance of the contribution type (i.e. article versus comment)<br />affiliated keyword weight ~ expressiveness of tag about the target object<br />source object weight = (source object’s reputation for keyword) / (global rep. value for that keyword)<br />DIFFERENT WEIGHTS FOR DIFFERENT KEYWORDS<br />
  16. 16. Concept reputation model – 5/6claim value<br />apples (64)<br />pears (33)<br />kiwis (0)<br />apples (0.6)<br />pears (0.2)<br />kiwis (0.2)<br />claim weight ~ expressiveness of the claim type (i.e. rating versus click)<br />target object weight ~ importance of the contribution type (i.e. article versus comment)<br />affiliated keyword weight ~ expressiveness of tag about the target object<br />source object weight = (source object’s reputation for keyword) / (global rep. value for that keyword)<br />claim value = rating (implicit/explicit), i.e. 4/5 stars<br />
  17. 17. Concept reputation model – 6/6claim value<br />apples (64)<br />pears (33)<br />kiwis (0)<br />apples (0.6)<br />pears (0.2)<br />kiwis (0.2)<br />claim weight ~ expressiveness of the claim type (i.e. rating versus click)<br />target object weight ~ importance of the contribution type (i.e. article versus comment)<br />affiliated keyword weight ~ expressiveness of tag about the target object<br />source object weight = (source object’s reputation for keyword) / (global rep. value for that keyword)<br />claim value = rating (implicit/explicit)<br /> 3 claims (one for each affiliate keyword)<br />
  18. 18. Why is this a useful approach?<br />Target object weight<br />Affiliate keyword weight <br />Claim for keyword k<br />Claim weight<br />Authority<br />Claim value (rating)<br />
  19. 19. Why is this a useful approach?<br />Target object weight<br />Affiliate keyword weight <br />Account for:<br /><ul><li>Several relevant context factors, such as authority
  20. 20. Extensible & configurable
  21. 21. including other weights and metrics (i.e. trust value, network centrality, etc.)
  22. 22. integrating formal ontologies
  23. 23. taking into account all relevant interactions
  24. 24. Starting point for the design of such a system
  25. 25. Rich profiles</li></ul>Requirements<br /><ul><li>Sufficient interactions & contributions
  26. 26. Rather large distributed online network or community</li></ul>Claim for keyword k<br />Claim weight<br />Authority<br />Claim value (rating)<br />
  27. 27. Reputation clouds (object model)<br />
  28. 28. Peer Support Community<br />Trying to get funding (50k) for first prototype<br />Context <br />Blackboard apps & Tags<br />Comment & Rating functionality<br />Application of reputation model<br />Source: Teacher or Student<br />Target: comment, answer<br />Claims: like, page visit, follow<br />Reuse of reputation<br />Award, status, social comparison, gaming mechanisms (competition)<br />
  29. 29. Contextualized support: content and user support<br />
  30. 30. But…<br />Not smart to bet on 1 horse, when 3 already dropped out of the race…<br />
  31. 31. So.. New scope (since 2 weeks)<br />Social mechanisms to design incentive structures to support informal learning in online communities<br />Hennis, T. A., & Kolfschoten, G. L. (2010). Understanding Social Mechanisms in Online Communities. In G. D. Vreede (Ed.), Group Decision and Negotiation 2010. Delft, the Netherlands.<br />Hennis, T. A., & Lukosch, H. (2011). Social Mechanisms to Motivate Learning with Remote Experiments - Design choices to foster online peer-based learning. CSEDU 2011.<br />Veen, W., Staalduinen, J.-P. V., & Hennis, T. A. (2010). Informal self-regulated learning in corporate organizations. In G. Dettori & D. Persico (Eds.), Fostering Self-regulated learning through ICTs. Genova, Italy: Institute for Educational Technologies Italyʼs National Research Council.<br />
  32. 32. Bouwman et al. (2007)<br />We argue that social software systems should trigger mechanisms that allow us to associate with or form social groups, whether online or in the real world.<br />Such mechanisms would acknowledge human motivations, like eagerness for exploration, curiosity, inquisitiveness, civilization, valuation of belonging, achieving self-realization, enjoying one-self.<br />Bouman, W., Hoogenboom, T., Jansen, R., Schoondorp, M., Bruin, B. de, & Huizing, A. (2007). The realm of sociality: notes on the design of social software. Amsterdam.<br />
  33. 33. Research objectives<br />Designing incentives: which mechanism to apply when and how?<br />Design of processes<br />Supportive technologies<br />Focused on <br />informal learning<br />in organizations<br />during initial phase – startup phase<br />Cases<br />Philips Lighting<br />Mediamatic (various communities)<br />
  34. 34. Step 1 – improve list of mechanisms (literature)<br />Matching objectives<br />Organizational objectives<br />User models<br />Fit / Embedding in practice<br />Rhythm<br />Leadership and roles<br />Heterogeneity & Diversity<br />Learning & Networking<br />Reputation & Identity<br />Reciprocity & Feedback<br />Common Ground & Privacy<br />Self-efficacy & Social comparison<br />Autonomy? Empowerment?<br />Curiosity & Provocation<br />IMPROVE<br />
  35. 35. Step 2 – Design & Evaluate<br />Philips Lighting<br />3 communities by December<br />Mediamatic Design team<br />Design & Test new things<br />Evaluate existing communities using <br /> the Anymeta platform<br />Qualitative<br />Quantitative  50+ small-medium sized online communities<br />Blogging communities<br />Storytelling<br />Event communities & Professional networks<br />
  36. 36. Rating & Reputation<br />Reciprocity & Feedback<br />Matching online and offline networks through RFID<br />Notifications & Activity<br />
  37. 37. Profiling & Identity<br />Interaction types ~ Motivation?<br />Personalization & Recommendations<br />
  38. 38. Keyword based<br />Recommend content, people, projects, …<br />
  39. 39.
  40. 40. Definitions<br />A social mechanism is a plausible hypothesis, or set of plausible hypotheses, that could be the explanation of some social phenomenon. <br />An incentive is any factor (financial or non-financial) that enables or motivates a particular course of action, or counts as a reason for preferring one choice to the alternatives.<br />
  41. 41. Overall picture<br />Typicalvaluejudgements/claims<br />Context parameters<br /> aggregateandinherit/inference<br />repute<br />value judgements:<br />use/rate/recommend<br />context parameters:<br />tag<br />contribute<br />
  42. 42. 1: Contribution“I write a blog post”<br />Knowledge (topic)<br />What kind of contribution?<br />What kind of topic?<br />contributions<br />Competencies (process)<br />What kind of action?<br />Which competencies involved?<br />
  43. 43. 2: Value + context (cont.)”people rate, comment, tag etc…”<br />value statement: <br /><ul><li>evaluate value  use/rate/recommend
  44. 44. contextualize  categorize/tag/embed</li></li></ul><li>Another dimension to reputation<br /> - not just value, according to others<br /> - but a contextualized value, assigned to<br /> 1. a concept or topic<br /> 2. a process or competency<br />Photography (4)<br />Ruby on Rails (40)<br />PHP (54)<br />88<br />3<br />17<br />Dynamically updated!<br />Project management (0.8)<br />Social skills (0.4)<br /> Writing skills (0.3)<br />Idea generation (0.3)<br />
  45. 45. Example: researcher<br />Value statements:<br />. IF (Impact Factor)<br />. citing <br />. rating/reviews<br />. linking<br />Context:<br />. author & user profiles<br />. citing article or blog<br />. journal<br />. content analysis<br />. keywords<br />---------------<br />--------------<br />-----------------<br />----------------<br />-------------<br />---------------<br />--------------<br />-----------------<br />----------------<br />-------------<br />DOI<br />IF<br />ORCID<br />reputation<br />citations<br />published_in<br />write<br />papers<br />---------------<br />--------------<br />-----------------<br />----------------<br />-------------<br />link<br />ratings<br />BlogRank<br />
  46. 46. StackOverflow dump<br />Analyze the concept of authority<br />Steps:<br />Define the sources, targets, claims, and weights<br />Sources: SO-users / Targets: Answers, Questions, Users / Claims: votes up/down<br />Context: keywords<br />Conduct analysis and compare the results with the traditional reputation<br />Improve algorithm<br />
  47. 47. <foods> <br /> <pizza title=“Deluxe Pizza”><br /> <name>The Deluxe</name><br /> <toppings><br /> <topping>peppers</topping><br /> <topping>pepperoni</topping><br /> <topping>mushrooms</topping><br /> <topping>cheese</topping><br /> <topping>tomato sauce</topping><br /> </toppings><br /> <price>7.99</price><br /> </pizza><br /></foods> <br />Research challenges<br />Reputationontologyforp2p learning<br />Formalize common “Sources”, “Claims”, and “Targets” in these communities<br />Combine formalontologiestodescribe topics and skills, anddynamicfolksonomiestodescribecontextual parameters<br />Reuse of reputation information<br />Open Standards & Search<br />Tool development<br />implement, test andimprovereputation system<br />

×