Download to read offline
Uploaded byThieme Hennis
Presentation for Doctoral Consortium at UMAP'11
The document discusses designing social mechanisms to support informal learning in online communities. It proposes improving a list of social mechanisms from literature and designing and evaluating mechanisms for two case studies. The overall goal is to design incentives using appropriate mechanisms to motivate learning based on organizational objectives and user models. The research will analyze reputation systems to model expertise, focusing on sources, claims, targets and weights to better understand authority in communities.
More Related Content
Similar to Presentation for Doctoral Consortium at UMAP'11
Presentation for Doctoral Consortium at UMAP'11
- 1. A reputation systemto model expertise in online communitiesDoctoral Consortium UMAP2011+ some social mechanisms
- 2. From reputation tosocial mechanismsPeer-based learning in online communitiesChallengesMotivationQualityInitial research direction: reputation
- 3. ReputationprinciplesLinkedwith trustReciprocity &GoodbehaviourSomeoneelse’s story about meLinkedwithidentity; long-livedA currency, a resourceNarrative, dynamicBased on claims, transactions, opinion, rating, endorsements, …Based on indirect informationContext of communityContextualTrade-off between trust & privacyRegular assessment of reputationqualityWindley, Phillip J., Kevin Tew, and Devlin Daley. A Framework for Building Reputation Systems. In WWW2007.
- 4. Main challengeWhat:Aggregating andinterpreting meaningfulinteractions between & among objects* in online communities Why:to give insight in the value on the object level based on user feedback and usage.* Objects are people and information objects
- 5. So what didI do?LiteratureLearning theories, knowledge managementTrust and reputationLook at different successful reputation systems/techniquesGoogle PageRank, eBay, StackOverflow, Guru, etc.How? value flowcontext integrationsustainabilityHennis, T., Lukosch, S., & Veen, W. (2011). Reputation in peer-based learning environments. In O. C. Santos & J. G. Boticario (Eds.), Educational Recommender Systems and Technologies. IGI.
- 6. Value flowSource objectsPeople,organizations, …Target objectsBlog posts, articles, …Claims (value statements)Rates, links, recommendations, etc.Implicit, explicit(Farmer & Glass, 2009)
- 7.
- 8.
- 11.
- 12. Concept reputation model– 1/6claim weightclaim weight ~ expressiveness of the claim type (i.e. rating versus click)
- 13. Concept reputation model– 2/6target object weightclaim weight ~ expressiveness of the claim type (i.e. rating versus click)target object weight ~ importance of the contribution type (i.e. article versus comment)
- 14. Concept reputation model– 3/6affiliated keyword weightapples (0.8)pears (0.2)claim weight ~ expressiveness of the claim type (i.e. rating versus click)target object weight ~ importance of the contribution type (i.e. article versus comment)affiliated keyword weight ~ expressiveness of tag about the target object
- 15. Concept reputation model– 4/6source object weight = authoritymatch keywords with reputation!apples (64)pears (33)kiwis (0)apples (0.6)pears (0.2)kiwis (0.2)claim weight ~ expressiveness of the claim type (i.e. rating versus click)target object weight ~ importance of the contribution type (i.e. article versus comment)affiliated keyword weight ~ expressiveness of tag about the target objectsource object weight = (source object’s reputation for keyword) / (global rep. value for that keyword)DIFFERENT WEIGHTS FOR DIFFERENT KEYWORDS
- 16. Concept reputation model– 5/6claim valueapples (64)pears (33)kiwis (0)apples (0.6)pears (0.2)kiwis (0.2)claim weight ~ expressiveness of the claim type (i.e. rating versus click)target object weight ~ importance of the contribution type (i.e. article versus comment)affiliated keyword weight ~ expressiveness of tag about the target objectsource object weight = (source object’s reputation for keyword) / (global rep. value for that keyword)claim value = rating (implicit/explicit), i.e. 4/5 stars
- 17. Concept reputation model– 6/6claim valueapples (64)pears (33)kiwis (0)apples (0.6)pears (0.2)kiwis (0.2)claim weight ~ expressiveness of the claim type (i.e. rating versus click)target object weight ~ importance of the contribution type (i.e. article versus comment)affiliated keyword weight ~ expressiveness of tag about the target objectsource object weight = (source object’s reputation for keyword) / (global rep. value for that keyword)claim value = rating (implicit/explicit) 3 claims (one for each affiliate keyword)
- 18. Why is thisa useful approach?Target object weightAffiliate keyword weight Claim for keyword kClaim weightAuthorityClaim value (rating)
- 19. Why is thisa useful approach?Target object weightAffiliate keyword weight Account for:Several relevant context factors, such as authority
- 20.
- 21. including other weightsand metrics (i.e. trust value, network centrality, etc.)
- 22.
- 23. taking into accountall relevant interactions
- 24. Starting point forthe design of such a system
- 25. Rich profilesRequirementsSufficient interactions& contributions
- 26. Rather large distributedonline network or communityClaim for keyword kClaim weightAuthorityClaim value (rating)
- 27.
- 28. Peer Support CommunityTryingto get funding (50k) for first prototypeContext Blackboard apps & TagsComment & Rating functionalityApplication of reputation modelSource: Teacher or StudentTarget: comment, answerClaims: like, page visit, followReuse of reputationAward, status, social comparison, gaming mechanisms (competition)
- 29. Contextualized support: contentand user support
- 30. But…Not smart tobet on 1 horse, when 3 already dropped out of the race…
- 31. So.. New scope(since 2 weeks)Social mechanisms to design incentive structures to support informal learning in online communitiesHennis, T. A., & Kolfschoten, G. L. (2010). Understanding Social Mechanisms in Online Communities. In G. D. Vreede (Ed.), Group Decision and Negotiation 2010. Delft, the Netherlands.Hennis, T. A., & Lukosch, H. (2011). Social Mechanisms to Motivate Learning with Remote Experiments - Design choices to foster online peer-based learning. CSEDU 2011.Veen, W., Staalduinen, J.-P. V., & Hennis, T. A. (2010). Informal self-regulated learning in corporate organizations. In G. Dettori & D. Persico (Eds.), Fostering Self-regulated learning through ICTs. Genova, Italy: Institute for Educational Technologies Italyʼs National Research Council.
- 32. Bouwman et al.(2007)We argue that social software systems should trigger mechanisms that allow us to associate with or form social groups, whether online or in the real world.Such mechanisms would acknowledge human motivations, like eagerness for exploration, curiosity, inquisitiveness, civilization, valuation of belonging, achieving self-realization, enjoying one-self.Bouman, W., Hoogenboom, T., Jansen, R., Schoondorp, M., Bruin, B. de, & Huizing, A. (2007). The realm of sociality: notes on the design of social software. Amsterdam.
- 33. Research objectivesDesigning incentives:which mechanism to apply when and how?Design of processesSupportive technologiesFocused on informal learningin organizationsduring initial phase – startup phaseCasesPhilips LightingMediamatic (various communities)
- 34. Step 1 –improve list of mechanisms (literature)Matching objectivesOrganizational objectivesUser modelsFit / Embedding in practiceRhythmLeadership and rolesHeterogeneity & DiversityLearning & NetworkingReputation & IdentityReciprocity & FeedbackCommon Ground & PrivacySelf-efficacy & Social comparisonAutonomy? Empowerment?Curiosity & ProvocationIMPROVE
- 35. Step 2 –Design & EvaluatePhilips Lighting3 communities by DecemberMediamatic Design teamDesign & Test new thingsEvaluate existing communities using the Anymeta platformQualitativeQuantitative 50+ small-medium sized online communitiesBlogging communitiesStorytellingEvent communities & Professional networks
- 36. Rating & ReputationReciprocity& FeedbackMatching online and offline networks through RFIDNotifications & Activity
- 37. Profiling & IdentityInteractiontypes ~ Motivation?Personalization & Recommendations
- 38. Keyword basedRecommend content,people, projects, …
- 40. DefinitionsA social mechanismis a plausible hypothesis, or set of plausible hypotheses, that could be the explanation of some social phenomenon. An incentive is any factor (financial or non-financial) that enables or motivates a particular course of action, or counts as a reason for preferring one choice to the alternatives.
- 41. Overall pictureTypicalvaluejudgements/claimsContext parametersaggregateandinherit/inferencereputevalue judgements:use/rate/recommendcontext parameters:tagcontribute
- 42. 1: Contribution“I writea blog post”Knowledge (topic)What kind of contribution?What kind of topic?contributionsCompetencies (process)What kind of action?Which competencies involved?
- 43. 2: Value +context (cont.)”people rate, comment, tag etc…”value statement: evaluate value use/rate/recommend
- 44. contextualize categorize/tag/embedAnotherdimension to reputation - not just value, according to others - but a contextualized value, assigned to 1. a concept or topic 2. a process or competencyPhotography (4)Ruby on Rails (40)PHP (54)88317Dynamically updated!Project management (0.8)Social skills (0.4) Writing skills (0.3)Idea generation (0.3)
- 45. Example: researcherValue statements:.IF (Impact Factor). citing . rating/reviews. linkingContext:. author & user profiles. citing article or blog. journal. content analysis. keywords------------------------------------------------------------------------------------------------------------------------------------------------------DOIIFORCIDreputationcitationspublished_inwritepapers---------------------------------------------------------------------------linkratingsBlogRank
- 46. StackOverflow dumpAnalyze theconcept of authoritySteps:Define the sources, targets, claims, and weightsSources: SO-users / Targets: Answers, Questions, Users / Claims: votes up/downContext: keywordsConduct analysis and compare the results with the traditional reputationImprove algorithm
- 47. <foods> <pizza title=“Deluxe Pizza”> <name>The Deluxe</name> <toppings> <topping>peppers</topping> <topping>pepperoni</topping> <topping>mushrooms</topping> <topping>cheese</topping> <topping>tomato sauce</topping> </toppings> <price>7.99</price> </pizza></foods> Research challengesReputationontologyforp2p learningFormalize common “Sources”, “Claims”, and “Targets” in these communitiesCombine formalontologiestodescribe topics and skills, anddynamicfolksonomiestodescribecontextual parametersReuse of reputation informationOpen Standards & SearchTool developmentimplement, test andimprovereputation system
Editor's Notes
- #2 I will discuss with you 2 things: what I did, and what I will do. Because they do not really align, I will try to go through the first part as fast as possible.
- #3 The research is based on a vision of education and learning that is lifelong, interactive, networked, and intrinsically related with flexible employment patterns. In this scenario, your online reputation is like a social currency and can lead to new job opportunities. The infrastructure for this scenario is not yet fully established.Learning is becoming a more social, interactive, lifelong activity taking place more and more on the Web.Educational institutions (as well as professional organizations) will move their formal education partially to the Web, meaning that they will host or support wikis and learning networks where learning takes place.Open CoursewareInstitutions and employers need (analytical) tools to assure quality and assess individuals in these online learning environments.Learners need tools to find the right resources and people who can assist them. Tutors need a reason to help others, other than direct extrinsic reward (money).Online communities can be effective learning environments, but there are issues concerned with Trust in people and content in online communitiesMotivation of people to contribute / add value (engagement)In centralized knowledge environments, we often see extrinsic reward systems and institutionalized quality assurance mechanisms to ensure enough quality and contributions. When there are no resources available to do this (such as in peer-based online learning environments), or when this proves to be inefficient use of resources, a more decentralized approach is needed. Successful online communities or marketplaces adopt a variety of social mechanisms and tools to ensure trust, quality and motivation. Reputation systems can cut at both edges; one the one hand, information about value and quality is generated for improved recommendations or search. On the other hand, this information is used to motivate individuals to add value/behave in a desired way.
- #5 Information about the perceived value of a contribution by a community can be used to recognize and reward this person (motivation) and to predict the quality/value of the person and inform future interactors (trust). For example, employers or colleagues in a large organization could be interested in the online reputation of individuals, when they need someone to do a certain task, consult an expert, or want to assemble a group for a project. Also, in more open and social learning environments, reputation can be used to match learners and tutors, or reward contributors of valuable content. These online reputation profiles are not (yet) widely trusted as a measurement of someone’s or something’s quality or value. Can reputation become an instrument for improving self-organization in peer-based learning environments?
- #6 So I looked into literature on trust and reputation systems, on peer-based learning,
- #8 So, on this picture we have a dentist acting as a car mechanic. We know the example: the transitivity of trust (and reputation) depends on the context. We can say, this is a great guy, but why, and when? We need to give the context of his greatness, otherwise it is useless information. I would say, the more specific the context, the better, because it is easier to infer from specific to generic than the other way around. And is it a static context that is restricted by the system, or dynamically added through tags?This is an important challenge in knowledge-intensive online communities: what is the context of value? What is value?
- #9 A nice example, and very relevant for communities of professionals, of how a reputation system contributes to the sustainability, is StackOverflow. As we see in this picture, the reputation of a user is establised based on questions and answers, and how other users vote for those contributions. The user then earns the tags of that question. Those tag-based profiles form the basis of the employment search database, where employers can look for qualified IT people.Sheet 4:SolutionA reputation system that can deal with the dynamics of information is StackOverflow. (plaatje Jon Skeet)So we developed a method that automates the modeling of reputation that is contextualized using keywords, similar to StackOverflow, but more generic so it can be adapted for any knowledge-sharing environment. In addition, it is better able to model relevant context factors, such as authority. (it works a bit like Google: links coming from pages with a lot of visitors carry more weight in the calculation of PageRank than links coming from not popular websites.)Structure of the argument:Collaborative online environments to share experiences, learn and collaborateusually face 2 main challenges: motivation of individuals to contribute quality of the contributionsReputation is a two-edged sword tackling both challenges at the same time: social recognition of highly valued contributions and peer-assessment of the shared information through implicit and explicit ratingEpistemology: what is knowledge? What is valuable knowledge?Context! Difficulty with knowledge-intensive organizations: information is continuously being added and created, and a static taxonomy will not suffice. Plaatje van taxonomy & tag cloud (beidenzijnnodig)In environments where new knowledge is created, you must be able to add keywords to information, to make it fit to your own context. These keywords cannot be known in advance. Hence, reputation (and trust) also depend on context: when your car broke down, you are interested in a mechanic, not a wine-expert.SolutionA reputation system that can deal with the dynamics of information is StackOverflow. (plaatje Jon Skeet)So we developed a method that automates the modeling of reputation that is contextualized using keywords, similar to StackOverflow, but more generic so it can be adapted for any knowledge-sharing environment. In addition, it is better able to model relevant context factors, such as authority. (it works a bit like Google: links coming from pages with a lot of visitors carry more weight in the calculation of PageRank than links coming from not popular websites.)
- #10 Here we see a profile of the number 1 user on the platform, the infamous Jon Skeet.
- #11 And what we see here, is I think very interesting, and relevant for the learning and education field. Because your StackOverflow reputation is in a way nothing else than a diploma showing your qualifications, even with grades! It might now only apply to some industry areas, including IT, but I think this mechanism is very important.
- #12 Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- #13 Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- #14 Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- #15 Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- #16 Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- #17 Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- #18 Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- #19 Wat u hierziet is eengeneriek model wat we hebbenontwikkelddatondersteuningbiedtaan het ontwerp van eenreputatiesysteem. Hierbijhouden we rekening met de genoemdecomplexiteitrond context en waardering, en gaan we uit van het online gedrag van individuen in eenleercommunity. Je brengtals het ware in kaart hoe mensenzichgedragen, en je kanook door bepaaldetypenbijdragenmeertewaarderen, ook het gedragsturen.
- #20 Wat u hierziet is eengeneriek model wat we hebbenontwikkelddatondersteuningbiedtaan het ontwerp van eenreputatiesysteem. Hierbijhouden we rekening met de genoemdecomplexiteitrond context en waardering, en gaan we uit van het online gedrag van individuen in eenleercommunity. Je brengtals het ware in kaart hoe mensenzichgedragen, en je kanook door bepaaldetypenbijdragenmeertewaarderen, ook het gedragsturen.
- #23 Users can see who is an expert only after the question has been asked (otherwise they will directly ask their colleague without posting the question: you want the interaction to take place online).
- #40 On StackOverflow, users do aggregate a specific reputation profile using the affiliate keyword method, BUT it does not consider authority. We can have a look what difference it will make to reputations of SO-users. Maybe the algorithm can be improved, because of recursive behavior or because the high-authority persons have too much influence on the reputation of low-reputation objects.