V Jornadas eMadrid sobre “Educación Digital”. Roberto Centeno, Universidad Nacional de Educación a Distancia: Mecanismos de reputación en MOOCs. 2015-06-30
What is the Martin Trust Center for MIT Entrepreneurship & Why Is it So Awesome?
Similar to V Jornadas eMadrid sobre “Educación Digital”. Roberto Centeno, Universidad Nacional de Educación a Distancia: Mecanismos de reputación en MOOCs
Recommender System _Module 1_Introduction to Recommender System.pptxSatyam Sharma
Similar to V Jornadas eMadrid sobre “Educación Digital”. Roberto Centeno, Universidad Nacional de Educación a Distancia: Mecanismos de reputación en MOOCs (20)
V Jornadas eMadrid sobre “Educación Digital”. Roberto Centeno, Universidad Nacional de Educación a Distancia: Mecanismos de reputación en MOOCs
1. On the inaccuracy of numerical ratings:
A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto Centeno
rcenteno@lsi.uned.es
Dpto. Lenguajes y Sistemas Informáticos
UNED
5. Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation
6. many types: professional links, friendships, purchases, ...
complex: dynamism, complexity of the social structure, many
nodes (users, entities, ..)
how can we identify and locate appropriate entities/services to
consume? (more and more available information online)
Not enough experience so.. online review systems a (Yelp,
Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation
7. many types: professional links, friendships, purchases, ...
complex: dynamism, complexity of the social structure, many
nodes (users, entities, ..)
how can we identify and locate appropriate entities/services to
consume? (more and more available information online)
Not enough experience so.. online review systems a (Yelp,
Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation
Trust
Reputation
opinions of third parties
Confidence
local experiences
8. many types
complex
nodes (users, entities, ..)
how can we identify and locate appropriate entities/services to
consume?
Not enough experience so..
Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation
Trust
Reputation
opinions of third parties
Confidence
local experiences
9. objective: extract reputation of entities (users, objects, …)
how: gathering and aggregating opinions
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
14. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
Capturing preferences through numerical opinions
15. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
Capturing preferences through numerical opinions
16. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
PROBLEM 1
‣DIFICULT TO MAP PREFERENCES INTO
NUMERICAL OPINIONS
‣SUBJECTIVITY!!!
Capturing preferences through numerical opinions
17. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Capturing preferences through numerical opinions
18. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
PROBLEM 2
BIAS PROBLEMS!!!
Capturing preferences through numerical opinions
19. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Die Hard 3
Gone with the wind 0
Ben-Hur 4
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Capturing preferences through numerical opinions
20. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Die Hard 3
Gone with the wind 0
Ben-Hur 4
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Capturing preferences through numerical opinions
21. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Die Hard 0.2
Gone with the wind 0.1
Ben-Hur 1.0
Capturing preferences through numerical opinions
22. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Die Hard 0.2
Gone with the wind 0.1
Ben-Hur 1.0
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Capturing preferences through numerical opinions
23. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
24. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies by users
25. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies by users
clearly biased to positive ratings
26. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies given by top critics
27. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies given by top critics
not influenced and biased by others’ ratings
28. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies given by top critics
not influenced and biased by others’ ratings
PROBLEM CONFIRMATION
potential bias problems when mapping
opinions onto numerical values, reputation
rankings may vary; and likely to cause
differences between the true quality of an
entity and its rating aggregated from opinions
33. 11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwise queries:
34. 11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwise queries:
FROM
Ben-Hur
[0..1]
[Awful, fairly bad, It’s OK,
Will enjoy, Must see]
Gone with
the wind
:
15. Ben-Hur 4.3
:
23. Gone with the wind 4.1
:
35. 11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwise queries:
FROM
Ben-Hur
[0..1]
[Awful, fairly bad, It’s OK,
Will enjoy, Must see]
Gone with
the wind
:
15. Ben-Hur 4.3
:
23. Gone with the wind 4.1
:
TO Which movie do you prefer, Ben-Hur
or Gone with the wind?
:
15. Ben-Hur
:
23. Gone with the wind
:
easier for users to state opinions when the queries compare objects in a
pairwise fashion…
“… between these two objects, which one do you prefer?”
36. 12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Reputation (opinions aggregation) as an iterative process based on …
37. 12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (opinions aggregation) as an iterative process based on …
38. 12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (opinions aggregation) as an iterative process based on …
A
B
C
D
A
D
D
Match: pairwise comparison between two
entities
Dynamics: every match sent to a set of
users that reply to the query
Policies:
‣ Entity selection
‣Tournament schedule
‣ Users selection
‣Winner determination
39. 13
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (II)
Policies
Entities selection: which pair of entities should I select to be
compared?
‣ Mixed: current ranking vs new objects (ExploitationVs Exploration)
‣ Random
‣ Domain-dependent: objects with no information/fuzzy positions
Tournament schedule: how to initialize the tournament
‣ Random schedule (iterative process)
40. 14
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (III)
Policies
Users selection: who receives the queries (matches)?
‣ Random selection
‣Clustering of users by their preferences (representative users)
‣ Using (social) network properties: degree distribution, centrality of
nodes, …
Winner determination: how to decide which entity wins in a match
‣Voting procedures: preference replies from users count as votes
‣ Alternatives: absolute majority / full agreement (voting protocols)
‣ If there is no winner, no object gets through the next round
41. 15
3. From opinion ratings to pairwise queries: PWRM
Comparative aggregation: from matches to a ranking
When:After each match, the ranking is updated (iterative method)
How: Adaptation of a method for aggregating partial pairwise
comparison results into a ranking (Negahban et al., 2012)
‣Ranking approximation = random walk on G (weighted graph):
‣ An edge <ei,ej> if the pair has already been compared
‣The weights define the outcome of the comparisons
‣ Random walk uses a transition matrix P where:
‣ It moves from state ei to state ej with probability equal to the
chance that entity ej is preferred over entity ei
‣ Under these conditions, a vector w is a valid stationary distribution
for matrix P (wT
t+1 = wT · P)
‣ w defines the scores for each entity => ranking
42. 16
3. From opinion ratings to pairwise queries: PWRM
PWRM’s iterative process for building a reputation ranking
Require: a social network G = (U, E, LU , LE)
Require: a subset of E0
✓ E entities to be evaluated
1: for t 2 time do
2: Ei EntitiesSelectionPolicy.selectEntitiesToEvaluate(E0
)
3: KTEi
scheduleTournament(Ei)
4: for m 2 matches(KTEi ) do
5: nb UsersSelectionPolicy.getUsersToAsk(U)
6: send(m, nb)
7: votes receive()
8: winner WinnerDeterminationPolicy.getWinner(votes)
9: Ri AggregationMechanism.updateRanking(m, winner)
10: setWinnerNextRound(winner, KTE0 )
11: end for
12: end for
13: return Ri where E0
are ranked by their estimated reputation
47. 19
4.Towards ranking resources in MOOCs
Applying PWRM into MOOCs
‣MOOCs modeled as a Social Network (Online Review Systems)
‣Apply PWRM for ranking learning resources in MOOCs
‣Allowing users (students/teachers) to find the best resources
‣Formalize a MOOC from a peer based system point of view
Idea:
Let M = hU, R, LR, LU i be a MOOC, where:
• U = {u1, . . . , un} is a set of users (teachers or students);
• R = {r1, . . . , rm} is the set of learning resources uploaded in the course;
• LR = {hui, rji/ui 2 U; rj 2 R} is the set of links among users and re-
sources, representing that user ui has uploaded the resource rj in the
course;
• LU = {huk, rmi/uk 2 U; rm 2 R} is the set of links also between users
and resources representing that user uk has used the resource rm.
48. 20
4.Towards ranking resources in MOOCs
PWRM as a function
‣ Objective: to query users about resources to give their opinions in order to build
a global ranking of resources regarding their reputation, so..
M = hU, R, LR, LU , ranki
• rank : R0
⇥ O ! {1, . . . , |R0
|} is a function in charge of defining a total
ordering (ranking) over a subset of resources R0
2 R, taking into account
the set of opinions O given by users;
• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rji
and representing a pairwise query sent to a set of users participating in
the MOOC, where learning resources ri and rj are compared.
49. 20
4.Towards ranking resources in MOOCs
PWRM as a function
‣ Objective: to query users about resources to give their opinions in order to build
a global ranking of resources regarding their reputation, so..
M = hU, R, LR, LU , ranki
• rank : R0
⇥ O ! {1, . . . , |R0
|} is a function in charge of defining a total
ordering (ranking) over a subset of resources R0
2 R, taking into account
the set of opinions O given by users;
• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rji
and representing a pairwise query sent to a set of users participating in
the MOOC, where learning resources ri and rj are compared.
rank = PWRM
50. 21
4.Towards ranking resources in MOOCs
PWRM algorithm in MOOCs
Require: a MOOC M = hU, R, LR, LU , ranki
Require: a subset of R0
✓ R of learning resources to be ranked
1: for t 2 time do
2: Ri ResourcesSelectionPolicy.selectResourcesToEvaluate(R0
)
3: KTRi
scheduleTournament(Ri)
4: for m 2 matches(KTRi ) do
5: Ui UsersSelectionPolicy.getUsersToAsk(U)
6: send(m, Ui)
7: Oi ReceiveOpinions()
8: winner WinnerDeterminationPolicy.getWinner(votes, Oi)
9: Ranki AggregationMechanism.updateRanking(Oi, winner)
10: promoteResourceWinnerToNextRound(winner, KTRi )
11: end for
12: end for
13: return Ranki where the subset R0
of learning resources are ranked by their
reputation
51. 22
4.Towards ranking resources in MOOCs
PWRM algorithm in MOOCs: Policies
‣ Resource selection policy:
- resources clustered regarding their typology (e.g. videos, recorded class…)
- regarding the number of opinions received by each resource (lowest/
highest number)
- opinions in terms of the result of each match (matches with tight results)
‣ User selection policy:
- taking advantage of the underlying structure generated by interactions
between users and resources
‣Winner determination policy:
- voting theory: simple majority, complete agreement, …
53. 24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy to manipulate
‣ bias problems due to difficulty/subjectivity to map opinions into numerical values
Our Approach: PWRM
(1) based on comparative
(2) preference aggregation in reputation rankings (iterative process - tournaments)
(3) applied to MOOCs (ranking learning resources)
5. Conclusions & Future work
54. 24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy to manipulate
‣ bias problems
Our Approach: PWRM
(1) based on comparative opinions, elicited through pairwise preference request
(2) preference aggregation in reputation rankings (iterative process - tournaments)
(3) applied to MOOCs (ranking learning resources)
5. Conclusions & Future work
55. 25
Future work
5. Conclusions & Future work
‣ Adding social network properties:
- cluster users, centrality, betweenness, …
‣ Partial cooperative users:
- incentive mechanisms fostering cooperation (“what do you think users
prefer,A or B?”)
‣ Reputation of MOOCs:
- resources = courses, finding opinions in other opinions sites: twitter,
Facebook, forums, etc..
‣ Individual recommendation:
- resources/courses: from global reputation ranking to individual
recommendations
57. On the inaccuracy of numerical ratings:
A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto Centeno
rcenteno@lsi.uned.es
Dpto. Lenguajes y Sistemas Informáticos
UNED