The document discusses a new reputation mechanism for MOOCs called Pairwise Preference Ranking Mechanism (PWRM) which addresses the inaccuracies of numerical ratings by using pairwise queries for preference elicitation. The traditional method of capturing opinions through numerical ratings is criticized for its subjectivity and potential biases, and PWRM proposes a proactive method that facilitates comparative aggregation of opinions in an iterative process. By modeling MOOCs as social networks, PWRM aims to provide a dynamic approach to ranking educational resources based on user preferences.

1. On the inaccuracy of numerical ratings:
A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto Centeno
rcenteno@lsi.uned.es
Dpto. Lenguajes y Sistemas Informáticos
UNED

2. Outline
1. Introduction
2. Motivation
3. From opinion ratings to pairwise queries: PWRM
4. Towards ranking resources in MOOCs
5. Conclusions & future work
2

3. Introduction
3

4. Social Networks (Online Review Systems) & Reputation
4
1. Introduction

5. Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation

6. many types: professional links, friendships, purchases, ...
complex: dynamism, complexity of the social structure, many
nodes (users, entities, ..)
how can we identify and locate appropriate entities/services to
consume? (more and more available information online)
Not enough experience so.. online review systems a (Yelp,
Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation

7. many types: professional links, friendships, purchases, ...
complex: dynamism, complexity of the social structure, many
nodes (users, entities, ..)
how can we identify and locate appropriate entities/services to
consume? (more and more available information online)
Not enough experience so.. online review systems a (Yelp,
Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation
Trust
Reputation
opinions of third parties
Confidence
local experiences

8. many types
complex
nodes (users, entities, ..)
how can we identify and locate appropriate entities/services to
consume?
Not enough experience so..
Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation
Trust
Reputation
opinions of third parties
Confidence
local experiences

9. objective: extract reputation of entities (users, objects, …)
how: gathering and aggregating opinions
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms

10. objective:
how:
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms

11. objective:
how:
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms

12. objective:
how:
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms

13. Motivation
6

14. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
Capturing preferences through numerical opinions

15. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
Capturing preferences through numerical opinions

16. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
PROBLEM 1
‣DIFICULT TO MAP PREFERENCES INTO
NUMERICAL OPINIONS
‣SUBJECTIVITY!!!
Capturing preferences through numerical opinions

17. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Capturing preferences through numerical opinions

18. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
PROBLEM 2
BIAS PROBLEMS!!!
Capturing preferences through numerical opinions

19. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Die Hard 3
Gone with the wind 0
Ben-Hur 4
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Capturing preferences through numerical opinions

20. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Die Hard 3
Gone with the wind 0
Ben-Hur 4
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Capturing preferences through numerical opinions

21. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Die Hard 0.2
Gone with the wind 0.1
Ben-Hur 1.0
Capturing preferences through numerical opinions

22. passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Die Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Die Hard 0.2
Gone with the wind 0.1
Ben-Hur 1.0
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Capturing preferences through numerical opinions

23. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset

24. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies by users

25. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies by users
clearly biased to positive ratings

26. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies given by top critics

27. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies given by top critics
not influenced and biased by others’ ratings

28. 8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000
most reviewed movies given by top critics
not influenced and biased by others’ ratings
PROBLEM CONFIRMATION
potential bias problems when mapping
opinions onto numerical values, reputation
rankings may vary; and likely to cause
differences between the true quality of an
entity and its rating aggregated from opinions

29. 9
2. Motivation
FROM
Reputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Solution proposed

30. 9
2. Motivation
FROM
Reputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Reputation Rankings:
pro-active method + comparative opinions + comparative aggregation
TO
Solution proposed

31. 9
2. Motivation
FROM
Reputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Reputation Rankings:
pro-active method + comparative opinions + comparative aggregation
Pairwise preference
elicitation
Aggregation mechanism
TO
Solution proposed

32. From opinion ratings to
pairwise queries: PWRM
10

33. 11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwise queries:

34. 11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwise queries:
FROM
Ben-Hur
[0..1]
[Awful, fairly bad, It’s OK,
Will enjoy, Must see]
Gone with
the wind
:
15. Ben-Hur 4.3
:
23. Gone with the wind 4.1
:

35. 11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwise queries:
FROM
Ben-Hur
[0..1]
[Awful, fairly bad, It’s OK,
Will enjoy, Must see]
Gone with
the wind
:
15. Ben-Hur 4.3
:
23. Gone with the wind 4.1
:
TO Which movie do you prefer, Ben-Hur
or Gone with the wind?
:
15. Ben-Hur
:
23. Gone with the wind
:
easier for users to state opinions when the queries compare objects in a
pairwise fashion…
“… between these two objects, which one do you prefer?”

36. 12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Reputation (opinions aggregation) as an iterative process based on …

37. 12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (opinions aggregation) as an iterative process based on …

38. 12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (opinions aggregation) as an iterative process based on …
A
B
C
D
A
D
D
Match: pairwise comparison between two
entities
Dynamics: every match sent to a set of
users that reply to the query
Policies:
‣ Entity selection
‣Tournament schedule
‣ Users selection
‣Winner determination

39. 13
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (II)
Policies
Entities selection: which pair of entities should I select to be
compared?
‣ Mixed: current ranking vs new objects (ExploitationVs Exploration)
‣ Random
‣ Domain-dependent: objects with no information/fuzzy positions
Tournament schedule: how to initialize the tournament
‣ Random schedule (iterative process)

40. 14
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (III)
Policies
Users selection: who receives the queries (matches)?
‣ Random selection
‣Clustering of users by their preferences (representative users)
‣ Using (social) network properties: degree distribution, centrality of
nodes, …
Winner determination: how to decide which entity wins in a match
‣Voting procedures: preference replies from users count as votes
‣ Alternatives: absolute majority / full agreement (voting protocols)
‣ If there is no winner, no object gets through the next round

41. 15
3. From opinion ratings to pairwise queries: PWRM
Comparative aggregation: from matches to a ranking
When:After each match, the ranking is updated (iterative method)
How: Adaptation of a method for aggregating partial pairwise
comparison results into a ranking (Negahban et al., 2012)
‣Ranking approximation = random walk on G (weighted graph):
‣ An edge <ei,ej> if the pair has already been compared
‣The weights define the outcome of the comparisons
‣ Random walk uses a transition matrix P where:
‣ It moves from state ei to state ej with probability equal to the
chance that entity ej is preferred over entity ei
‣ Under these conditions, a vector w is a valid stationary distribution
for matrix P (wT
t+1 = wT · P)
‣ w defines the scores for each entity => ranking

42. 16
3. From opinion ratings to pairwise queries: PWRM
PWRM’s iterative process for building a reputation ranking
Require: a social network G = (U, E, LU , LE)
Require: a subset of E0
✓ E entities to be evaluated
1: for t 2 time do
2: Ei EntitiesSelectionPolicy.selectEntitiesToEvaluate(E0
)
3: KTEi
scheduleTournament(Ei)
4: for m 2 matches(KTEi ) do
5: nb UsersSelectionPolicy.getUsersToAsk(U)
6: send(m, nb)
7: votes receive()
8: winner WinnerDeterminationPolicy.getWinner(votes)
9: Ri AggregationMechanism.updateRanking(m, winner)
10: setWinnerNextRound(winner, KTE0 )
11: end for
12: end for
13: return Ri where E0
are ranked by their estimated reputation

43. Towards ranking resources in
MOOCs
17

44. Towards ranking resources in
MOOCs
17

45. 18
4.Towards ranking resources in MOOCs
Opinions in MOOCs?

46. 18
4.Towards ranking resources in MOOCs
Opinions in MOOCs?
opinions to rank
courses/resources

47. 19
4.Towards ranking resources in MOOCs
Applying PWRM into MOOCs
‣MOOCs modeled as a Social Network (Online Review Systems)
‣Apply PWRM for ranking learning resources in MOOCs
‣Allowing users (students/teachers) to find the best resources
‣Formalize a MOOC from a peer based system point of view
Idea:
Let M = hU, R, LR, LU i be a MOOC, where:
• U = {u1, . . . , un} is a set of users (teachers or students);
• R = {r1, . . . , rm} is the set of learning resources uploaded in the course;
• LR = {hui, rji/ui 2 U; rj 2 R} is the set of links among users and re-
sources, representing that user ui has uploaded the resource rj in the
course;
• LU = {huk, rmi/uk 2 U; rm 2 R} is the set of links also between users
and resources representing that user uk has used the resource rm.

48. 20
4.Towards ranking resources in MOOCs
PWRM as a function
‣ Objective: to query users about resources to give their opinions in order to build
a global ranking of resources regarding their reputation, so..
M = hU, R, LR, LU , ranki
• rank : R0
⇥ O ! {1, . . . , |R0
|} is a function in charge of defining a total
ordering (ranking) over a subset of resources R0
2 R, taking into account
the set of opinions O given by users;
• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rji
and representing a pairwise query sent to a set of users participating in
the MOOC, where learning resources ri and rj are compared.

49. 20
4.Towards ranking resources in MOOCs
PWRM as a function
‣ Objective: to query users about resources to give their opinions in order to build
a global ranking of resources regarding their reputation, so..
M = hU, R, LR, LU , ranki
• rank : R0
⇥ O ! {1, . . . , |R0
|} is a function in charge of defining a total
ordering (ranking) over a subset of resources R0
2 R, taking into account
the set of opinions O given by users;
• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rji
and representing a pairwise query sent to a set of users participating in
the MOOC, where learning resources ri and rj are compared.
rank = PWRM

50. 21
4.Towards ranking resources in MOOCs
PWRM algorithm in MOOCs
Require: a MOOC M = hU, R, LR, LU , ranki
Require: a subset of R0
✓ R of learning resources to be ranked
1: for t 2 time do
2: Ri ResourcesSelectionPolicy.selectResourcesToEvaluate(R0
)
3: KTRi
scheduleTournament(Ri)
4: for m 2 matches(KTRi ) do
5: Ui UsersSelectionPolicy.getUsersToAsk(U)
6: send(m, Ui)
7: Oi ReceiveOpinions()
8: winner WinnerDeterminationPolicy.getWinner(votes, Oi)
9: Ranki AggregationMechanism.updateRanking(Oi, winner)
10: promoteResourceWinnerToNextRound(winner, KTRi )
11: end for
12: end for
13: return Ranki where the subset R0
of learning resources are ranked by their
reputation

51. 22
4.Towards ranking resources in MOOCs
PWRM algorithm in MOOCs: Policies
‣ Resource selection policy:
- resources clustered regarding their typology (e.g. videos, recorded class…)
- regarding the number of opinions received by each resource (lowest/
highest number)
- opinions in terms of the result of each match (matches with tight results)
‣ User selection policy:
- taking advantage of the underlying structure generated by interactions
between users and resources
‣Winner determination policy:
- voting theory: simple majority, complete agreement, …

52. Conclusions & Future Work
23

53. 24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy to manipulate
‣ bias problems due to difficulty/subjectivity to map opinions into numerical values
Our Approach: PWRM
(1) based on comparative
(2) preference aggregation in reputation rankings (iterative process - tournaments)
(3) applied to MOOCs (ranking learning resources)
5. Conclusions & Future work

54. 24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy to manipulate
‣ bias problems
Our Approach: PWRM
(1) based on comparative opinions, elicited through pairwise preference request
(2) preference aggregation in reputation rankings (iterative process - tournaments)
(3) applied to MOOCs (ranking learning resources)
5. Conclusions & Future work

55. 25
Future work
5. Conclusions & Future work
‣ Adding social network properties:
- cluster users, centrality, betweenness, …
‣ Partial cooperative users:
- incentive mechanisms fostering cooperation (“what do you think users
prefer,A or B?”)
‣ Reputation of MOOCs:
- resources = courses, finding opinions in other opinions sites: twitter,
Facebook, forums, etc..
‣ Individual recommendation:
- resources/courses: from global reputation ranking to individual
recommendations

56. That’s all
Thank you for your attention!!
26

57. On the inaccuracy of numerical ratings:
A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto Centeno
rcenteno@lsi.uned.es
Dpto. Lenguajes y Sistemas Informáticos
UNED