Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
On the inaccuracy of numerical ratings:
A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto Centeno
rcen...
Outline
1. Introduction
2. Motivation
3. From opinion ratings to pairwise queries: PWRM
4. Towards ranking resources in MO...
Introduction
3
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review
Systems)
Reputation
many types: professional links, friendships, purchases, ...
complex: dynamism, complexity of the social structure, many
no...
many types: professional links, friendships, purchases, ...
complex: dynamism, complexity of the social structure, many
no...
many types
complex
nodes (users, entities, ..)
how can we identify and locate appropriate entities/services to
consume?
No...
objective: extract reputation of entities (users, objects, …)
how: gathering and aggregating opinions
examples:
5
1. Intro...
objective:
how:
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
objective:
how:
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
objective:
how:
examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
Motivation
6
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
passive: expecting users’ opinions
capturing opinions through numerical ratings + textual information
estimate reputation ...
8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzin...
8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzin...
8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzin...
8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzin...
8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzin...
8
2. Motivation
best model fittingVS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzin...
9
2. Motivation
FROM
Reputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Sol...
9
2. Motivation
FROM
Reputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Rep...
9
2. Motivation
FROM
Reputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Rep...
From opinion ratings to
pairwise queries: PWRM
10
11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwi...
11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwi...
11
3. From opinion ratings to pairwise queries: PWRM
Comparative opinions: Pairwise preference elicitation
Based on pairwi...
12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Reputation (opinions aggregation) as...
12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (o...
12
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (o...
13
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (II)
Policies
Entities selection: which ...
14
3. From opinion ratings to pairwise queries: PWRM
Pairwise comparison dynamics (III)
Policies
Users selection: who rece...
15
3. From opinion ratings to pairwise queries: PWRM
Comparative aggregation: from matches to a ranking
When:After each ma...
16
3. From opinion ratings to pairwise queries: PWRM
PWRM’s iterative process for building a reputation ranking
Require: a...
Towards ranking resources in
MOOCs
17
Towards ranking resources in
MOOCs
17
18
4.Towards ranking resources in MOOCs
Opinions in MOOCs?
18
4.Towards ranking resources in MOOCs
Opinions in MOOCs?
opinions to rank
courses/resources
19
4.Towards ranking resources in MOOCs
Applying PWRM into MOOCs
‣MOOCs modeled as a Social Network (Online Review Systems...
20
4.Towards ranking resources in MOOCs
PWRM as a function
‣ Objective: to query users about resources to give their opini...
20
4.Towards ranking resources in MOOCs
PWRM as a function
‣ Objective: to query users about resources to give their opini...
21
4.Towards ranking resources in MOOCs
PWRM algorithm in MOOCs
Require: a MOOC M = hU, R, LR, LU , ranki
Require: a subse...
22
4.Towards ranking resources in MOOCs
PWRM algorithm in MOOCs: Policies
‣ Resource selection policy:
- resources cluster...
Conclusions & Future Work
23
24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy t...
24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy t...
25
Future work
5. Conclusions & Future work
‣ Adding social network properties:
- cluster users, centrality, betweenness, ...
That’s all
Thank you for your attention!!
26
On the inaccuracy of numerical ratings:
A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto Centeno
rcen...
Upcoming SlideShare
Loading in …5
×

V Jornadas eMadrid sobre “Educación Digital”. Roberto Centeno, Universidad Nacional de Educación a Distancia: Mecanismos de reputación en MOOCs

527 views

Published on

V Jornadas eMadrid sobre “Educación Digital”. Roberto Centeno, Universidad Nacional de Educación a Distancia: Mecanismos de reputación en MOOCs. 2015-06-30

Published in: Education
  • Be the first to comment

  • Be the first to like this

V Jornadas eMadrid sobre “Educación Digital”. Roberto Centeno, Universidad Nacional de Educación a Distancia: Mecanismos de reputación en MOOCs

  1. 1. On the inaccuracy of numerical ratings: A pairwise based reputation mechanism in MOOCs July 1st, 2015 Roberto Centeno rcenteno@lsi.uned.es Dpto. Lenguajes y Sistemas Informáticos UNED
  2. 2. Outline 1. Introduction 2. Motivation 3. From opinion ratings to pairwise queries: PWRM 4. Towards ranking resources in MOOCs 5. Conclusions & future work 2
  3. 3. Introduction 3
  4. 4. Social Networks (Online Review Systems) & Reputation 4 1. Introduction
  5. 5. Social Networks (Online Review Systems) & Reputation 4 1. Introduction Social Networks (Online Review Systems) Reputation
  6. 6. many types: professional links, friendships, purchases, ... complex: dynamism, complexity of the social structure, many nodes (users, entities, ..) how can we identify and locate appropriate entities/services to consume? (more and more available information online) Not enough experience so.. online review systems a (Yelp, Tripadvisor, ..) as a means to obtain opinions, rankings, etc… Social Networks (Online Review Systems) & Reputation 4 1. Introduction Social Networks (Online Review Systems) Reputation
  7. 7. many types: professional links, friendships, purchases, ... complex: dynamism, complexity of the social structure, many nodes (users, entities, ..) how can we identify and locate appropriate entities/services to consume? (more and more available information online) Not enough experience so.. online review systems a (Yelp, Tripadvisor, ..) as a means to obtain opinions, rankings, etc… Social Networks (Online Review Systems) & Reputation 4 1. Introduction Social Networks (Online Review Systems) Reputation Trust Reputation opinions of third parties Confidence local experiences
  8. 8. many types complex nodes (users, entities, ..) how can we identify and locate appropriate entities/services to consume? Not enough experience so.. Tripadvisor, ..) as a means to obtain opinions, rankings, etc… Social Networks (Online Review Systems) & Reputation 4 1. Introduction Social Networks (Online Review Systems) Reputation Trust Reputation opinions of third parties Confidence local experiences
  9. 9. objective: extract reputation of entities (users, objects, …) how: gathering and aggregating opinions examples: 5 1. Introduction Reputation Mechanisms in Social Networks Reputation Mechanisms
  10. 10. objective: how: examples: 5 1. Introduction Reputation Mechanisms in Social Networks Reputation Mechanisms
  11. 11. objective: how: examples: 5 1. Introduction Reputation Mechanisms in Social Networks Reputation Mechanisms
  12. 12. objective: how: examples: 5 1. Introduction Reputation Mechanisms in Social Networks Reputation Mechanisms
  13. 13. Motivation 6
  14. 14. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… Capturing preferences through numerical opinions
  15. 15. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… Capturing preferences through numerical opinions
  16. 16. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… PROBLEM 1 ‣DIFICULT TO MAP PREFERENCES INTO NUMERICAL OPINIONS ‣SUBJECTIVITY!!! Capturing preferences through numerical opinions
  17. 17. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… AGR Capturing preferences through numerical opinions
  18. 18. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… AGR PROBLEM 2 BIAS PROBLEMS!!! Capturing preferences through numerical opinions
  19. 19. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… AGR Die Hard 1 Gone with the wind 2 Ben-Hur 3 Ben-Hur ≻ Gone with the wind ≻ Die Hard Die Hard 3 Gone with the wind 0 Ben-Hur 4 Ben-Hur ≻ Die Hard ≻ Gone with the wind Capturing preferences through numerical opinions
  20. 20. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… AGR Die Hard 1 Gone with the wind 2 Ben-Hur 3 Ben-Hur ≻ Gone with the wind ≻ Die Hard Die Hard 3 Gone with the wind 0 Ben-Hur 4 Ben-Hur ≻ Die Hard ≻ Gone with the wind Ben-Hur ≻ Die Hard ≻ Gone with the wind Capturing preferences through numerical opinions
  21. 21. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… AGR Die Hard 1 Gone with the wind 2 Ben-Hur 3 Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind Ben-Hur ≻ Die Hard ≻ Gone with the wind Die Hard 0.2 Gone with the wind 0.1 Ben-Hur 1.0 Capturing preferences through numerical opinions
  22. 22. passive: expecting users’ opinions capturing opinions through numerical ratings + textual information estimate reputation based on aggregating ratings (average ratings) 7 2. Motivation Reputation Mechanisms (traditionally)… AGR Die Hard 1 Gone with the wind 2 Ben-Hur 3 Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind Ben-Hur ≻ Die Hard ≻ Gone with the wind Die Hard 0.2 Gone with the wind 0.1 Ben-Hur 1.0 Ben-Hur ≻ Gone with the wind ≻ Die Hard Capturing preferences through numerical opinions
  23. 23. 8 2. Motivation best model fittingVS best fit to a normal distribution Bias problems derived from numerical ratings Analyzing the distribution of average ratings from HetRec-2011 dataset
  24. 24. 8 2. Motivation best model fittingVS best fit to a normal distribution Bias problems derived from numerical ratings Analyzing the distribution of average ratings from HetRec-2011 dataset average rating distribution of the 100, 1.000 and 4.000 most reviewed movies by users
  25. 25. 8 2. Motivation best model fittingVS best fit to a normal distribution Bias problems derived from numerical ratings Analyzing the distribution of average ratings from HetRec-2011 dataset average rating distribution of the 100, 1.000 and 4.000 most reviewed movies by users clearly biased to positive ratings
  26. 26. 8 2. Motivation best model fittingVS best fit to a normal distribution Bias problems derived from numerical ratings Analyzing the distribution of average ratings from HetRec-2011 dataset average rating distribution of the 100, 1.000 and 4.000 most reviewed movies given by top critics
  27. 27. 8 2. Motivation best model fittingVS best fit to a normal distribution Bias problems derived from numerical ratings Analyzing the distribution of average ratings from HetRec-2011 dataset average rating distribution of the 100, 1.000 and 4.000 most reviewed movies given by top critics not influenced and biased by others’ ratings
  28. 28. 8 2. Motivation best model fittingVS best fit to a normal distribution Bias problems derived from numerical ratings Analyzing the distribution of average ratings from HetRec-2011 dataset average rating distribution of the 100, 1.000 and 4.000 most reviewed movies given by top critics not influenced and biased by others’ ratings PROBLEM CONFIRMATION potential bias problems when mapping opinions onto numerical values, reputation rankings may vary; and likely to cause differences between the true quality of an entity and its rating aggregated from opinions
  29. 29. 9 2. Motivation FROM Reputation Rankings: passive method + numerical opinions + opinions aggregation (average ratings) Solution proposed
  30. 30. 9 2. Motivation FROM Reputation Rankings: passive method + numerical opinions + opinions aggregation (average ratings) Reputation Rankings: pro-active method + comparative opinions + comparative aggregation TO Solution proposed
  31. 31. 9 2. Motivation FROM Reputation Rankings: passive method + numerical opinions + opinions aggregation (average ratings) Reputation Rankings: pro-active method + comparative opinions + comparative aggregation Pairwise preference elicitation Aggregation mechanism TO Solution proposed
  32. 32. From opinion ratings to pairwise queries: PWRM 10
  33. 33. 11 3. From opinion ratings to pairwise queries: PWRM Comparative opinions: Pairwise preference elicitation Based on pairwise queries:
  34. 34. 11 3. From opinion ratings to pairwise queries: PWRM Comparative opinions: Pairwise preference elicitation Based on pairwise queries: FROM Ben-Hur [0..1] [Awful, fairly bad, It’s OK, Will enjoy, Must see] Gone with the wind : 15. Ben-Hur 4.3 : 23. Gone with the wind 4.1 :
  35. 35. 11 3. From opinion ratings to pairwise queries: PWRM Comparative opinions: Pairwise preference elicitation Based on pairwise queries: FROM Ben-Hur [0..1] [Awful, fairly bad, It’s OK, Will enjoy, Must see] Gone with the wind : 15. Ben-Hur 4.3 : 23. Gone with the wind 4.1 : TO Which movie do you prefer, Ben-Hur or Gone with the wind? : 15. Ben-Hur : 23. Gone with the wind : easier for users to state opinions when the queries compare objects in a pairwise fashion… “… between these two objects, which one do you prefer?”
  36. 36. 12 3. From opinion ratings to pairwise queries: PWRM Pairwise comparison dynamics (I) Reputation (opinions aggregation) as an iterative process based on …
  37. 37. 12 3. From opinion ratings to pairwise queries: PWRM Pairwise comparison dynamics (I) Knock-Out tournaments: Reputation (opinions aggregation) as an iterative process based on …
  38. 38. 12 3. From opinion ratings to pairwise queries: PWRM Pairwise comparison dynamics (I) Knock-Out tournaments: Reputation (opinions aggregation) as an iterative process based on … A B C D A D D Match: pairwise comparison between two entities Dynamics: every match sent to a set of users that reply to the query Policies: ‣ Entity selection ‣Tournament schedule ‣ Users selection ‣Winner determination
  39. 39. 13 3. From opinion ratings to pairwise queries: PWRM Pairwise comparison dynamics (II) Policies Entities selection: which pair of entities should I select to be compared? ‣ Mixed: current ranking vs new objects (ExploitationVs Exploration) ‣ Random ‣ Domain-dependent: objects with no information/fuzzy positions Tournament schedule: how to initialize the tournament ‣ Random schedule (iterative process)
  40. 40. 14 3. From opinion ratings to pairwise queries: PWRM Pairwise comparison dynamics (III) Policies Users selection: who receives the queries (matches)? ‣ Random selection ‣Clustering of users by their preferences (representative users) ‣ Using (social) network properties: degree distribution, centrality of nodes, … Winner determination: how to decide which entity wins in a match ‣Voting procedures: preference replies from users count as votes ‣ Alternatives: absolute majority / full agreement (voting protocols) ‣ If there is no winner, no object gets through the next round
  41. 41. 15 3. From opinion ratings to pairwise queries: PWRM Comparative aggregation: from matches to a ranking When:After each match, the ranking is updated (iterative method) How: Adaptation of a method for aggregating partial pairwise comparison results into a ranking (Negahban et al., 2012) ‣Ranking approximation = random walk on G (weighted graph): ‣ An edge <ei,ej> if the pair has already been compared ‣The weights define the outcome of the comparisons ‣ Random walk uses a transition matrix P where: ‣ It moves from state ei to state ej with probability equal to the chance that entity ej is preferred over entity ei ‣ Under these conditions, a vector w is a valid stationary distribution for matrix P (wT t+1 = wT · P) ‣ w defines the scores for each entity => ranking
  42. 42. 16 3. From opinion ratings to pairwise queries: PWRM PWRM’s iterative process for building a reputation ranking Require: a social network G = (U, E, LU , LE) Require: a subset of E0 ✓ E entities to be evaluated 1: for t 2 time do 2: Ei EntitiesSelectionPolicy.selectEntitiesToEvaluate(E0 ) 3: KTEi scheduleTournament(Ei) 4: for m 2 matches(KTEi ) do 5: nb UsersSelectionPolicy.getUsersToAsk(U) 6: send(m, nb) 7: votes receive() 8: winner WinnerDeterminationPolicy.getWinner(votes) 9: Ri AggregationMechanism.updateRanking(m, winner) 10: setWinnerNextRound(winner, KTE0 ) 11: end for 12: end for 13: return Ri where E0 are ranked by their estimated reputation
  43. 43. Towards ranking resources in MOOCs 17
  44. 44. Towards ranking resources in MOOCs 17
  45. 45. 18 4.Towards ranking resources in MOOCs Opinions in MOOCs?
  46. 46. 18 4.Towards ranking resources in MOOCs Opinions in MOOCs? opinions to rank courses/resources
  47. 47. 19 4.Towards ranking resources in MOOCs Applying PWRM into MOOCs ‣MOOCs modeled as a Social Network (Online Review Systems) ‣Apply PWRM for ranking learning resources in MOOCs ‣Allowing users (students/teachers) to find the best resources ‣Formalize a MOOC from a peer based system point of view Idea: Let M = hU, R, LR, LU i be a MOOC, where: • U = {u1, . . . , un} is a set of users (teachers or students); • R = {r1, . . . , rm} is the set of learning resources uploaded in the course; • LR = {hui, rji/ui 2 U; rj 2 R} is the set of links among users and re- sources, representing that user ui has uploaded the resource rj in the course; • LU = {huk, rmi/uk 2 U; rm 2 R} is the set of links also between users and resources representing that user uk has used the resource rm.
  48. 48. 20 4.Towards ranking resources in MOOCs PWRM as a function ‣ Objective: to query users about resources to give their opinions in order to build a global ranking of resources regarding their reputation, so.. M = hU, R, LR, LU , ranki • rank : R0 ⇥ O ! {1, . . . , |R0 |} is a function in charge of defining a total ordering (ranking) over a subset of resources R0 2 R, taking into account the set of opinions O given by users; • oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rji and representing a pairwise query sent to a set of users participating in the MOOC, where learning resources ri and rj are compared.
  49. 49. 20 4.Towards ranking resources in MOOCs PWRM as a function ‣ Objective: to query users about resources to give their opinions in order to build a global ranking of resources regarding their reputation, so.. M = hU, R, LR, LU , ranki • rank : R0 ⇥ O ! {1, . . . , |R0 |} is a function in charge of defining a total ordering (ranking) over a subset of resources R0 2 R, taking into account the set of opinions O given by users; • oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rji and representing a pairwise query sent to a set of users participating in the MOOC, where learning resources ri and rj are compared. rank = PWRM
  50. 50. 21 4.Towards ranking resources in MOOCs PWRM algorithm in MOOCs Require: a MOOC M = hU, R, LR, LU , ranki Require: a subset of R0 ✓ R of learning resources to be ranked 1: for t 2 time do 2: Ri ResourcesSelectionPolicy.selectResourcesToEvaluate(R0 ) 3: KTRi scheduleTournament(Ri) 4: for m 2 matches(KTRi ) do 5: Ui UsersSelectionPolicy.getUsersToAsk(U) 6: send(m, Ui) 7: Oi ReceiveOpinions() 8: winner WinnerDeterminationPolicy.getWinner(votes, Oi) 9: Ranki AggregationMechanism.updateRanking(Oi, winner) 10: promoteResourceWinnerToNextRound(winner, KTRi ) 11: end for 12: end for 13: return Ranki where the subset R0 of learning resources are ranked by their reputation
  51. 51. 22 4.Towards ranking resources in MOOCs PWRM algorithm in MOOCs: Policies ‣ Resource selection policy: - resources clustered regarding their typology (e.g. videos, recorded class…) - regarding the number of opinions received by each resource (lowest/ highest number) - opinions in terms of the result of each match (matches with tight results) ‣ User selection policy: - taking advantage of the underlying structure generated by interactions between users and resources ‣Winner determination policy: - voting theory: simple majority, complete agreement, …
  52. 52. Conclusions & Future Work 23
  53. 53. 24 Contributions Current reputation mechanisms ✓ follow a very passive/static and quantitative dependent approach ‣ easy to manipulate ‣ bias problems due to difficulty/subjectivity to map opinions into numerical values Our Approach: PWRM (1) based on comparative (2) preference aggregation in reputation rankings (iterative process - tournaments) (3) applied to MOOCs (ranking learning resources) 5. Conclusions & Future work
  54. 54. 24 Contributions Current reputation mechanisms ✓ follow a very passive/static and quantitative dependent approach ‣ easy to manipulate ‣ bias problems Our Approach: PWRM (1) based on comparative opinions, elicited through pairwise preference request (2) preference aggregation in reputation rankings (iterative process - tournaments) (3) applied to MOOCs (ranking learning resources) 5. Conclusions & Future work
  55. 55. 25 Future work 5. Conclusions & Future work ‣ Adding social network properties: - cluster users, centrality, betweenness, … ‣ Partial cooperative users: - incentive mechanisms fostering cooperation (“what do you think users prefer,A or B?”) ‣ Reputation of MOOCs: - resources = courses, finding opinions in other opinions sites: twitter, Facebook, forums, etc.. ‣ Individual recommendation: - resources/courses: from global reputation ranking to individual recommendations
  56. 56. That’s all Thank you for your attention!! 26
  57. 57. On the inaccuracy of numerical ratings: A pairwise based reputation mechanism in MOOCs July 1st, 2015 Roberto Centeno rcenteno@lsi.uned.es Dpto. Lenguajes y Sistemas Informáticos UNED

×