Your SlideShare is downloading. ×
0
Relevance Ranking Metrics for Learning Objects Xavier Ochoa, ESPOL, Ecuador Erik Duval, KULeuven, Belgium
Agenda <ul><li>Why Relevance Ranking? </li></ul><ul><li>What is Relevance? </li></ul><ul><li>Relevance Ranking Metrics </l...
Economy of Abundance
Economy of Abundance Put your LMS here!
Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>=  </li></ul><ul><li>Difficult to find most relevant </li></ul>
Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>=  </li></ul><ul><li>Difficult to find most relevant </li></ul>...
Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>=  </li></ul><ul><li>Difficult to find most relevant </li></ul>...
How we do it now? <ul><li>Manual rating: </li></ul><ul><ul><li>Meaningful but  not Scalable </li></ul></ul><ul><li>Text ba...
What is Relevance?
Relevance Ranking Metrics Relevance  Ranking Metric B Metric A Metric C Context
Relevance Ranking Metrics Relevance  Ranking Meaningful Metric B Metric A Metric C Context
Relevance Ranking Metrics Relevance  Ranking Scalable Metric B Metric A Metric C Context
Topical Relevance
Personal Relevance
Situational Relevance
Do they work? <ul><li>Exploratory Study </li></ul><ul><ul><li>10 users (8 Prof. and 2 R.A.) </li></ul></ul><ul><ul><li>MIT...
 
 
 
Ranking the Rankings <ul><li>Baseline Rank: TF-IDF similarity algorithm </li></ul><ul><li>Re-rank according to Basic Metri...
Results
Basic Topical Relevance Metric Trees
Basic Personal Relevance Metric Net.
Basic Situational Relevance Metric XML
Linear Combination
Linear Combination
Results <ul><li>Even basic metrics improve the ranking </li></ul><ul><li>Linear Combination should be learnt </li></ul><ul...
What is next? <ul><li>Implementation in ARIADNE NEXT </li></ul><ul><li>Capture user behavior and learn from it for 3 month...
Conclusions <ul><li>Meaningful and Scalable metrics are possible </li></ul><ul><li>Cheap to implement  </li></ul><ul><li>P...
Thank You! Dank U! Gracias! <ul><li>Questions, Comments, Critics… are all welcome!! </li></ul>Xavier Ochoa [email_address]...
Upcoming SlideShare
Loading in...5
×

Relevance Ranking of Learning Objects

1,129

Published on

Presentation at EC-TEL 2007 (Crete). The need for relevance rank is presented and basic metrics are presented and evaluated.

Published in: Economy & Finance, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,129
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
42
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Relevance Ranking of Learning Objects"

  1. 1. Relevance Ranking Metrics for Learning Objects Xavier Ochoa, ESPOL, Ecuador Erik Duval, KULeuven, Belgium
  2. 2. Agenda <ul><li>Why Relevance Ranking? </li></ul><ul><li>What is Relevance? </li></ul><ul><li>Relevance Ranking Metrics </li></ul><ul><li>Do they work? </li></ul><ul><li>What is next? </li></ul>
  3. 3. Economy of Abundance
  4. 4. Economy of Abundance Put your LMS here!
  5. 5. Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>= </li></ul><ul><li>Difficult to find most relevant </li></ul>
  6. 6. Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>= </li></ul><ul><li>Difficult to find most relevant </li></ul>Solution: Do it the Google way
  7. 7. Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>= </li></ul><ul><li>Difficult to find most relevant </li></ul>Relevance Ranking MEANINGFUL & SCALABLE
  8. 8. How we do it now? <ul><li>Manual rating: </li></ul><ul><ul><li>Meaningful but not Scalable </li></ul></ul><ul><li>Text based Ranking Algorithms: </li></ul><ul><ul><li>Scalable but not Meaningful </li></ul></ul>
  9. 9. What is Relevance?
  10. 10. Relevance Ranking Metrics Relevance Ranking Metric B Metric A Metric C Context
  11. 11. Relevance Ranking Metrics Relevance Ranking Meaningful Metric B Metric A Metric C Context
  12. 12. Relevance Ranking Metrics Relevance Ranking Scalable Metric B Metric A Metric C Context
  13. 13. Topical Relevance
  14. 14. Personal Relevance
  15. 15. Situational Relevance
  16. 16. Do they work? <ul><li>Exploratory Study </li></ul><ul><ul><li>10 users (8 Prof. and 2 R.A.) </li></ul></ul><ul><ul><li>MIT OCW learning objects (34,640 LO) </li></ul></ul><ul><ul><li>Search and Select Objects to create 10 lessons (10 different topics on CS) </li></ul></ul><ul><ul><li>Web application was used </li></ul></ul>
  17. 20. Ranking the Rankings <ul><li>Baseline Rank: TF-IDF similarity algorithm </li></ul><ul><li>Re-rank according to Basic Metrics </li></ul><ul><li>All rankings compared with manual relevance ranking </li></ul><ul><li>Kendall Tau was used to measure similaritiy between lists </li></ul>
  18. 21. Results
  19. 22. Basic Topical Relevance Metric Trees
  20. 23. Basic Personal Relevance Metric Net.
  21. 24. Basic Situational Relevance Metric XML
  22. 25. Linear Combination
  23. 26. Linear Combination
  24. 27. Results <ul><li>Even basic metrics improve the ranking </li></ul><ul><li>Linear Combination should be learnt </li></ul><ul><li>Limited (and Synthetic) Study! </li></ul>
  25. 28. What is next? <ul><li>Implementation in ARIADNE NEXT </li></ul><ul><li>Capture user behavior and learn from it for 3 months </li></ul><ul><li>Natural (Real) evaluation of the metrics </li></ul><ul><li>Develop BETTER metrics </li></ul>
  26. 29. Conclusions <ul><li>Meaningful and Scalable metrics are possible </li></ul><ul><li>Cheap to implement </li></ul><ul><li>Provide significant improvement </li></ul><ul><li>While not optimal, they could be use as a base-line for further research </li></ul>
  27. 30. Thank You! Dank U! Gracias! <ul><li>Questions, Comments, Critics… are all welcome!! </li></ul>Xavier Ochoa [email_address] http://www.cti.espol.edu.ec/xavier Erik Duval Erik.Duval@cs.kuleuven.be http://www.cs.kuleuven.ac.be/~erikd
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×