Relevance Ranking of Learning Objects

1,318 views

Published on

Presentation at EC-TEL 2007 (Crete). The need for relevance rank is presented and basic metrics are presented and evaluated.

Published in: Economy & Finance, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,318
On SlideShare
0
From Embeds
0
Number of Embeds
27
Actions
Shares
0
Downloads
48
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Relevance Ranking of Learning Objects

  1. 1. Relevance Ranking Metrics for Learning Objects Xavier Ochoa, ESPOL, Ecuador Erik Duval, KULeuven, Belgium
  2. 2. Agenda <ul><li>Why Relevance Ranking? </li></ul><ul><li>What is Relevance? </li></ul><ul><li>Relevance Ranking Metrics </li></ul><ul><li>Do they work? </li></ul><ul><li>What is next? </li></ul>
  3. 3. Economy of Abundance
  4. 4. Economy of Abundance Put your LMS here!
  5. 5. Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>= </li></ul><ul><li>Difficult to find most relevant </li></ul>
  6. 6. Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>= </li></ul><ul><li>Difficult to find most relevant </li></ul>Solution: Do it the Google way
  7. 7. Why Relevance Ranking? <ul><li>Abundance </li></ul><ul><li>= </li></ul><ul><li>Difficult to find most relevant </li></ul>Relevance Ranking MEANINGFUL & SCALABLE
  8. 8. How we do it now? <ul><li>Manual rating: </li></ul><ul><ul><li>Meaningful but not Scalable </li></ul></ul><ul><li>Text based Ranking Algorithms: </li></ul><ul><ul><li>Scalable but not Meaningful </li></ul></ul>
  9. 9. What is Relevance?
  10. 10. Relevance Ranking Metrics Relevance Ranking Metric B Metric A Metric C Context
  11. 11. Relevance Ranking Metrics Relevance Ranking Meaningful Metric B Metric A Metric C Context
  12. 12. Relevance Ranking Metrics Relevance Ranking Scalable Metric B Metric A Metric C Context
  13. 13. Topical Relevance
  14. 14. Personal Relevance
  15. 15. Situational Relevance
  16. 16. Do they work? <ul><li>Exploratory Study </li></ul><ul><ul><li>10 users (8 Prof. and 2 R.A.) </li></ul></ul><ul><ul><li>MIT OCW learning objects (34,640 LO) </li></ul></ul><ul><ul><li>Search and Select Objects to create 10 lessons (10 different topics on CS) </li></ul></ul><ul><ul><li>Web application was used </li></ul></ul>
  17. 20. Ranking the Rankings <ul><li>Baseline Rank: TF-IDF similarity algorithm </li></ul><ul><li>Re-rank according to Basic Metrics </li></ul><ul><li>All rankings compared with manual relevance ranking </li></ul><ul><li>Kendall Tau was used to measure similaritiy between lists </li></ul>
  18. 21. Results
  19. 22. Basic Topical Relevance Metric Trees
  20. 23. Basic Personal Relevance Metric Net.
  21. 24. Basic Situational Relevance Metric XML
  22. 25. Linear Combination
  23. 26. Linear Combination
  24. 27. Results <ul><li>Even basic metrics improve the ranking </li></ul><ul><li>Linear Combination should be learnt </li></ul><ul><li>Limited (and Synthetic) Study! </li></ul>
  25. 28. What is next? <ul><li>Implementation in ARIADNE NEXT </li></ul><ul><li>Capture user behavior and learn from it for 3 months </li></ul><ul><li>Natural (Real) evaluation of the metrics </li></ul><ul><li>Develop BETTER metrics </li></ul>
  26. 29. Conclusions <ul><li>Meaningful and Scalable metrics are possible </li></ul><ul><li>Cheap to implement </li></ul><ul><li>Provide significant improvement </li></ul><ul><li>While not optimal, they could be use as a base-line for further research </li></ul>
  27. 30. Thank You! Dank U! Gracias! <ul><li>Questions, Comments, Critics… are all welcome!! </li></ul>Xavier Ochoa [email_address] http://www.cti.espol.edu.ec/xavier Erik Duval Erik.Duval@cs.kuleuven.be http://www.cs.kuleuven.ac.be/~erikd

×