The document discusses the importance of relevance ranking metrics for learning objects, emphasizing the challenges posed by the abundance of resources. It outlines an exploratory study that tested various ranking algorithms against manual ratings, showing that basic metrics can enhance selection processes. The authors conclude that while current metrics are not optimal, they can serve as a foundational tool for future research and improvement.

1. Relevance Ranking Metrics for Learning Objects Xavier Ochoa, ESPOL, Ecuador Erik Duval, KULeuven, Belgium

2. Agenda Why Relevance Ranking? What is Relevance? Relevance Ranking Metrics Do they work? What is next?

3. Economy of Abundance

4. Economy of Abundance Put your LMS here!

5. Why Relevance Ranking? Abundance = Difficult to find most relevant

6. Why Relevance Ranking? Abundance = Difficult to find most relevant Solution: Do it the Google way

7. Why Relevance Ranking? Abundance = Difficult to find most relevant Relevance Ranking MEANINGFUL & SCALABLE

8. How we do it now? Manual rating: Meaningful but not Scalable Text based Ranking Algorithms: Scalable but not Meaningful

9. What is Relevance?

10. Relevance Ranking Metrics Relevance Ranking Metric B Metric A Metric C Context

11. Relevance Ranking Metrics Relevance Ranking Meaningful Metric B Metric A Metric C Context

12. Relevance Ranking Metrics Relevance Ranking Scalable Metric B Metric A Metric C Context

13. Topical Relevance

14. Personal Relevance

15. Situational Relevance

16. Do they work? Exploratory Study 10 users (8 Prof. and 2 R.A.) MIT OCW learning objects (34,640 LO) Search and Select Objects to create 10 lessons (10 different topics on CS) Web application was used

20. Ranking the Rankings Baseline Rank: TF-IDF similarity algorithm Re-rank according to Basic Metrics All rankings compared with manual relevance ranking Kendall Tau was used to measure similaritiy between lists

21. Results

22. Basic Topical Relevance Metric Trees

23. Basic Personal Relevance Metric Net.

24. Basic Situational Relevance Metric XML

25. Linear Combination

26. Linear Combination

27. Results Even basic metrics improve the ranking Linear Combination should be learnt Limited (and Synthetic) Study!

28. What is next? Implementation in ARIADNE NEXT Capture user behavior and learn from it for 3 months Natural (Real) evaluation of the metrics Develop BETTER metrics

29. Conclusions Meaningful and Scalable metrics are possible Cheap to implement Provide significant improvement While not optimal, they could be use as a base-line for further research

30. Thank You! Dank U! Gracias! Questions, Comments, Critics… are all welcome!! Xavier Ochoa [email_address] http://www.cti.espol.edu.ec/xavier Erik Duval Erik.Duval@cs.kuleuven.be http://www.cs.kuleuven.ac.be/~erikd