‘Knowing about collaboration andcommunication’ (the 3rd row with *) is not addressed by SAM, but is added to check a possible bias. The highest rated was ‘knowing how much time students spent’ and ‘Awareness of what students are doing’ Finding students in trouble and the best students was also rated rather low. Awareness of resource use has been mostly met, but can be improved by differentiating external resources (the external resource use issue is indecisive).
Actual use was high
For this evaluation we wanted to get expert feedback and see how SAM would operate in a large course. SAM was deployed in an open onlinecourse on Learning and Knowledge Analytics (LAK)5 – an emerging research domain in TEL that focuses on better measurement, analysis, visualization and reporting of data about learners . More details on iteration 2 and 3 are available in . allow re-ordering of the axes through drag-and-drop for better metrics comparison. To cope with the line density better, configurable histograms (12) are added to the axes.270 participants
Providing feedback most importantBoth LAK and CGIAR teachers want to understand the document use. The main differences between LAK and CGIAR teachers are: LAK rates finding students at risk higher and finding good students lower, online tool use is not so interesting for LAK teachers and collaboration is more important. Awareness is also rated high. Comparing with the objectives, awareness and resource use is again the most important.
How can data sets be shared according to privacy and legal protection rights? How to develop a respective policy to use and share data sets? How to pre-process data sets to make them suitable for other researchers? How to define common evaluation criteria for TEL recommender systems? How to develop overview methods to monitor the performance of TEL recommender systems on data sets?
1. Research and Deployment of Analytics in Learning Settings PAWS Meeting 9 April 2012 School of Information Sciences, University of Pittsburgh Katrien Verbert
2. Human-Computer Interaction Awareness Sense-making prof. Erik Duval Computer Graphics prof. Phil Dutré Language Intelligence Information Retrieval prof. Sien Moens Flexible Interaction between people and information http://hci.cs.kuleuven.be/
7. Duval, Erik. Attention please! Learning analytics for visualization and recommendation, Proceedings of LAK11: 1stInternational Conference on Learning Analytics and Knowledge, pages 9-17, ACM (2011)
8. objectives • self-monitoring for learners • awareness for teachers • learning resource use and recommendations • part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]
9. overview • Student Activity Meter • Step Up! • Recommender systems for learning • Future research plans
10. Student activity meter (SAM): demo. http://ariadne.cs.kuleuven.be/monitorwidget-rwtheval/ or http://bit.ly/I8AYV1
11. Design Based Research Methodology • Rapid prototyping • Evaluate Ideas in short iteration cycles of Design, Implementation Evaluation • Focus on Usefulness Usability • Think-aloud evaluations, SUS (System Usability Scale) surveys, usability lab, ...
12. Iteration one • usability and user satisfaction evaluation • 12CS students, using a -based time tracker • 2 evaluation sessions: • task based interview with think aloud (after 1 week of tracking) • user satisfaction (SUS MSDT) (after 1 month)
13. User satisfaction • average SUS score: 73%
14. iteration two • 20 persons: 3 CGIAR, 2 Law, 8 CS teachers 7 CS TA s. • An online survey about usefulness, teacher issues and how the tool can resolve these. • on average: 40 mins are spent using SAM.
15. CGIAR CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ?!Being aware of what students are doing ✔ ✔Knowing about collaboration and communication ✔ ✗Knowing which documents are used and how much ✔ ✔ ✔Knowing how and when online tools have been used ?!Finding the students who are not doing well ✔ ?!Finding the best students ?! ?!Knowing how much time students spent ?! ✔Knowing if external learning resources are used ✔ ?!
16. demogra evaluation design negative positive phics goal changes • ↑learnability usability, • ↓errors 12 CS satisfaction, small usabilityI. 1st iteration • good satisfaction students preliminary issues • usefulness usefulness positive • provides assessing 19 resource awareness teacher needs,II. teachers help function recomm. not • all vis. useful use TA s useful • many uses usefulness • 90% wants it
17. iteration three • open course on learning and knowledge analytics, http://bit.ly/dWYVbX • 12 visual analytics enthousiasts + experts (who also teach) • almost identical survey to CGIAR case.
18. LAK CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ✔Being aware of what students are doing ✔ ?!Knowing about collaboration and communication ✔ ✗Knowing which documents are used and how much ✔ ?!Knowing how and when online tools have been used ✗ ?!Finding the students who are not doing well ✔ ?!Finding the best students ?! ✗Knowing how much time students spent ?! ✔Knowing if external learning resources are used ?! ?!
19. ideas from experts 2 the used resource types 5 detailed information per student 4 detailed information of 2 students 3 detailed usage stats of resources 1 stats or vis. on content creation
20. demogra evaluation design negative positive phics goal changes usability, • ↑learnability 12 CS satisfaction, small usability • ↓errors I. 1st iteration students preliminary issues • good satisfaction usefulness • usefulness positive assessing • provides awareness 19 resource teacher • all vis. useful II. teachers help function recomm. not needs, use • many uses TA s useful usefulness • 90% want it • provides awareness assessing re-orderable most and feedback 12 teacher parallel addressed • many uses III. participan needs, expert coordinates needs are • 66% want it ts feedback, use with indecisive • recomm. can be usefulness histograms useful
21. Iteration four • a CS course on C++ programming • 11people: 7 teachers, 2 TA s 1 course planner • richerdata set: tracking from programming environment • qualitative study using a structured face-2-face interview
22. USER SATISFACTION • average SUS score: 69,69% all: want to continue using it 9/11: give it to students
23. demo- evaluation design negative positive graphics goal changes usability, • ↑learnability 12 CS satisfaction, small usability • ↓errors I. 1st iteration students preliminary issues • good satisfaction usefulness • usefulness positive • provides awareness 19 assessing teacher resource • all vis. useful II. teachers needs, use help function recomm. not • many uses TA s usefulness useful • 90% want it • provides awareness and assessing teacher most 12 re-orderable feedback needs, expert addressedIII. participant PC with • many uses feedback, use needs are s histograms • 66% want it usefulness indecisive • recomm. can be useful • provides time overview ﬁlter search, conﬂicting • provides course overview 11 use, usefulness icons, zooming visions of • PC assist with detectingIV. teachers satisfaction in line chart, students doing problems TA s editing PC axes well or at risk • many uses insights • 100% want it
24. conclusion • SAMenables to ﬁnd a wide variety of new insights • a better course overview • understanding student time spending • almostall participants want to continue using SAM 26
25. Santos Odriozola, Jose Luis; Govaerts, Sten; Verbert, Katrien; Duval, ErikGoal-oriented visualizations of activity tracking: a case study with engineering students, Proceedings of LAK12: 2ndInternational Conference on Learning Analytics and Knowledge, pages 10, ACM (to appear)
26. Human-Computer Interaction Course
29. User satisfaction • average SUS score: 77%
30. Nikos Manouselis, Hendrik Drachsler, Katrien Verbert and Erik Duval. Recommender Systems for Learning.SpringerBriefs in Computer Science, 90 pages, Springer US (to appear).
32. challenges • Evaluation • Data sets • Context • User interfaces
33. EVALUATION DATA SETS
34. Verbert, Katrien; Drachsler, Hendrik; Manouselis, Nikos; Wolpers, Martin; Vuorikari, Riina; Duval, Erik.Dataset-driven research for improving TEL recommender systems, LAK11:1st InternationalConference on Learning Analytics and Knowledge, pages 44-53 (2011)
36. how to achieve objectives • Setting up a website / maintain TELeurope group community • Setup a open data repository for sharing educational datasets and related researches outcomes • Organizing annual workshop and SI • Organizing a data competition like in TREC
37. dataTEL challenge dataTEL cafe event • a call for TEL datasets • eight data sets submitted http://bit.ly/ieqmWW
55. References • Duval, E. (2011). Attention please!: learning analytics for visualization and recommendation. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, (pp. 9-17), ACM. • D. Keim, G. Andrienko, J.-D. Fekete, C. Go ̈rg, J. Kohlhammer, and G. Melanc ̧on. Visual Analytics: Deﬁnition, Process, and Challenges. In A. Kerren, J. Stasko, J.-D. Fekete, and C. North, editors, Information Visualization, volume 4950 of Lecture Notes in Computer Science, pages 154–175. Springer Berlin / Heidelberg, 2008 • J. Klerkx and E. Duval. Visualising social bookmarks. Journal of Digital Information, 10(2):1–40, 2009 • J. ODonovan, B. Gretarsson, S.Bostandjiev, C. Hall, and T. Hollerer. SmallWorlds: Visualizing Social Recommendations. In G. Melançon, T. Munzner, and D. Weiskopf (eds) Eurographics/ IEEE-VGTC Symposium on Visualization 2010, Volume 29 (2010), Number 3, 10 pages • Siemens, G. Gasevic, D. (eds) (2011). Proceedings of the 1st conference on Learning Analytics and Knowledge 2011. ACM.