The document describes three evaluations of the Student Activity Meter (SAM) tool: 1) Usability study with 12 computer science students found they understood the visualizations well but some issues were identified and addressed. 2) Survey of 20 people involved in CGIAR courses found SAM helped increase teacher awareness of student activities and resource use but did not support collaboration tracking. 3) Survey of 12 participants in an open online course also identified benefits but desired more metrics and comparison tools. Teachers indicated SAM could provide feedback and monitor student progress.

1. EVALUATING THE STUDENT
ACTIVITY METER:
TWO CASE STUDIES.
Sten Govaerts, Katrien Verbert & Erik Duval
http://www.role-project.eu
1
Thursday 8 December 11

2. OVERVIEW
• Problem statement
• Objectives
• Design & Implementation
• Evaluations
• Conclusion & Future work
2
Thursday 8 December 11

3. PROBLEM
3
Thursday 8 December 11

4. PROBLEM
3
Thursday 8 December 11

5. OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learningresource use and
recommendations
4
Thursday 8 December 11

6. STUDENT ACTIVITY METER
(SAM): DEMO.
5
Thursday 8 December 11

7. THREE EVALUATIONS
• with CS students
• ICWL 2010 paper: “Visualizing Activities for
Self-reflection and Awareness”
• with CGIAR courses and
teachers
• with Learning and Knowledge
Analytics course participants
6
Thursday 8 December 11

8. 2010 PAPER
• usability and user satisfaction evaluation
with 12 CS students
• in general, people understand the
visualizations well!
• solved some issues and requests since
Thursday 8 December 11

9. 2010 PAPER
• usability and user satisfaction evaluation
with 12 CS students
• in general, people understand the
visualizations well!
• solved some issues and requests since
Thursday 8 December 11

10. 2010 PAPER
• usability and user satisfaction evaluation
with 12 CS students
• in general, people understand the
visualizations well!
• solved some issues and requests since
Thursday 8 December 11

11. 2010 PAPER
• usability and user satisfaction evaluation
with 12 CS students
• in general, people understand the
visualizations well!
• solved some issues and requests since
Thursday 8 December 11

12. METHODOLOGY.
• design-based research: give students
& teachers a tool and evaluate
• usability studies
• paper mockups
• surveys
8
Thursday 8 December 11

13. CASE STUDY
• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s.
• An online survey about usefulness, teacher issues and how
the tool can resolve these.
• on average: 40 mins are spent using SAM.
9
Thursday 8 December 11

14. CGIAR CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ?
Being aware of what students are doing ✔ ✔
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ✔
Knowing how and when online tools have been used ? ✔
Finding the students who are not doing well ✔ ?
Finding the best students ? ?
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ✔ ? 10
Thursday 8 December 11

15. CGIAR CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ?
Being aware of what students are doing ✔ ✔
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ✔
Knowing how and when online tools have been used ? ✔
Finding the students who are not doing well ✔ ?
Finding the best students ? ?
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ✔ ? 10
Thursday 8 December 11

16. CGIAR CASE STUDY
Bar chart ✔
Parallel coordinates ✔
Line chart ✔
11
Thursday 8 December 11

17. CGIAR CASE STUDY
12
Thursday 8 December 11

18. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1
comparison tool use for course more metrics
design optimization wanted
12
Thursday 8 December 11

19. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1
comparison tool use for course more metrics
design optimization wanted
12
Thursday 8 December 11

20. CASE STUDY
• open course on learning and knowledge analytics, http://bit.ly/
dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
13
Thursday 8 December 11

21. CASE STUDY
• open course on learning and knowledge analytics, http://bit.ly/
dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
13
Thursday 8 December 11

22. LAK CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ✔
Being aware of what students are doing ✔ ?
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ?
Knowing how and when online tools have been used ✗ ?
Finding the students who are not doing well ✔ ?
Finding the best students ? ✗
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ? ? 14
Thursday 8 December 11

23. LAK CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ✔
Being aware of what students are doing ✔ ?
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ?
Knowing how and when online tools have been used ✗ ?
Finding the students who are not doing well ✔ ?
Finding the best students ? ✗
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ? ? 14
Thursday 8 December 11

24. LAK CASE STUDY
Bar chart ?
Parallel coordinates ?
Line chart ?
15
Thursday 8 December 11

25. LAK CASE STUDY
16
Thursday 8 December 11

26. LAK CASE STUDY
more
metrics
verify the status of the
classroom activity
chronological course dwell time
self-reflection to measure
progress and increase
motivation find students experiencing
problems and low
more data engagement
16
Thursday 8 December 11

27. LAK CASE STUDY
more
metrics
verify the status of the
classroom activity
chronological course dwell time
self-reflection to measure
progress and increase
motivation find students experiencing
problems and low
more data engagement
16
Thursday 8 December 11

28. IDEA’S FROM EXPERTS
the used resource types
detailed information per student
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
17
Thursday 8 December 11

29. IDEA’S FROM EXPERTS
2 the used resource types
5 detailed information per student
4 detailed information of 2 students
3 detailed usage stats of resources
1 stats or vis. on content creation
17
Thursday 8 December 11

30. CONTINUED WORK...
18
Thursday 8 December 11

31. CONTINUED WORK...
18
Thursday 8 December 11

32. FUTURE WORK
• extend SAM with feedback of
experts
• evaluate the usefulness in a larger
group of students
• evaluate with teachers during and
after a course.
19
Thursday 8 December 11

33. YOUR DATA IN SAM ?!
ask me!
sten.govaerts@cs.kuleuven.be
20
Thursday 8 December 11

34. 2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 21
Thursday 8 December 11