0
THE STUDENT ACTIVITY     METER FOR AWARENESS AND          SELF-REFLECTION          Sten Govaerts, Katrien Verbert, Erik Du...
http://bit.ly/stenpub                                              2Tuesday 20 March 12
OVERVIEW    • Problem          statement    • Objectives    • Design          & Implementation    • Evaluations    • Concl...
PROBLEM                      http://www.ypfp.org/classroom                                                           4Tues...
PROBLEM                      http://www.ypfp.org/classroom                                                           4Tues...
OBJECTIVES    • self-monitoring              for learners    • awareness              for teachers    • time            tr...
STUDENT ACTIVITY METER                  (SAM): DEMO.                      http://dilbert.com/strips/comic/2000-02-24/     ...
STUDENT ACTIVITY METER                                      7Tuesday 20 March 12
STUDENT ACTIVITY METER                                      8Tuesday 20 March 12
STUDENT ACTIVITY METER                                      9Tuesday 20 March 12
TASK    • How             would you evaluate SAM?        • how/where          do you get data from?        • which        ...
METHODOLOGY.   • design-based   research [Obrenović 2011]: give       students & teachers a tool and evaluate   • iterativ...
iter.           evaluation goal       methodology         demo-graphics      data               usability, satisfaction, t...
ITERATION ONE     • usability      and user satisfaction evaluation     • 12   CS students, using a                   -   ...
LEARNABILITY, ERRORS &                       EFFICIENCY    • in     general, people understand the visualizations well!   ...
LEARNABILITY, ERRORS &                       EFFICIENCY    • in     general, people understand the visualizations well!   ...
LEARNABILITY, ERRORS &                       EFFICIENCY    • in     general, people understand the visualizations well!   ...
LEARNABILITY, ERRORS &                       EFFICIENCY    • in     general, people understand the visualizations well!   ...
LEARNABILITY, ERRORS &                       EFFICIENCY    • in     general, people understand the visualizations well!   ...
USER SATISFACTION    • average         SUS score: 73%                                       [Bangor et al. 2008]Tuesday 20...
USER SATISFACTION    • average         SUS score: 73%                                       [Bangor et al. 2008]Tuesday 20...
USER SATISFACTION    • average         SUS score: 73%                                       [Bangor et al. 2008]Tuesday 20...
USER SATISFACTION    • average         SUS score: 73%                                       [Bangor et al. 2008]Tuesday 20...
demogra     evaluation         design                                                            negative             posi...
ITERATION TWO    • 20       persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s.    • An   online survey about usefulness, ...
CGIAR CASE STUDY                                                  1    2      3   4   5   1     issue for                 ...
CGIAR CASE STUDY                                                       issue for                                          ...
CGIAR CASE STUDY                                                       issue for                                          ...
CGIAR CASE STUDY                                         19Tuesday 20 March 12
CGIAR CASE STUDY        want to search    detect outliers         for students                                 more detail...
CGIAR CASE STUDY        want to search    detect outliers         for students                                 more detail...
demogra      evaluation        design                                                               negative              ...
ITERATION THREE    • opencourse on learning and knowledge analytics, http://bit.ly/       dWYVbX    • 12       visual anal...
ITERATION THREE    • opencourse on learning and knowledge analytics, http://bit.ly/       dWYVbX    • 12       visual anal...
LAK CASE STUDY                                                       issue for                                            ...
LAK CASE STUDY                                                       issue for                                            ...
LAK CASE STUDY                                                       issue for                                            ...
LAK CASE STUDY                                       23Tuesday 20 March 12
LAK CASE STUDY                                                                                             24             ...
LAK CASE STUDY                                                find students experiencing                                   ...
LAK CASE STUDY                                                find students experiencing                                   ...
IDEA’S FROM EXPERTS                  the used resource types                  detailed information per student            ...
IDEA’S FROM EXPERTS          2       the used resource types          5       detailed information per student          4 ...
IDEA’S FROM EXPERTS          2       the used resource types          5       detailed information per student          4 ...
demogra       evaluation          design                                                               negative           ...
ITERATION FOUR    •a      CS course on C++ programming    • 11       people: 7 teachers, 2 TA’s & 1 course planner    • ri...
ITERATION FOUR    •a      CS course on C++ programming    • 11       people: 7 teachers, 2 TA’s & 1 course planner    • ri...
ACHIEVABLE TASKS    conflicting visions on get a time                                       identify students doing     stu...
ACHIEVABLE TASKS    conflicting visions on get a time                                       identify students doing     stu...
USER SATISFACTION    • average         SUS score: 69,69%                                          [Bangor et al. 2008]    ...
USER SATISFACTION    • average         SUS score: 69,69%                            all: want to continue using it        ...
demo-       evaluation           design                                                              negative             ...
CONCLUSION   •   SAM enables:        • to find students doing well and at risk        • a better course overview        • u...
2          slides will appear on http://www.slideshare.net/stengovaerts          slides will appear on http://www.slidesha...
Upcoming SlideShare
Loading in...5
×

The Student Activity Meter for Awareness and Self-reflection

880

Published on

These slides present the iterative design and evaluation the student activity meter.

Presented for students of the KULeuven CHI course of prof. Erik Duval.

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
880
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "The Student Activity Meter for Awareness and Self-reflection"

  1. 1. THE STUDENT ACTIVITY METER FOR AWARENESS AND SELF-REFLECTION Sten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo. SigCHI 2012, Long Case Study. 1Tuesday 20 March 12
  2. 2. http://bit.ly/stenpub 2Tuesday 20 March 12
  3. 3. OVERVIEW • Problem statement • Objectives • Design & Implementation • Evaluations • Conclusion 3Tuesday 20 March 12
  4. 4. PROBLEM http://www.ypfp.org/classroom 4Tuesday 20 March 12
  5. 5. PROBLEM http://www.ypfp.org/classroom 4Tuesday 20 March 12
  6. 6. OBJECTIVES • self-monitoring for learners • awareness for teachers • time tracking • learning resource use and recommendations http://free-extras.com/images/obama_hope_progress-2848.htm • part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011] 5Tuesday 20 March 12
  7. 7. STUDENT ACTIVITY METER (SAM): DEMO. http://dilbert.com/strips/comic/2000-02-24/ 6Tuesday 20 March 12
  8. 8. STUDENT ACTIVITY METER 7Tuesday 20 March 12
  9. 9. STUDENT ACTIVITY METER 8Tuesday 20 March 12
  10. 10. STUDENT ACTIVITY METER 9Tuesday 20 March 12
  11. 11. TASK • How would you evaluate SAM? • how/where do you get data from? • which criteria could you evaluate? • which methods use to evaluate those criteria? • iterative? which iterations? http://callofduty.wikia.com/wiki/Task_Force_141 10Tuesday 20 March 12
  12. 12. METHODOLOGY. • design-based research [Obrenović 2011]: give students & teachers a tool and evaluate • iterative process of design & evaluation • paper mockups • usability studies • surveys on usefulness http://funnytogo.com/pictures/evolution/timeline.htm 11Tuesday 20 March 12
  13. 13. iter. evaluation goal methodology demo-graphics data usability, satisfaction, task-based interview 12 CS students timetracking I. preliminary with think-aloud, SUS at KULeuven via Twitter usefulness & MSDT assessing teacher online survey with 19 teachers & II. needs, use & Likert items, yes/no TA’s with Moodle logs usefulness and open questions CGIAR courses assessing teacher online survey with 12 participants needs, expert III. Likert items, yes/no from LAK Moodle logs feedback, use & and open questions MOOC usefulness structured face-to-face 11 teachers & VM with use, usefulness & IV. interview with tasks & TA’s at Carlos III extensive satisfaction open questions Uni. Madrid tool tracking 12Tuesday 20 March 12
  14. 14. ITERATION ONE • usability and user satisfaction evaluation • 12 CS students, using a - based time tracker •2 evaluation sessions: • task based interview with think aloud (after 1 week of tracking) • user satisfaction (SUS & MSDT) (after 1 month)Tuesday 20 March 12
  15. 15. LEARNABILITY, ERRORS & EFFICIENCY • in general, people understand the visualizations well! • some issues were uncovered...Tuesday 20 March 12
  16. 16. LEARNABILITY, ERRORS & EFFICIENCY • in general, people understand the visualizations well! • some issues were uncovered...Tuesday 20 March 12
  17. 17. LEARNABILITY, ERRORS & EFFICIENCY • in general, people understand the visualizations well! • some issues were uncovered...Tuesday 20 March 12
  18. 18. LEARNABILITY, ERRORS & EFFICIENCY • in general, people understand the visualizations well! • some issues were uncovered...Tuesday 20 March 12
  19. 19. LEARNABILITY, ERRORS & EFFICIENCY • in general, people understand the visualizations well! • some issues were uncovered...Tuesday 20 March 12
  20. 20. USER SATISFACTION • average SUS score: 73% [Bangor et al. 2008]Tuesday 20 March 12
  21. 21. USER SATISFACTION • average SUS score: 73% [Bangor et al. 2008]Tuesday 20 March 12
  22. 22. USER SATISFACTION • average SUS score: 73% [Bangor et al. 2008]Tuesday 20 March 12
  23. 23. USER SATISFACTION • average SUS score: 73% [Bangor et al. 2008]Tuesday 20 March 12
  24. 24. demogra evaluation design negative positive phics goal changes •↑learnability usability, •↓errors 12 CS satisfaction, small usability I. 1st iteration •good satisfaction students preliminary issues •usefulness usefulness positiveTuesday 20 March 12
  25. 25. ITERATION TWO • 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s. • An online survey about usefulness, teacher issues and how the tool can resolve these. • on average: 40 mins are spent using SAM. 17Tuesday 20 March 12
  26. 26. CGIAR CASE STUDY 1 2 3 4 5 1 issue for 2 3 4 5 teacher addressed Provide feedback to the Provide students to the students feedback ✔ ? ✔ ✔ Being aware of what Being aware of are doing students what students are doing ✗ Knowing about collaboration Knowing about collaboration and communication and communication * ✔ Knowing which documents Knowing which and how muchare used and how much are used documents ✔ ✔ ? Knowing how and when Knowing howtools have been used tools have been used online and when online ✔ Finding the students who are not doing well Finding the students who are not doing well ✔ ? Finding the best students Finding the besthow much time Knowing students ? ? students spent Knowing how much time students Knowing if external learning spent ? ✔ ? resources are used Knowing if external learning resources Issues are used ✔ Addressed by SAM 18Tuesday 20 March 12
  27. 27. CGIAR CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ? Being aware of what students are doing ✔ ✔ Knowing about collaboration and communication ✔ ✗ Knowing which documents are used and how much ✔ ✔ Knowing how and when online tools have been used ? ✔ Finding the students who are not doing well ✔ ? Finding the best students ? ? Knowing how much time students spent ? ✔ Knowing if external learning resources are used ✔ ? 18Tuesday 20 March 12
  28. 28. CGIAR CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ? Being aware of what students are doing ✔ ✔ Knowing about collaboration and communication ✔ ✗ Knowing which documents are used and how much ✔ ✔ Knowing how and when online tools have been used ? ✔ Finding the students who are not doing well ✔ ? Finding the best students ? ? Knowing how much time students spent ? ✔ Knowing if external learning resources are used ✔ ? 18Tuesday 20 March 12
  29. 29. CGIAR CASE STUDY 19Tuesday 20 March 12
  30. 30. CGIAR CASE STUDY want to search detect outliers for students more details on good indicator for effort student understand the workload progress evolution compare students obtain course overview increase awareness want better 1 to 1 comparison tool use for course more metrics design optimization wanted 19Tuesday 20 March 12
  31. 31. CGIAR CASE STUDY want to search detect outliers for students more details on good indicator for effort student understand the workload progress evolution compare students obtain course overview increase awareness want better 1 to 1 comparison tool use for course more metrics design optimization wanted 19Tuesday 20 March 12
  32. 32. demogra evaluation design negative positive phics goal changes usability, •↑learnability 12 CS satisfaction, •↓errors I. 1st iteration small usability issues students preliminary •good satisfaction usefulness •usefulness positive •provides assessing 19 resource awareness teacher II. teachers help function recomm. not •all vis. useful needs, use & & TA’s useful •many uses usefulness •90% wants itTuesday 20 March 12
  33. 33. ITERATION THREE • opencourse on learning and knowledge analytics, http://bit.ly/ dWYVbX • 12 visual analytics enthousiasts + experts (who also teach) • almost identical survey to CGIAR case. 21Tuesday 20 March 12
  34. 34. ITERATION THREE • opencourse on learning and knowledge analytics, http://bit.ly/ dWYVbX • 12 visual analytics enthousiasts + experts (who also teach) • almost identical survey to CGIAR case. 21Tuesday 20 March 12
  35. 35. LAK CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ✔ Being aware of what students are doing ✔ ? Knowing about collaboration and communication ✔ ✗ Knowing which documents are used and how much ✔ ? Knowing how and when online tools have been used ✗ ? Finding the students who are not doing well ✔ ? Finding the best students ? ✗ Knowing how much time students spent ? ✔ Knowing if external learning resources are used ? ? 22Tuesday 20 March 12
  36. 36. LAK CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ✔ Being aware of what students are doing ✔ ? Knowing about collaboration and communication ✔ ✗ Knowing which documents are used and how much ✔ ? Knowing how and when online tools have been used ✗ ? Finding the students who are not doing well ✔ ? Finding the best students ? ✗ Knowing how much time students spent ? ✔ Knowing if external learning resources are used ? ? 22Tuesday 20 March 12
  37. 37. LAK CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ✔ Being aware of what students are doing ✔ ? Knowing about collaboration and communication ✔ ✗ Knowing which documents are used and how much ✔ ? Knowing how and when online tools have been used ✗ ? Finding the students who are not doing well ✔ ? Finding the best students ? ✗ Knowing how much time students spent ? ✔ Knowing if external learning resources are used ? ? 22Tuesday 20 March 12
  38. 38. LAK CASE STUDY 23Tuesday 20 March 12
  39. 39. LAK CASE STUDY 24 http://maranash.blogspot.com/2011/01/conference-shout-out.htmlTuesday 20 March 12
  40. 40. LAK CASE STUDY find students experiencing problems and low verify the status of the engagement classroom activity more chronological course dwell time metrics self-reflection to measure progress and increase motivation more data 24 http://maranash.blogspot.com/2011/01/conference-shout-out.htmlTuesday 20 March 12
  41. 41. LAK CASE STUDY find students experiencing problems and low verify the status of the engagement classroom activity more chronological course dwell time metrics self-reflection to measure progress and increase motivation more data 24 http://maranash.blogspot.com/2011/01/conference-shout-out.htmlTuesday 20 March 12
  42. 42. IDEA’S FROM EXPERTS the used resource types detailed information per student detailed information of 2 students detailed usage stats of resources stats or vis. on content creation 25Tuesday 20 March 12
  43. 43. IDEA’S FROM EXPERTS 2 the used resource types 5 detailed information per student 4 detailed information of 2 students 3 detailed usage stats of resources 1 stats or vis. on content creation 25Tuesday 20 March 12
  44. 44. IDEA’S FROM EXPERTS 2 the used resource types 5 detailed information per student 4 detailed information of 2 students 3 detailed usage stats of resources 1 stats or vis. on content creation 25Tuesday 20 March 12
  45. 45. demogra evaluation design negative positive phics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errors I. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive •provides awareness assessing teacher resource 19 teachers •all vis. useful II. needs, use & help function recomm. not & TA’s •many uses usefulness useful •90% want it •provides awareness assessing re-orderable most and feedback 12 teacher parallel addressed •many uses III. participan needs, expert coordinates needs are •66% want it ts feedback, use with indecisive •recomm. can be & usefulness histograms usefulTuesday 20 March 12
  46. 46. ITERATION FOUR •a CS course on C++ programming • 11 people: 7 teachers, 2 TA’s & 1 course planner • richer data set: tracking from programming environment • qualitativestudy using a structured face-2-face interview with 25 open questions. 27Tuesday 20 March 12
  47. 47. ITERATION FOUR •a CS course on C++ programming • 11 people: 7 teachers, 2 TA’s & 1 course planner • richer data set: tracking from programming environment • qualitativestudy using a structured face-2-face interview with 25 open questions. 27Tuesday 20 March 12
  48. 48. ACHIEVABLE TASKS conflicting visions on get a time identify students doing students doing well summary well and at risk and at risk better course monitor tool and overview resource use understand time spent better detect students course planning for assistance http://www.tdpri.com/forum/telecaster-discussion- forum/48431-ic503z-cap-52-ri-blackguard-book.html 28Tuesday 20 March 12
  49. 49. ACHIEVABLE TASKS conflicting visions on get a time identify students doing students doing well summary well and at risk and at risk better course monitor tool and overview resource use understand time spent better detect students course planning for assistance http://www.tdpri.com/forum/telecaster-discussion- forum/48431-ic503z-cap-52-ri-blackguard-book.html 28Tuesday 20 March 12
  50. 50. USER SATISFACTION • average SUS score: 69,69% [Bangor et al. 2008] 29Tuesday 20 March 12
  51. 51. USER SATISFACTION • average SUS score: 69,69% all: want to continue using it 9/11: give it to students [Bangor et al. 2008] 29Tuesday 20 March 12
  52. 52. demo- evaluation design negative positive graphics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errors I. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive •provides awareness 19 assessing teacher resource •all vis. useful II. teachers needs, use & help function recomm. not •many uses & TA’s usefulness useful •90% want it •provides awareness and assessing teacher most 12 re-orderable feedback needs, expert addressed III. participan PC with •many uses feedback, use & needs are ts histograms •66% want it usefulness indecisive •recomm. can be useful filter & •provides time overview search, conflicting •provides course 11 icons, visions of overview use, usefulness IV. teacher zooming in students •PC assist with detecting & satisfaction s & TA’s line chart, doing well problems editing PC or at risk •many uses & insights axes •100% want itTuesday 20 March 12
  53. 53. CONCLUSION • SAM enables: • to find students doing well and at risk • a better course overview • understanding student time spending • almost all participants want to continue using SAM • SAM enables to find a wide variety of new insights 31Tuesday 20 March 12
  54. 54. 2 slides will appear on http://www.slideshare.net/stengovaerts slides will appear on http://www.slideshare.net/stengovaerts 32Tuesday 20 March 12
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×