THE STUDENT ACTIVITYMETER FOR AWARENESS AND     SELF-REFLECTIONSten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo....
our team:Human-Computer Interaction• technology enhanced learning• music• research 2.0    2
PROBLEMhttp://www.ypfp.org/classroom                                     3
PROBLEMhttp://www.ypfp.org/classroom                                     3
OBJECTIVES• self-monitoring   for learners                                   http://free-extras.com/images/obama_hope_prog...
OBJECTIVES• self-monitoring   for learners• awareness   for teachers                                   http://free-extras....
OBJECTIVES• self-monitoring   for learners• awareness    for teachers• time   tracking                                   h...
OBJECTIVES• self-monitoring   for learners• awareness    for teachers• time   tracking• learning       resource use and re...
OBJECTIVES• self-monitoring     for learners• awareness     for teachers• time   tracking• learning       resource use and...
STUDENT ACTIVITY METER     (SAM): DEMO.        http://dilbert.com/strips/comic/2000-02-24/                                ...
http://research.uow.edu.au/learningnetworks/seeing/snapp/index.htmlSNAPP
GISMO   http://www.noe-kaleidoscope.org/pub/researcher/resources/video/aied/mazza.html
The Participation Tool   http://www.cs.uu.nl/docs/vakken/uem/Janssen.pdf
METHODOLOGY.• design-basedresearch [Obrenović 2011]: design a tool, give students & teachers and evaluate• iterative   pro...
iter.     evaluation goal        methodology         demo-graphics        data        usability, satisfaction, task-based ...
11
11
demogra    evaluation         design                                                  negative             positive       ...
ITERATION TWO &                    THREE• CGIAR: 19   teachers - LAK: 12• An online survey about usefulness, teacher issue...
ITERATION TWO &                    THREE• CGIAR: 19   teachers - LAK: 12• An online survey about usefulness, teacher issue...
CGIAR              LAK                                    1     2   3    4   issue for2 add- issue for add-               ...
CGIAR              LAK                                                   issue for add- issue for add-                    ...
CGIAR              LAK                                                   issue for add- issue for add-                    ...
CGIAR CASE STUDY                   15
CGIAR CASE STUDY want to search    detect outliers  for students                            more details ongood indicator ...
CGIAR CASE STUDY want to search    detect outliers  for students                            more details ongood indicator ...
LAK CASE STUDY                                                                       16      http://maranash.blogspot.com/...
LAK CASE STUDY                                    find students experiencing                                       problems...
LAK CASE STUDY                                    find students experiencing                                       problems...
demogra    evaluation        design                                                  negative                 positive    ...
ITERATION FOUR•a   CS course on C++ programming• 11   people: 7 teachers, 2 TA’s & 1 course planner• richer   data set: tr...
ITERATION FOUR•a   CS course on C++ programming• 11   people: 7 teachers, 2 TA’s & 1 course planner• richer   data set: tr...
ACHIEVABLE TASKSconflicting visions on get a time                                   identify students doing students doing ...
ACHIEVABLE TASKSconflicting visions on get a time                                   identify students doing students doing ...
USABILITY & SATISFACTION• average   SUS score: 69,69%                                [Bangor et al. 2008]                 ...
USABILITY & SATISFACTION• average   SUS score: 69,69%                  all: want to continue using it                  9/1...
demo-      evaluation           design                                                      negative                 posit...
CONCLUSION•   SAM is perceived useful and contributes to better    awareness.                                             ...
CONCLUSION•   SAM is perceived useful and contributes to better    awareness.•   SAM enables:                             ...
CONCLUSION•   SAM is perceived useful and contributes to better    awareness.•   SAM enables:    • to   detect potential s...
CONCLUSION•   SAM is perceived useful and contributes to better    awareness.•   SAM enables:    • to detect potential stu...
CONCLUSION•   SAM is perceived useful and contributes to better    awareness.•   SAM enables:    • to detect potential stu...
CONCLUSION•   SAM is perceived useful and contributes to better    awareness.•   SAM enables:    • to detect potential stu...
CONCLUSION•   SAM is perceived useful and contributes to better    awareness.•   SAM enables:    • to detect potential stu...
LESSONS LEARNED•   worked well:                                                                        23                 ...
LESSONS LEARNED•   worked well:    • Evaluate   visualisations iteratively and early                                      ...
LESSONS LEARNED•   worked well:    • Evaluate visualisations iteratively and early    • Use real-world settings           ...
LESSONS LEARNED•   worked well:    • Evaluate  visualisations iteratively and early    • Use real-world settings    • Visu...
LESSONS LEARNED•   worked well:    • Evaluate  visualisations iteratively and early    • Use real-world settings    • Visu...
LESSONS LEARNED•   worked well:    • Evaluate  visualisations iteratively and early    • Use real-world settings    • Visu...
LESSONS LEARNED•   worked well:    • Evaluate  visualisations iteratively and early    • Use real-world settings    • Visu...
LESSONS LEARNED•   worked well:    • Evaluate  visualisations iteratively and early    • Use real-world settings    • Visu...
http://www.fitbit.com/        http://media.cnbc.com/i/CNBC/Sections/    News_And_Analysis/_News/_SPECIAL_REPORTS/       Bet...
LAK13: Learning Analytics and Knowledge Conf.  8-12 April 2013           Leuven, Belgium            http://lakconference.o...
2                contact: sten@cs.kuleuven.beslides will appear on http://www.slideshare.net/stengovaerts                 ...
If you are interested in visualisations and                            2  tabletops, we have a PhD position available.    ...
Upcoming SlideShare
Loading in...5
×

The Student Activity Meter for Awareness and Self-reflection

995

Published on

presentation of the student activity meter at SigCHI 2012 in Austin, TX.

Published in: Education, Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
995
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
26
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

The Student Activity Meter for Awareness and Self-reflection

  1. 1. THE STUDENT ACTIVITYMETER FOR AWARENESS AND SELF-REFLECTIONSten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo. SigCHI 2012, Long Case Study. 1
  2. 2. our team:Human-Computer Interaction• technology enhanced learning• music• research 2.0 2
  3. 3. PROBLEMhttp://www.ypfp.org/classroom 3
  4. 4. PROBLEMhttp://www.ypfp.org/classroom 3
  5. 5. OBJECTIVES• self-monitoring for learners http://free-extras.com/images/obama_hope_progress-2848.htm 4
  6. 6. OBJECTIVES• self-monitoring for learners• awareness for teachers http://free-extras.com/images/obama_hope_progress-2848.htm 4
  7. 7. OBJECTIVES• self-monitoring for learners• awareness for teachers• time tracking http://free-extras.com/images/obama_hope_progress-2848.htm 4
  8. 8. OBJECTIVES• self-monitoring for learners• awareness for teachers• time tracking• learning resource use and recommendations http://free-extras.com/images/obama_hope_progress-2848.htm 4
  9. 9. OBJECTIVES• self-monitoring for learners• awareness for teachers• time tracking• learning resource use and recommendations http://free-extras.com/images/obama_hope_progress-2848.htm• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011] 4
  10. 10. STUDENT ACTIVITY METER (SAM): DEMO. http://dilbert.com/strips/comic/2000-02-24/ 5
  11. 11. http://research.uow.edu.au/learningnetworks/seeing/snapp/index.htmlSNAPP
  12. 12. GISMO http://www.noe-kaleidoscope.org/pub/researcher/resources/video/aied/mazza.html
  13. 13. The Participation Tool http://www.cs.uu.nl/docs/vakken/uem/Janssen.pdf
  14. 14. METHODOLOGY.• design-basedresearch [Obrenović 2011]: design a tool, give students & teachers and evaluate• iterative process of design & evaluation• paper mockups• usability studies• surveys on usefulness http://funnytogo.com/pictures/evolution/timeline.htm 9
  15. 15. iter. evaluation goal methodology demo-graphics data usability, satisfaction, task-based interview 12 CS students timetracking I. preliminary with think-aloud, SUS at KULeuven via Twitter usefulness & MSDT assessing teacher online survey with 19 teachers & II. needs, use & Likert items, yes/no TA’s with Moodle logs usefulness and open questions CGIAR courses assessing teacher online survey with 12 participants needs, expert III. Likert items, yes/no from LAK Moodle logs feedback, use & and open questions MOOC usefulness structured face-to-face 11 teachers & VM with use, usefulness &IV. interview with tasks & TA’s at Carlos III extensive satisfaction open questions Uni. Madrid tool tracking 10
  16. 16. 11
  17. 17. 11
  18. 18. demogra evaluation design negative positive phics goal changes •↑learnability usability, •↓errors 12 CS satisfaction, small usabilityI. 1st iteration •good satisfaction students preliminary issues •usefulness usefulness positive
  19. 19. ITERATION TWO & THREE• CGIAR: 19 teachers - LAK: 12• An online survey about usefulness, teacher issues and how the tool can resolve these. 13
  20. 20. ITERATION TWO & THREE• CGIAR: 19 teachers - LAK: 12• An online survey about usefulness, teacher issues and how the tool can resolve these. 13
  21. 21. CGIAR LAK 1 2 3 4 issue for2 add- issue for add- 5 1 3 4 5 teacher ressed teacher ressed Provide feedback to theProvide feedback to the students students ✔ ? ✔ ✔ Being aware of whatBeing aware of whatdoing students are students are doing ✔ ✔ ✔ ? ✗ ✗ Knowing about collaborationKnowing and communication * and communication about collaboration ✔ ✔ ? Knowing which documentsKnowing are used and how much used and how much which documents are ✔ ✔ ✔ ? ✗ ? Knowing how and whenKnowing online and whenbeen used have been used how tools have online tools ✔ Finding the students who are not doing wellFinding the students who are not doing well ✔ ? ✔ ? Finding the best studentsFinding the best students Knowing how much time ? ? ? ✗ ? ✔ ? students spentKnowing Knowing if externalstudents how much time learning spent ✔ ? ? ? resources are usedKnowing if external learning resources areIssues used ✔Addressed by SAM 14
  22. 22. CGIAR LAK issue for add- issue for add- teacher ressed teacher ressedProvide feedback to the students ✔ ? ✔ ✔Being aware of what students are doing ✔ ✔ ✔ ?Knowing about collaboration and communication ✔ ✗ ✔ ✗Knowing which documents are used and how much ✔ ✔ ✔ ?Knowing how and when online tools have been used ? ✔ ✗ ?Finding the students who are not doing well ✔ ? ✔ ?Finding the best students ? ? ? ✗Knowing how much time students spent ? ✔ ? ✔Knowing if external learning resources are used ✔ ? ? ? 14
  23. 23. CGIAR LAK issue for add- issue for add- teacher ressed teacher ressedProvide feedback to the students ✔ ? ✔ ✔Being aware of what students are doing ✔ ✔ ✔ ?Knowing about collaboration and communication ✔ ✗ ✔ ✗Knowing which documents are used and how much ✔ ✔ ✔ ?Knowing how and when online tools have been used ? ✔ ✗ ?Finding the students who are not doing well ✔ ? ✔ ?Finding the best students ? ? ? ✗Knowing how much time students spent ? ✔ ? ✔Knowing if external learning resources are used ✔ ? ? ? 14
  24. 24. CGIAR CASE STUDY 15
  25. 25. CGIAR CASE STUDY want to search detect outliers for students more details ongood indicator for effort student understand the workload progress evolution compare students obtain course overview increase awarenesswant better 1 to 1 comparison tool use for course more metrics design optimization wanted 15
  26. 26. CGIAR CASE STUDY want to search detect outliers for students more details ongood indicator for effort student understand the workload progress evolution compare students obtain course overview increase awarenesswant better 1 to 1 comparison tool use for course more metrics design optimization wanted 15
  27. 27. LAK CASE STUDY 16 http://maranash.blogspot.com/2011/01/conference-shout-out.html
  28. 28. LAK CASE STUDY find students experiencing problems and low verify the status of the engagement classroom activity more chronological course dwell time metricsself-reflection to measure progress and increase motivation more data 16 http://maranash.blogspot.com/2011/01/conference-shout-out.html
  29. 29. LAK CASE STUDY find students experiencing problems and low verify the status of the engagement classroom activity more chronological course dwell time metricsself-reflection to measure progress and increase motivation more data 16 http://maranash.blogspot.com/2011/01/conference-shout-out.html
  30. 30. demogra evaluation design negative positive phics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errorsI. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive assessing •provides awareness 19 resource teacher •all vis. usefulII. teachers help function recomm. not needs, use & •many uses & TA’s useful usefulness •90% want it •provides awareness assessing re-orderable most and feedback 12 teacher parallel addressed •many usesIII. participan needs, expert coordinates needs are •66% want it ts feedback, use with indecisive •recomm. can be & usefulness histograms useful
  31. 31. ITERATION FOUR•a CS course on C++ programming• 11 people: 7 teachers, 2 TA’s & 1 course planner• richer data set: tracking from programming environment• qualitative study using a structured face-2-face interview with 25 open questions. 18
  32. 32. ITERATION FOUR•a CS course on C++ programming• 11 people: 7 teachers, 2 TA’s & 1 course planner• richer data set: tracking from programming environment• qualitative study using a structured face-2-face interview with 25 open questions. 18
  33. 33. ACHIEVABLE TASKSconflicting visions on get a time identify students doing students doing well summary well and at risk and at risk better course monitor tool and overview resource use understand time spent better detect students course planning for assistance http://www.tdpri.com/forum/telecaster-discussion- forum/48431-ic503z-cap-52-ri-blackguard-book.html 19
  34. 34. ACHIEVABLE TASKSconflicting visions on get a time identify students doing students doing well summary well and at risk and at risk better course monitor tool and overview resource use understand time spent better detect students course planning for assistance http://www.tdpri.com/forum/telecaster-discussion- forum/48431-ic503z-cap-52-ri-blackguard-book.html 19
  35. 35. USABILITY & SATISFACTION• average SUS score: 69,69% [Bangor et al. 2008] 20
  36. 36. USABILITY & SATISFACTION• average SUS score: 69,69% all: want to continue using it 9/11: give it to students [Bangor et al. 2008] 20
  37. 37. demo- evaluation design negative positive graphics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errorsI. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive •provides awareness 19 assessing teacher resource •all vis. usefulII. teachers needs, use & help function recomm. not •many uses & TA’s usefulness useful •90% want it •provides awareness and assessing teacher most 12 re-orderable feedback needs, expert addressedIII. participan PC with •many uses feedback, use & needs are ts histograms •66% want it usefulness indecisive •recomm. can be useful filter & •provides time overview search, conflicting •provides course 11 icons, visions of overview use, usefulnessIV. teacher zooming in students •PC assist with detecting & satisfaction s & TA’s line chart, doing well problems editing PC or at risk •many uses & insights axes •100% want it
  38. 38. CONCLUSION• SAM is perceived useful and contributes to better awareness. 22
  39. 39. CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: 22
  40. 40. CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk 22
  41. 41. CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview 22
  42. 42. CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview • understanding student time spending 22
  43. 43. CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview • understanding student time spending • to find a wide variety of new insights 22
  44. 44. CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview • understanding student time spending • to find a wide variety of new insights• almost all want to continue using SAM 22
  45. 45. LESSONS LEARNED• worked well: 23 http://slurmed.com/fanart/dexus5/005_professor.png
  46. 46. LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early 23 http://slurmed.com/fanart/dexus5/005_professor.png
  47. 47. LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings 23 http://slurmed.com/fanart/dexus5/005_professor.png
  48. 48. LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness 23 http://slurmed.com/fanart/dexus5/005_professor.png
  49. 49. LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: 23 http://slurmed.com/fanart/dexus5/005_professor.png
  50. 50. LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: • finding real-world settings 23 http://slurmed.com/fanart/dexus5/005_professor.png
  51. 51. LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: • finding real-world settings • getting tracking data 23 http://slurmed.com/fanart/dexus5/005_professor.png
  52. 52. LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: • finding real-world settings • getting tracking data• longitudinal study would be useful 23 http://slurmed.com/fanart/dexus5/005_professor.png
  53. 53. http://www.fitbit.com/ http://media.cnbc.com/i/CNBC/Sections/ News_And_Analysis/_News/_SPECIAL_REPORTS/ BetterYourBusiness_2012/BYB-Digital%20- http://quantifiedself.com/ %20April/Slideshows/Digital_Appliances_ab/10- Smart-Dig-Appliances-Wifi-scale-.jpghttp://nikerunning.nike.com/nikeos/p/nikeplus/en_EMEA/sportband 24
  54. 54. LAK13: Learning Analytics and Knowledge Conf. 8-12 April 2013 Leuven, Belgium http://lakconference.org/
  55. 55. 2 contact: sten@cs.kuleuven.beslides will appear on http://www.slideshare.net/stengovaerts @erikduvalslides will appear on http://www.slideshare.net/stengovaerts 26
  56. 56. If you are interested in visualisations and 2 tabletops, we have a PhD position available. contact: sten@cs.kuleuven.beslides will appear on http://www.slideshare.net/stengovaerts @erikduvalslides will appear on http://www.slideshare.net/stengovaerts 26
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×