• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
The Student Activity Meter for Awareness and Self-reflection
 

The Student Activity Meter for Awareness and Self-reflection

on

  • 870 views

presentation of the student activity meter at SigCHI 2012 in Austin, TX.

presentation of the student activity meter at SigCHI 2012 in Austin, TX.

Statistics

Views

Total Views
870
Views on SlideShare
869
Embed Views
1

Actions

Likes
1
Downloads
6
Comments
0

1 Embed 1

https://twitter.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    The Student Activity Meter for Awareness and Self-reflection The Student Activity Meter for Awareness and Self-reflection Presentation Transcript

    • THE STUDENT ACTIVITYMETER FOR AWARENESS AND SELF-REFLECTIONSten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo. SigCHI 2012, Long Case Study. 1
    • our team:Human-Computer Interaction• technology enhanced learning• music• research 2.0 2
    • PROBLEMhttp://www.ypfp.org/classroom 3
    • PROBLEMhttp://www.ypfp.org/classroom 3
    • OBJECTIVES• self-monitoring for learners http://free-extras.com/images/obama_hope_progress-2848.htm 4
    • OBJECTIVES• self-monitoring for learners• awareness for teachers http://free-extras.com/images/obama_hope_progress-2848.htm 4
    • OBJECTIVES• self-monitoring for learners• awareness for teachers• time tracking http://free-extras.com/images/obama_hope_progress-2848.htm 4
    • OBJECTIVES• self-monitoring for learners• awareness for teachers• time tracking• learning resource use and recommendations http://free-extras.com/images/obama_hope_progress-2848.htm 4
    • OBJECTIVES• self-monitoring for learners• awareness for teachers• time tracking• learning resource use and recommendations http://free-extras.com/images/obama_hope_progress-2848.htm• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011] 4
    • STUDENT ACTIVITY METER (SAM): DEMO. http://dilbert.com/strips/comic/2000-02-24/ 5
    • http://research.uow.edu.au/learningnetworks/seeing/snapp/index.htmlSNAPP
    • GISMO http://www.noe-kaleidoscope.org/pub/researcher/resources/video/aied/mazza.html
    • The Participation Tool http://www.cs.uu.nl/docs/vakken/uem/Janssen.pdf
    • METHODOLOGY.• design-basedresearch [Obrenović 2011]: design a tool, give students & teachers and evaluate• iterative process of design & evaluation• paper mockups• usability studies• surveys on usefulness http://funnytogo.com/pictures/evolution/timeline.htm 9
    • iter. evaluation goal methodology demo-graphics data usability, satisfaction, task-based interview 12 CS students timetracking I. preliminary with think-aloud, SUS at KULeuven via Twitter usefulness & MSDT assessing teacher online survey with 19 teachers & II. needs, use & Likert items, yes/no TA’s with Moodle logs usefulness and open questions CGIAR courses assessing teacher online survey with 12 participants needs, expert III. Likert items, yes/no from LAK Moodle logs feedback, use & and open questions MOOC usefulness structured face-to-face 11 teachers & VM with use, usefulness &IV. interview with tasks & TA’s at Carlos III extensive satisfaction open questions Uni. Madrid tool tracking 10
    • 11
    • 11
    • demogra evaluation design negative positive phics goal changes •↑learnability usability, •↓errors 12 CS satisfaction, small usabilityI. 1st iteration •good satisfaction students preliminary issues •usefulness usefulness positive
    • ITERATION TWO & THREE• CGIAR: 19 teachers - LAK: 12• An online survey about usefulness, teacher issues and how the tool can resolve these. 13
    • ITERATION TWO & THREE• CGIAR: 19 teachers - LAK: 12• An online survey about usefulness, teacher issues and how the tool can resolve these. 13
    • CGIAR LAK 1 2 3 4 issue for2 add- issue for add- 5 1 3 4 5 teacher ressed teacher ressed Provide feedback to theProvide feedback to the students students ✔ ? ✔ ✔ Being aware of whatBeing aware of whatdoing students are students are doing ✔ ✔ ✔ ? ✗ ✗ Knowing about collaborationKnowing and communication * and communication about collaboration ✔ ✔ ? Knowing which documentsKnowing are used and how much used and how much which documents are ✔ ✔ ✔ ? ✗ ? Knowing how and whenKnowing online and whenbeen used have been used how tools have online tools ✔ Finding the students who are not doing wellFinding the students who are not doing well ✔ ? ✔ ? Finding the best studentsFinding the best students Knowing how much time ? ? ? ✗ ? ✔ ? students spentKnowing Knowing if externalstudents how much time learning spent ✔ ? ? ? resources are usedKnowing if external learning resources areIssues used ✔Addressed by SAM 14
    • CGIAR LAK issue for add- issue for add- teacher ressed teacher ressedProvide feedback to the students ✔ ? ✔ ✔Being aware of what students are doing ✔ ✔ ✔ ?Knowing about collaboration and communication ✔ ✗ ✔ ✗Knowing which documents are used and how much ✔ ✔ ✔ ?Knowing how and when online tools have been used ? ✔ ✗ ?Finding the students who are not doing well ✔ ? ✔ ?Finding the best students ? ? ? ✗Knowing how much time students spent ? ✔ ? ✔Knowing if external learning resources are used ✔ ? ? ? 14
    • CGIAR LAK issue for add- issue for add- teacher ressed teacher ressedProvide feedback to the students ✔ ? ✔ ✔Being aware of what students are doing ✔ ✔ ✔ ?Knowing about collaboration and communication ✔ ✗ ✔ ✗Knowing which documents are used and how much ✔ ✔ ✔ ?Knowing how and when online tools have been used ? ✔ ✗ ?Finding the students who are not doing well ✔ ? ✔ ?Finding the best students ? ? ? ✗Knowing how much time students spent ? ✔ ? ✔Knowing if external learning resources are used ✔ ? ? ? 14
    • CGIAR CASE STUDY 15
    • CGIAR CASE STUDY want to search detect outliers for students more details ongood indicator for effort student understand the workload progress evolution compare students obtain course overview increase awarenesswant better 1 to 1 comparison tool use for course more metrics design optimization wanted 15
    • CGIAR CASE STUDY want to search detect outliers for students more details ongood indicator for effort student understand the workload progress evolution compare students obtain course overview increase awarenesswant better 1 to 1 comparison tool use for course more metrics design optimization wanted 15
    • LAK CASE STUDY 16 http://maranash.blogspot.com/2011/01/conference-shout-out.html
    • LAK CASE STUDY find students experiencing problems and low verify the status of the engagement classroom activity more chronological course dwell time metricsself-reflection to measure progress and increase motivation more data 16 http://maranash.blogspot.com/2011/01/conference-shout-out.html
    • LAK CASE STUDY find students experiencing problems and low verify the status of the engagement classroom activity more chronological course dwell time metricsself-reflection to measure progress and increase motivation more data 16 http://maranash.blogspot.com/2011/01/conference-shout-out.html
    • demogra evaluation design negative positive phics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errorsI. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive assessing •provides awareness 19 resource teacher •all vis. usefulII. teachers help function recomm. not needs, use & •many uses & TA’s useful usefulness •90% want it •provides awareness assessing re-orderable most and feedback 12 teacher parallel addressed •many usesIII. participan needs, expert coordinates needs are •66% want it ts feedback, use with indecisive •recomm. can be & usefulness histograms useful
    • ITERATION FOUR•a CS course on C++ programming• 11 people: 7 teachers, 2 TA’s & 1 course planner• richer data set: tracking from programming environment• qualitative study using a structured face-2-face interview with 25 open questions. 18
    • ITERATION FOUR•a CS course on C++ programming• 11 people: 7 teachers, 2 TA’s & 1 course planner• richer data set: tracking from programming environment• qualitative study using a structured face-2-face interview with 25 open questions. 18
    • ACHIEVABLE TASKSconflicting visions on get a time identify students doing students doing well summary well and at risk and at risk better course monitor tool and overview resource use understand time spent better detect students course planning for assistance http://www.tdpri.com/forum/telecaster-discussion- forum/48431-ic503z-cap-52-ri-blackguard-book.html 19
    • ACHIEVABLE TASKSconflicting visions on get a time identify students doing students doing well summary well and at risk and at risk better course monitor tool and overview resource use understand time spent better detect students course planning for assistance http://www.tdpri.com/forum/telecaster-discussion- forum/48431-ic503z-cap-52-ri-blackguard-book.html 19
    • USABILITY & SATISFACTION• average SUS score: 69,69% [Bangor et al. 2008] 20
    • USABILITY & SATISFACTION• average SUS score: 69,69% all: want to continue using it 9/11: give it to students [Bangor et al. 2008] 20
    • demo- evaluation design negative positive graphics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errorsI. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive •provides awareness 19 assessing teacher resource •all vis. usefulII. teachers needs, use & help function recomm. not •many uses & TA’s usefulness useful •90% want it •provides awareness and assessing teacher most 12 re-orderable feedback needs, expert addressedIII. participan PC with •many uses feedback, use & needs are ts histograms •66% want it usefulness indecisive •recomm. can be useful filter & •provides time overview search, conflicting •provides course 11 icons, visions of overview use, usefulnessIV. teacher zooming in students •PC assist with detecting & satisfaction s & TA’s line chart, doing well problems editing PC or at risk •many uses & insights axes •100% want it
    • CONCLUSION• SAM is perceived useful and contributes to better awareness. 22
    • CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: 22
    • CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk 22
    • CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview 22
    • CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview • understanding student time spending 22
    • CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview • understanding student time spending • to find a wide variety of new insights 22
    • CONCLUSION• SAM is perceived useful and contributes to better awareness.• SAM enables: • to detect potential students doing well and at risk • a better course overview • understanding student time spending • to find a wide variety of new insights• almost all want to continue using SAM 22
    • LESSONS LEARNED• worked well: 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: • finding real-world settings 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: • finding real-world settings • getting tracking data 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • LESSONS LEARNED• worked well: • Evaluate visualisations iteratively and early • Use real-world settings • Visualisations are powerful tool for creating awareness• worked less well: • finding real-world settings • getting tracking data• longitudinal study would be useful 23 http://slurmed.com/fanart/dexus5/005_professor.png
    • http://www.fitbit.com/ http://media.cnbc.com/i/CNBC/Sections/ News_And_Analysis/_News/_SPECIAL_REPORTS/ BetterYourBusiness_2012/BYB-Digital%20- http://quantifiedself.com/ %20April/Slideshows/Digital_Appliances_ab/10- Smart-Dig-Appliances-Wifi-scale-.jpghttp://nikerunning.nike.com/nikeos/p/nikeplus/en_EMEA/sportband 24
    • LAK13: Learning Analytics and Knowledge Conf. 8-12 April 2013 Leuven, Belgium http://lakconference.org/
    • 2 contact: sten@cs.kuleuven.beslides will appear on http://www.slideshare.net/stengovaerts @erikduvalslides will appear on http://www.slideshare.net/stengovaerts 26
    • If you are interested in visualisations and 2 tabletops, we have a PhD position available. contact: sten@cs.kuleuven.beslides will appear on http://www.slideshare.net/stengovaerts @erikduvalslides will appear on http://www.slideshare.net/stengovaerts 26