The Student Activity Meter for Awareness and Self-reflection

THE STUDENT ACTIVITY METER FOR AWARENESS AND SELF-REFLECTION Sten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo. SigCHI 2012, Long Case Study. 1Tuesday 20 March 12

http://bit.ly/stenpub 2Tuesday 20 March 12

OVERVIEW • Problem statement • Objectives • Design & Implementation • Evaluations • Conclusion 3Tuesday 20 March 12

PROBLEM http://www.ypfp.org/classroom 4Tuesday 20 March 12

OBJECTIVES • self-monitoring for learners • awareness for teachers • time tracking • learning resource use and recommendations http://free-extras.com/images/obama_hope_progress-2848.htm • part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011] 5Tuesday 20 March 12

STUDENT ACTIVITY METER (SAM): DEMO. http://dilbert.com/strips/comic/2000-02-24/ 6Tuesday 20 March 12

STUDENT ACTIVITY METER 7Tuesday 20 March 12

TASK • How would you evaluate SAM? • how/where do you get data from? • which criteria could you evaluate? • which methods use to evaluate those criteria? • iterative? which iterations? http://callofduty.wikia.com/wiki/Task_Force_141 10Tuesday 20 March 12

METHODOLOGY. • design-based research [Obrenović 2011]: give students & teachers a tool and evaluate • iterative process of design & evaluation • paper mockups • usability studies • surveys on usefulness http://funnytogo.com/pictures/evolution/timeline.htm 11Tuesday 20 March 12

iter. evaluation goal methodology demo-graphics data usability, satisfaction, task-based interview 12 CS students timetracking I. preliminary with think-aloud, SUS at KULeuven via Twitter usefulness & MSDT assessing teacher online survey with 19 teachers & II. needs, use & Likert items, yes/no TA’s with Moodle logs usefulness and open questions CGIAR courses assessing teacher online survey with 12 participants needs, expert III. Likert items, yes/no from LAK Moodle logs feedback, use & and open questions MOOC usefulness structured face-to-face 11 teachers & VM with use, usefulness & IV. interview with tasks & TA’s at Carlos III extensive satisfaction open questions Uni. Madrid tool tracking 12Tuesday 20 March 12

ITERATION ONE • usability and user satisfaction evaluation • 12 CS students, using a - based time tracker •2 evaluation sessions: • task based interview with think aloud (after 1 week of tracking) • user satisfaction (SUS & MSDT) (after 1 month)Tuesday 20 March 12

LEARNABILITY, ERRORS & EFFICIENCY • in general, people understand the visualizations well! • some issues were uncovered...Tuesday 20 March 12

USER SATISFACTION • average SUS score: 73% [Bangor et al. 2008]Tuesday 20 March 12

demogra evaluation design negative positive phics goal changes •↑learnability usability, •↓errors 12 CS satisfaction, small usability I. 1st iteration •good satisfaction students preliminary issues •usefulness usefulness positiveTuesday 20 March 12

ITERATION TWO • 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s. • An online survey about usefulness, teacher issues and how the tool can resolve these. • on average: 40 mins are spent using SAM. 17Tuesday 20 March 12

CGIAR CASE STUDY 1 2 3 4 5 1 issue for 2 3 4 5 teacher addressed Provide feedback to the Provide students to the students feedback ✔ ? ✔ ✔ Being aware of what Being aware of are doing students what students are doing ✗ Knowing about collaboration Knowing about collaboration and communication and communication * ✔ Knowing which documents Knowing which and how muchare used and how much are used documents ✔ ✔ ? Knowing how and when Knowing howtools have been used tools have been used online and when online ✔ Finding the students who are not doing well Finding the students who are not doing well ✔ ? Finding the best students Finding the besthow much time Knowing students ? ? students spent Knowing how much time students Knowing if external learning spent ? ✔ ? resources are used Knowing if external learning resources Issues are used ✔ Addressed by SAM 18Tuesday 20 March 12

CGIAR CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ? Being aware of what students are doing ✔ ✔ Knowing about collaboration and communication ✔ ✗ Knowing which documents are used and how much ✔ ✔ Knowing how and when online tools have been used ? ✔ Finding the students who are not doing well ✔ ? Finding the best students ? ? Knowing how much time students spent ? ✔ Knowing if external learning resources are used ✔ ? 18Tuesday 20 March 12

CGIAR CASE STUDY 19Tuesday 20 March 12

CGIAR CASE STUDY want to search detect outliers for students more details on good indicator for effort student understand the workload progress evolution compare students obtain course overview increase awareness want better 1 to 1 comparison tool use for course more metrics design optimization wanted 19Tuesday 20 March 12

demogra evaluation design negative positive phics goal changes usability, •↑learnability 12 CS satisfaction, •↓errors I. 1st iteration small usability issues students preliminary •good satisfaction usefulness •usefulness positive •provides assessing 19 resource awareness teacher II. teachers help function recomm. not •all vis. useful needs, use & & TA’s useful •many uses usefulness •90% wants itTuesday 20 March 12

ITERATION THREE • opencourse on learning and knowledge analytics, http://bit.ly/ dWYVbX • 12 visual analytics enthousiasts + experts (who also teach) • almost identical survey to CGIAR case. 21Tuesday 20 March 12

LAK CASE STUDY issue for teacher addressed Provide feedback to the students ✔ ✔ Being aware of what students are doing ✔ ? Knowing about collaboration and communication ✔ ✗ Knowing which documents are used and how much ✔ ? Knowing how and when online tools have been used ✗ ? Finding the students who are not doing well ✔ ? Finding the best students ? ✗ Knowing how much time students spent ? ✔ Knowing if external learning resources are used ? ? 22Tuesday 20 March 12

LAK CASE STUDY 23Tuesday 20 March 12

LAK CASE STUDY 24 http://maranash.blogspot.com/2011/01/conference-shout-out.htmlTuesday 20 March 12

LAK CASE STUDY ﬁnd students experiencing problems and low verify the status of the engagement classroom activity more chronological course dwell time metrics self-reﬂection to measure progress and increase motivation more data 24 http://maranash.blogspot.com/2011/01/conference-shout-out.htmlTuesday 20 March 12

IDEA’S FROM EXPERTS the used resource types detailed information per student detailed information of 2 students detailed usage stats of resources stats or vis. on content creation 25Tuesday 20 March 12

IDEA’S FROM EXPERTS 2 the used resource types 5 detailed information per student 4 detailed information of 2 students 3 detailed usage stats of resources 1 stats or vis. on content creation 25Tuesday 20 March 12

demogra evaluation design negative positive phics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errors I. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive •provides awareness assessing teacher resource 19 teachers •all vis. useful II. needs, use & help function recomm. not & TA’s •many uses usefulness useful •90% want it •provides awareness assessing re-orderable most and feedback 12 teacher parallel addressed •many uses III. participan needs, expert coordinates needs are •66% want it ts feedback, use with indecisive •recomm. can be & usefulness histograms usefulTuesday 20 March 12

ITERATION FOUR •a CS course on C++ programming • 11 people: 7 teachers, 2 TA’s & 1 course planner • richer data set: tracking from programming environment • qualitativestudy using a structured face-2-face interview with 25 open questions. 27Tuesday 20 March 12

ACHIEVABLE TASKS conﬂicting visions on get a time identify students doing students doing well summary well and at risk and at risk better course monitor tool and overview resource use understand time spent better detect students course planning for assistance http://www.tdpri.com/forum/telecaster-discussion- forum/48431-ic503z-cap-52-ri-blackguard-book.html 28Tuesday 20 March 12

USER SATISFACTION • average SUS score: 69,69% [Bangor et al. 2008] 29Tuesday 20 March 12

USER SATISFACTION • average SUS score: 69,69% all: want to continue using it 9/11: give it to students [Bangor et al. 2008] 29Tuesday 20 March 12

demo- evaluation design negative positive graphics goal changes usability, •↑learnability 12 CS satisfaction, small usability •↓errors I. 1st iteration students preliminary issues •good satisfaction usefulness •usefulness positive •provides awareness 19 assessing teacher resource •all vis. useful II. teachers needs, use & help function recomm. not •many uses & TA’s usefulness useful •90% want it •provides awareness and assessing teacher most 12 re-orderable feedback needs, expert addressed III. participan PC with •many uses feedback, use & needs are ts histograms •66% want it usefulness indecisive •recomm. can be useful ﬁlter & •provides time overview search, conﬂicting •provides course 11 icons, visions of overview use, usefulness IV. teacher zooming in students •PC assist with detecting & satisfaction s & TA’s line chart, doing well problems editing PC or at risk •many uses & insights axes •100% want itTuesday 20 March 12

CONCLUSION • SAM enables: • to ﬁnd students doing well and at risk • a better course overview • understanding student time spending • almost all participants want to continue using SAM • SAM enables to ﬁnd a wide variety of new insights 31Tuesday 20 March 12

2 slides will appear on http://www.slideshare.net/stengovaerts slides will appear on http://www.slideshare.net/stengovaerts 32Tuesday 20 March 12

The Student Activity Meter for Awareness and Self-reflection

by Sten Govaerts

Accessibility

Categories

Upload Details

Usage Rights

Statistics

The Student Activity Meter for Awareness and Self-reflection Presentation Transcript