My presentation at the Learning Analytics day at VOV Lerend Netwerk in Brussels on May 22, 2014.

1. Learning Analytics
dashboards
Sten Govaerts
EPFL, Zwitserland
slides partially based on “Information Dashboard” by Katrien Verbert

2. about me

3. KULeuven team:
Human-Computer Interaction
!
•technology enhanced learning
•music
•research
EPFL team:
REACT - Real-time Coordination &
Distributed Interaction Systems Group
!
•technology enhanced learning
•social media
•transportation
http://hci.cs.kuleuven.be/
http://react.epfl.ch/

4. •2008: context-aware music information retrieval
•2009
•2010
•2011
•2012
•2013
•2014

5. •2008: context-aware music information retrieval
•2009
•2010
•2011
•2012
•2013
•2014
information visualisation}

6. •2008: context-aware music information retrieval
•2009
•2010
•2011
•2012
•2013
•2014
social recommendations
}
information visualisation}

7. •2008: context-aware music information retrieval
•2009
•2010
•2011
•2012
•2013
•2014
social recommendations
}
information visualisation}
learning analytics
}

8. •2008: context-aware music information retrieval
•2009
•2010
•2011
•2012
•2013
•2014
social recommendations
}
information visualisation}
learning analytics
}inquiry learning with online labs
}

9. PhDstudent!
@KULeuven
•2008: context-aware music information retrieval
•2009
•2010
•2011
•2012
•2013
•2014
social recommendations
}
information visualisation}
learning analytics
}inquiry learning with online labs
}

10. PhDstudent!
@KULeuven
Postdoc
@EPFL
•2008: context-aware music information retrieval
•2009
•2010
•2011
•2012
•2013
•2014
social recommendations
}
information visualisation}
learning analytics
}inquiry learning with online labs
}

11. about you…

12. why are you interested
in dashboards?

13. what I planned…

14. what I planned…
what are dashboards?

15. what I planned…
what are dashboards?
dashboard examples

16. what I planned…
what are dashboards?
what is wrong with most dashboards?
dashboard examples

17. what I planned…
what are dashboards?
what is wrong with most dashboards?
how can you design better dashboards?
dashboard examples

18. what I planned…
what are dashboards?
what is wrong with most dashboards?
how can you design better dashboards?
dashboard examples
learning analytics
dashboard examples

19. what I planned…
what are dashboards?
what is wrong with most dashboards?
how can you design better dashboards?
dashboard examples
how can you quickly get started
making your own dashboards?
learning analytics
dashboard examples

20. Typical dashboards…

21. Typical dashboards…

22. Typical dashboards…

23. Typical dashboards…

24. Typical dashboards…

25. Typical dashboards…

26. What is an
information dashboard?
A dashboard is a visual display of the most
important information needed to achieve one
or more objectives; consolidated and
arranged on a single screen so the
information can be monitored at a glance.
- Stephen Few
Stephen Few, "Dashboard Confusion," Intelligent Enterprise, March 20, 2004.

27. Learning Analytics!
dashboards
(& visualisations ;-)

28. http://research.uow.edu.au/learningnetworks/seeing/snapp/index.html
SNAPP

29. GISMO http://www.noe-kaleidoscope.org/pub/researcher/resources/video/aied/mazza.html

30. http://www.cs.uu.nl/docs/vakken/uem/Janssen.pdfThe ParticipationTool

31. Backstage
Pohl A, Bry F, Schwarz J, Gottstein M (2012) Sensing the classroom: improving awareness and self-awareness of students with backstage. In:
Proceedings of the international conference on interactive and collaborative learning (ICL), Villach, Austria, 26–28 September 2012.

32. Dollár A, Steif PS (2012) Web-based statics course with learning dashboard for instructors. In: Uskov V (ed)
Proceedings of computers and advanced technology in education (CATE 2012), June 25–27, 2012, Napoli, Italy
OLI Dashboard

33. LARA
Learning Analytics Reflection
& Awareness environment

36. also on tabletop & large displays

37. J. L. Santos, K.Verbert, S.
Govaerts, and E. Duval.
Visualizing PLE Usage.
Proceedings of
EFEPLE11: 1st
Workshop on Exploring
the Fitness and
Evolvability of Personal
Learning Environments,
CEUR Laclusaz, France,
30-31 March 2011

38. Student Activity
Meter
Govaerts, Sten;Verbert, Katrien, Duval,
Erik; Pardo,Abelardo, The Student Activity
Meter for Awareness and Self-reflection.
http://ariadne.cs.kuleuven.be/monitorwidget-chi

39. Problem
http://www.ypfp.org/classroom

40. Problem
http://www.ypfp.org/classroom

41. Objectives
• self-monitoring for learners
http://free-extras.com/images/obama_hope_progress-2848.htm

42. Objectives
• self-monitoring for learners
• awareness for teachers
http://free-extras.com/images/obama_hope_progress-2848.htm

43. Objectives
• self-monitoring for learners
• awareness for teachers
• time tracking
http://free-extras.com/images/obama_hope_progress-2848.htm

44. Objectives
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning resource use and
recommendations http://free-extras.com/images/obama_hope_progress-2848.htm

45. STUDENT ACTIVITY METER

46. STUDENT ACTIVITY METER

47. STUDENT ACTIVITY METER

48. STUDENT ACTIVITY METER

49. Methodology.
• design-based research [Obrenović 2011]: design a tool, give
students & teachers and evaluate
• iterative process of design & evaluation
• paper mockups
• usability studies
• surveys on
usefulness
http://funnytogo.com/pictures/evolution/timeline.htm

50. iter. evaluation goal methodology demo-graphics data
I.
usability, satisfaction,
preliminary
usefulness
task-based interview
with think-aloud, SUS
& MSDT
12 CS students
at KULeuven
timetracking
viaTwitter
II.
assessing teacher
needs, use &
usefulness
online survey with
Likert items, yes/no
and open questions
19 teachers &
TA’s with
CGIAR courses
Moodle logs
III.
assessing teacher
needs, expert
feedback, use &
usefulness
online survey with
Likert items, yes/no
and open questions
12 participants
from LAK
MOOC
Moodle logs
IV.
use, usefulness &
satisfaction
structured face-to-face
interview with tasks &
open questions
11 teachers &
TA’s at Carlos III
Uni. Madrid
VM with
extensive
tool tracking

51. Conclusion
• SAM is perceived useful and contributes to better
awareness.

52. Conclusion
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:

53. Conclusion
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk

54. Conclusion
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview

55. Conclusion
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending

56. Conclusion
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
• to find a wide variety of new insights

57. Conclusion
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
• to find a wide variety of new insights
• almost all want to continue using SAM

58. Go-Lab Learning
Analytics dashboard

67. STRUCTURE
• explain main idea
• science edu -> to labs -> lab fire -> need something safer
• objectives golab
• partner list
• react’s job
• portal
• architecture

68. Orientation
Discussion
Investigation
Conceptua-
lisation
Conclusion

69. An overview of where students are
Currently there are 4 students in the orientation phase,
8 in the conceptualization phase, 3 in the investigation
phase and one discussing.
Student currently in phase
4 8 3 0 1

70. A detailed view of where each student is
Tommy is in the discussion phase, Becky in the
investigation phase, etc.
Student current phase
Tommy
Becky
Jessi
Alex

71. An overview of the time spent in each space
for all students
There are currently students in the conclusion and the
discussion phases, the average time spent in the
discussion phase so far is 20 seconds and the average
time spent in the investigation phase is 15:30 minutes
Average time per phase
2:50 3:01 15:30 4:20 0:20

72. The number of time students change phases
Alex has been changing 12 times between phases so
far, Jessi only 5 times.
Number of phase changes
Tommy Becky Jessi Alex
4 8 5 12

73. An overview of who handed in their
assignment
Tommy, Jessi and Alex have already handed in their
assignments.
Student reports
Tommy
Becky
Jessi
Alex
report.pdf
Jessi.pdf
Pat
pat.pdf

74. Student reports
Tommy
Becky
Jessi
Alex
report.pdf
Jessi.pdf
Pat
pat.pdf
Number of phase changes
Tommy Becky Jessi Alex
4 8 5 12
Connected Students
11
Connected Students
4
Connected Stud
6
Student current phase
Tommy
Becky
Jessi
Alex

75. common issues with
dashboards

76. multiple screens.

77. multiple screens.

78. multiple screens.

79. inadequate context
for the data.

80. inadequate context
for the data.

81. too much detail/precision.

82. choosing a deficient
measure.

83. choosing a deficient
measure.

84. inappropriate display
media.

85. inappropriate display
media.

86. inappropriate display
media.

87. meaningless variety.

88. poor visualisations.

89. poor visualisations.

90. poor visualisations.

91. poor visualisations.

93. bad encoding of data

94. not highlighting
important data.

95. useless decoration.

96. too much colour.

97. dashboards require thoughtful design…

98. how to design better
dashboards?

99. how to design better
dashboards?
1. reduce the non-data pixels

100. how to design better
dashboards?
1. reduce the non-data pixels
2. enhance the data pixels

101. how to design better
dashboards?
1. reduce the non-data pixels
2. enhance the data pixels
3. design for usability

102. how to design better
dashboards?
1. reduce the non-data pixels
2. enhance the data pixels
3. design for usability

103. reduce non-data pixels

104. reduce non-data pixels

105. reduce non-data pixels

106. reduce non-data pixels

107. reduce non-data pixels

108. Enhance data pixels

109. Enhance data pixels

110. Design for usability.

111. Design for usability.
1. organize information to
support its meaning & use

112. Design for usability.
1. organize information to
support its meaning & use
2. make it aesthetically
pleasing!

113. Design for usability.
1. organize information to
support its meaning & use
2. make it aesthetically
pleasing!
3. design for a launch pad

114. Design for usability.
1. organize information to
support its meaning & use
2. make it aesthetically
pleasing!
3. design for a launch pad

115. Design for usability.
1. organize information to
support its meaning & use
2. make it aesthetically
pleasing!
3. design for a launch pad

116. How to evaluate?
usability
System Usability Scale (SUS)

117. How to evaluate?
usability
System Usability Scale (SUS)

118. How to evaluate?
usability
interviews:
task list

119. How to evaluate?
usability
interviews:
task list think aloud+

120. How to evaluate?
usability
interviews:
task list think aloud+ observe+

121. How to evaluate?
usability
interviews:
task list think aloud+ observe+
understand what the user does
understand what the issues are
+

122. How to evaluate?
usefulness&impact

123. How to evaluate?
usefulness&impact
much HARDER…

124. How to evaluate?
usefulness&impact
interviews
much HARDER…

125. How to evaluate?
usefulness&impact
interviews
much HARDER…
perceived usefulness surveys

126. How to evaluate?
usefulness&impact
interviews
much HARDER…
perceived usefulness surveys
longitudinal studies

127. How to quickly build
your own dashboard?

128. How to quickly build
your own dashboard?
get data

129. How to quickly build
your own dashboard?
get data
(clean data)

130. How to quickly build
your own dashboard?
get data
(clean data)
visualise!

131. get data
import.io

132. get data
import.iokimonolabs.com

133. Organic Chemistry
Organic chemistry
Organical Chemestry
OrganicChemistry
Organ Chemistry
(clean data)

134. Organic Chemistry
Organic chemistry
Organical Chemestry
OrganicChemistry
Organ Chemistry
(clean data)
Google Refine/OpenRefine
http://openrefine.org/

135. Organic Chemistry
Organic chemistry
Organical Chemestry
OrganicChemistry
Organ Chemistry
(clean data)
Google Refine/OpenRefine
http://openrefine.org/
DataWrangler
http://vis.stanford.edu/wrangler/

136. visualise!
ManyEyeshttp://www.manyeyes.com/software/analytics/manyeyes/

137. visualise!
ManyEyeshttp://www.manyeyes.com/software/analytics/manyeyes/
RAW
http://app.raw.densitydesign.org/#%2F

138. visualise!
ManyEyeshttp://www.manyeyes.com/software/analytics/manyeyes/
RAW
http://app.raw.densitydesign.org/#%2F
video: http://www.ted.com/talks/
hans_rosling_shows_the_best_stats_you_ve_ever_seen
http://www.gapminder.org/

139. visualise!
ManyEyeshttp://www.manyeyes.com/software/analytics/manyeyes/
RAW
http://app.raw.densitydesign.org/#%2F
video: http://www.ted.com/talks/
hans_rosling_shows_the_best_stats_you_ve_ever_seen
http://www.gapminder.org/

140. visualise!
http://www.simile-widgets.org/exhibit/

141. visualise!
http://www.simile-widgets.org/exhibit/
http://www.simile-
widgets.org/timeline/

142. visualise!
http://www.simile-widgets.org/exhibit/
http://www.simile-
widgets.org/timeline/
Google Charts
https://developers.google.com/chart/

143. there are also all-in-
one solutions…

144. How many kids are
repeating a year in
elementary school?

145. • http://www.wolframalpha.com/input/?i=how+many+kids
+are+repeating+primary+school+in+Belgium
• http://www.wolframalpha.com/input/?i=university+with+the
+largest+enrollment&lk=3
• http://www.wolframalpha.com/input/?i=university+with
+the+largest+enrollment+in+Belgium
• http://www.wolframalpha.com/input/?i=Belgium+vs+USA
+education+expenditures+per+high+school+student

146. further reading…
http://pdf.th7.cn/down/files/1312/information_dashboard_design.pdf

147. Edward Tufte.

148. Questions?
slides on: http://www.slideshare.net/stengovaerts