SlideShare a Scribd company logo
https://mfeldstein.com/purdue-course-signals-data-issue-
Scalable, Actionable, and
Ethical Learning Dashboards
a reality check
Tinne De Laet
Tinne.DeLaet@kuleuven.be
@TinneDeLaet
Who am I? Why I am here?
woman
phdrobotics
Head Tutorial Services
Engineering Science KU
Leuven
.
associate
professor
mech.engine
er
founded 1425
research-intensive university
16 faculties → general
university
> 55 000 students
2016-2017-2018 Europe’s
most innovative university
(Reuters ranking)
“Learning analytics is
about collecting traces
that learners leave
behind and using
those traces to
improve learning.”
- Erik Duval
Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012, https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/ 5
Learning Analytics?
Learning Dashboards?
6Dashboard Confusion, Stephen Few, Intelligent Enterprise, March 20, 2004
“A dashboard is a visual display
of the most important
information needed to achieve
one or more objectives;
consolidated and arranged on a
single screen so the information
can be monitored at a glance.”
- Stephen Few
Successful Transition from secondary to higher
Education using Learning Analytics
enhance a successful transition from
secondary to higher education by means of
learning analytics
 design and build analytics dashboards,
 dashboards that go beyond identifying at-risk
students, allowing actionable feedback for all
students on a large scale.
Achieving Benefits from Learning Analytics
research strategies and practices for using
learning analytics to support students during
their first year at university
 developing the technological aspects of
learning analytics,
 focuses on how learning analytics can be used
to support students.
7
www.stela-project.eu
@STELA_project
2015-1-UK01-KA203-013767
www.ableproject.eu
@ABLE_project_eu
562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD
STELA ♥ ABLE
8
actionable feedback
student-centered
program level
inclusive
first-year experience
institution-wide
Learning Analytics
actual implementation
[!] Feedback must be “actionable”.
9
Warning!
Male students have
10% less probability to
be successful.
You are male.
Warning!
Your online activity is
lagging behind.
action?
?
action?
?

10
awareness
(self-)reflection
sensemaking
impact
data
questions
answers
behavior change
new meaning
Verbert K, Duval E, Klerkx J; Govaerts S, Santos JL (2013) Learning analytics dashboard applications. American Behavioural Scientist, 10 pages. Published online February 2013.
[!] Feedback must be “actionable”.
learning dashboards @KU Leuven
interaction
self-reflection
LISSA
STUDENT
ADVISOR
STUDENT LASSI –
learning skills
REX - scoresPOS – future
students
[!] Start with the available data.
Lots of data may eventually become
available in the future …
…. already start with what is available
12
(*)
(*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication.
British Journal of Educational Technology, 44(4), 616-628.
Case study
dashboard interaction student – study advisor
Study advisor – student conversations
14
Should I consider
another program?
Can I still finish the
bachelor in 3 years?
How should I compose
my program for next
year?
What is the personal
situation?
How can I help?
What is the best
next step?
[!] Use all available expertise.
15
visualization experts
practitioners / end-users
researchers LA
researchers first-year
study success
Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue.
In IEEE Transactions on Learning Technology (http://ieeexplore.ieee.org/document/7959628/).
LISSA dashboard
https://able.cs.kuleuven.be/demo-september/2016/2
[!] Wording matters.
17
73% chance of success
73% of students of earlier
cohorts with the same
study efficiency obtained
the bachelor degree
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
LISSA: status
18
26 programs >4500 students
114 student advisors
training of study advisors
• Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology
• http://blog.associatie.kuleuven.be/tinnedelaet/lissa-learning-dashboard-supporting-student-advisers-in-traditional-higher-education/
• Millecamp M., Gutiérrez F., Charleer S., Verbert K., De Laet T.# (2018). A qualitative evaluation of a learning dashboard to support advisor-student dialogues. Proceedings of the 8th
International Learning Analytics & Knowledge Conference. LAK. Sydney, 5-9 March 2018 (pp. 1-5) ACM.
dashboards for three examination
periods
observations, interviews,
questionnaires
LISSA: evaluation – observations
19
15 observations
insights
(-) factual
(+) interpretative
(!) reflective
Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology
Evaluation – interviews
“When students see the numbers, they are
surprised, but now they believe me.
Before, I used my gut feeling, now I feel
more certain of what I say as well”.
“It’s like a main thread
guiding the
conversation.”
“I can talk about what to do with the results,
instead of each time looking for the data and
puzzling it together.”
“Students don’t know where to look during the
conversation, and avoid eye contact.
The dashboard provides them a point of focus”.
“A student changed her
study method in June and
could now see it paid off.”
LISSA supports a personal dialogue.
 the level of usage depends on the experience
and style of the study advisors
 fact-based evidence at the side
 narrative thread
 key moments and student path help to
reconstruct personal track
“I can focus on the
student’s personal
path, rather than on
the facts.”
“Now, I can blame
the dashboard and
focus on
collaboratively looking
for the next step to
take.”
20
LISSA: evaluation – student
questionnaires
21
26 programs @KU Leuven
291 student questionnaires
first examination period
“Confronting, but
useful”
“I want to use this
dashboard at home.”
“Also show the sub-grades
for labs, … ”
“How can I know the data is
trustworth?”
“Can’t these visualizations be
send to students?” “Crisp and clear.”
22
0
0
1
1
1
1
4
2
1
4
4
3
29
21
36
37
49
42
176
112
156
132
141
169
80
155
93
116
92
72
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1. The dashboard is clarifying and surveyable.
2. The shown information regarding my study
situation is correct.
3. The shown position with respect to my fellow
students (histograms per exam and global…
4. A conversation with my student advisors helped
me to gain insight in my study trajectory.
5. The visualisation is of added value to the
conversation with the student advisor.
6. The shown information provide me insight in
my current situation.
Student questionnaire January 2018 (N=291)
Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree
[!] Do not oversimplify. Show
uncertainty.
23
• reality is complex
• measurement is limited
• individual circumstances
• need for nuance
• trigger reflection
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
[!] Be careful with predictive
algorithms.
24
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
• reality is complex
• measurement is limited
• individual circumstances
• need for nuance
• trigger reflection
Case study
student-facing dashboards
[!] Start with the available data.
26
data already available?
administrative (examples)
student records course grades
systems (examples)
LMS access logs advisor meetings
)
Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Small data as a conversation starter for learning analytics: exam results dashboard for first-year students in higher
education. Journal of Research in Innovative Teaching & Learning, , 1-14.
demo: https://learninganalytics.set.kuleuven.be/static-demo-rex/ (en) or https://learninganalytics.set.kuleuven.be/demo/rex-1718jan-ir (nl)
[!] Think beyond the obvious data.
27
• Don’t think too traditional.
• Many institutions are collecting survey
data for educational research.
[!] Not all data is usable.
28
example data from a traditional course with “VLE as a file system”
test scores
activity/week (#days)
weeks of the year
[!] Not all data is usable.
29
example data from a course with flipped classroom & blended learning
exam scores
activity (# of modules used)
Not a single student
using less than 10
modules passed the
course.
Most of the successful
students used 15
modules or more.
[!] Keep Learning Analytics in
mind when designing learning
activities.
30
Learning
Analytics
Learning Design
INFORM
ENABLE
If LA indeed contributes to improved
learning design…
… don’t make it an afterthought
31
Does my concentration
matter?
How is my time
management?
I feel uncertain.
Is this normal?
How can I improve
my concentration?
data already available?
administrative (examples)
student records course grades
[!] Think beyond the obvious data.
32
systems (examples)
LMS access logs advisor meetings
surveys (examples)
quality insurance LASSI
~ 30 LASSI questions
(shortened version)
“Learning Skills”
Example: When preparing for an
exam, I create questions that I
think might be included.
Example: I find it difficult to
maintain my concentration
while doing my coursework.
Example: I find it hard to stick
to a study schedule.
raw scores
(selected 5 out of 10)
CONCENTRATION
MOTIVATION
FAILURE ANXIETY
TEST STRATEGY
TIME MANAGEMENT
norm scores
(in Flemish HE context)
Example: STRONG
Example: AVERAGE
Example: LOW
Example: VERY STRONG
Example: VERY WEAK
33
 Meta cognitive abilities
Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an
open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017).
readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo
Dashboard learning skills
34
students complete LASSI
questionnaire
students received personalized email
with invitation for dashboard
4367 students in 26 programs
in 9 faculties @KU Leuven
demo:
https://learninganalytics.set.kuleuven.be/static-demo-lassi/ (en) of
https://learninganalytics.set.kuleuven.be/demo/lassi-1718 (nl)
2 programs @TU Delft
Feedback model
1. What is this about?
2. How am I doing?
3. How does this relates to
others?
4. Why is this relevant?
5. What can I do about it?
35
36
3. How does this relates to
others?
2. How am I doing?
1. What is this about?
@studyProgram@
@yourScore@
4. Why is this relevant?
5. What can I do about it?
37
38
5. What can I do about it?
Response
39
3868 students (89%)
used dashboard
Student feedback?
40
http://blog.associatie.kuleuven.be/tinnedelaet/learning-dashboard-for-actionable-feedback-on-learning-and-studying-skills/
How CLEAR is this info?
stars stars
Students that click through
Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.
In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.
41
 better learning skills
More intense users
Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.
In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.
42
 worse learning skills
What can we learn from dashboard usage?
43Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and
Microinteractions. accepted for ECTEL 2018
p<1e-5 p<1e-9p-test
[!] Give students “the key”.
44
• Student has the key to own
data.
• Student takes initiative to
share/discuss own data.
• GDPR as opportunity!
[!] Acceptance precedes impact.
45
• Involve stakeholders from the start and
value their input!
COmmunication
COoperation
• Demonstrate usefulness.
• Take care of ethics and privacy.
• Best scenario:
students & study advisors as ambassadors
COCO
Impact?
survey before intervention
 2nd year students 2016-2017
 experiences first-year feedback
 41 vragen, 5-point Likert scale
 pen & paper
dashboards
 LISSA
 LASSI (learning skills)
 3 x REX (grades)
Survey after intervention
 2nd year students 2017-2018
Impact?
During the first year I received sufficient information regarding my academic achievements.
47
Engineering Science (p<0.001)
Impact?
The information I received helped to position myself with respect to my peers.
48
Engineering Science (p<0.001)
Impact?
49
The information I received made me reflect.
The information I received made me adapt my behaviour.
[!] Context matters!
• available data
• national and institutional regulations
and culture
• educational vision
• educational system, size of population ..
• …
Don’t just copy existing LA solutions!
50
Summary
case studies 11 findings/recommendations
[!] Use all available expertise.
[!] Start with the available data.
[!] Look beyond the obvious data.
[!] Not all data is usable.
[!] Wording matters.
[!] Don’t oversimplify. Show uncertainty.
[!] Beware of predictive algorithms.
[!] Keep Learning Analytics in mind when designing
learning activities.
[!] Give students “the key” to their data.
[!] Acceptance precedes impact.
[!] Context matters!
 humble approach
 small data
 involvement of stakeholders, especially practitioners
 actionable feedback
 scalability
 traditional university settings
Is this Learning Analytics?
Future?
52
Continue and extend dashboards
@KU Leuven?
Transfer to other universities?
extension?
Project team @
53
Sven Charleer
AugmentHCI, Computer Science department
PhD researcher ABLE
Katrien Verbert
AugmentHCI, Computer Science department
Copromotor of STELA & ABLE
Carolien Van Soom
Leuven Engineering and Science Education Center
Head of Tutorial Services of Science
Copromotor of STELA & ABLE
Greet Langie
Leuven Engineering and Science Education Center
Vicedean (education) faculty of Engineering Technology
Copromotor of STELA & ABLE
Tinne De Laet
Leuven Engineering and Science Education Center
Head of Tutorial Services of Engineering Science
Coordinator of STELA
KU Leuven coordinator of ABLE
Francisco Gutiérrez
AugmentHCI, Computer Science department
PhD researcher ABLE
Tom Broos
Leuven Engineering and Science Education Center
AugmentHCI, Computer Science department
PhD researcher STELA
Martijn Millecamp
AugmentHCI, Computer Science department
PhD researcher ABLE
Special thanks to study advisors for their cooperation, advice, feedback, and support!
Jasper, Bart, Riet, Hilde, An, Katrien, …
♥
Response - REX
54
571
76 245 220 353 258 274 1060 187 283 843
242
163
38 130 120 200 151 162 647 128 230 771
269
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
clicked non clicked
Evaluation - REX
55
22 26
105
443
511
0
100
200
300
400
500
600
1 2 3 4 5
useful (stars)
26 29
134
441 429
0
100
200
300
400
500
600
1 2 3 4 5
clear (stars)
41
60
220
385
456
0
100
200
300
400
500
600
1 2 3 4 5
added value (stars)
Micro-interactions - REX
56
defaultalternative
)
Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and
Microinteractions. accepted for ECTEL 2018
Micro-interactions
57Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and
Microinteractions. accepted for ECTEL 2018
58
2
3
1
7
2
1
10
44
23
36
3
11
64
97
81
74
36
29
150
115
126
110
119
91
61
30
56
59
128
157
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
7. The dashboard makes me more aware on my current
study situation.
8. The dashboard makes me forecast the different
possibilities in my future study trajectory.
9. The dashboard helps me to reflect on my past and
current study behaviour or study trajectory.
10. The dashboards stimulates me to adapt my approach
in my studies for the future (study behaviour or study…
11. If I will have a new conversation after one of the next
examination periods, I hope that the visualisation will be…
12. I would like to consult the information on my own.
Student questionnaire January 2018 (N=291)
Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree
How to determine
thresholds for different
groups?
LISSA dashboard
59
 upper and lower group: clear message
 middle group as small as possible
 Do not overfit! (nuance)
Dashboard positioning test
60https://feedback.ijkingstoets.be/ijkingstoets-10-ir/index.html (10ir0demo)

More Related Content

What's hot

Learning analytics: Threats and opportunities
Learning analytics: Threats and opportunitiesLearning analytics: Threats and opportunities
Learning analytics: Threats and opportunities
Martin Hawksey
 
Learning analytics: the way forward
Learning analytics: the way forwardLearning analytics: the way forward
Learning analytics: the way forward
Rebecca Ferguson
 
Learning Analytics for online and on-campus education: experience and research
Learning Analytics for online and on-campus education: experience and researchLearning Analytics for online and on-campus education: experience and research
Learning Analytics for online and on-campus education: experience and research
Tinne De Laet
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of Derby
Ed Foster
 
LACE Masterclass Learning Analytics M&L Brussels 2014
LACE Masterclass Learning Analytics M&L Brussels 2014LACE Masterclass Learning Analytics M&L Brussels 2014
LACE Masterclass Learning Analytics M&L Brussels 2014
Kennisnet
 
Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?
Denise Whitelock
 
LAK15 panel - European Perspectives
 LAK15 panel - European Perspectives LAK15 panel - European Perspectives
LAK15 panel - European Perspectives
LACE Project
 
Higher Education & Game Principles: Context, Theory & Application - Daniel La...
Higher Education & Game Principles: Context, Theory & Application - Daniel La...Higher Education & Game Principles: Context, Theory & Application - Daniel La...
Higher Education & Game Principles: Context, Theory & Application - Daniel La...
Blackboard APAC
 
Requirements for Learning Analytics
Requirements for Learning AnalyticsRequirements for Learning Analytics
Requirements for Learning Analytics
Tore Hoel
 
Leading with Data: Impacting Change on Scale featuring Belinda Tynan
Leading with Data: Impacting Change on Scale featuring Belinda TynanLeading with Data: Impacting Change on Scale featuring Belinda Tynan
Leading with Data: Impacting Change on Scale featuring Belinda Tynan
Center for Online Innovation in Learning
 
Learning Design
Learning DesignLearning Design
Learning Design
Scot Aldred
 
Is uw instelling klaar voor learning analytics?
Is uw instelling klaar voor learning analytics?Is uw instelling klaar voor learning analytics?
Is uw instelling klaar voor learning analytics?
SURF Events
 
Learning analytics, learning design and MOOCs
Learning analytics, learning design and MOOCsLearning analytics, learning design and MOOCs
Learning analytics, learning design and MOOCs
Rebecca Ferguson
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?
Denise Whitelock
 
Chec presentation daniela and viv
Chec presentation daniela and vivChec presentation daniela and viv
Chec presentation daniela and viv
Vivienne Bozalek
 
The power of learning analytics to unpack learning and teaching: a critical p...
The power of learning analytics to unpack learning and teaching: a critical p...The power of learning analytics to unpack learning and teaching: a critical p...
The power of learning analytics to unpack learning and teaching: a critical p...
Bart Rienties
 
Using Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning NeedsUsing Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning Needs
DreamBox Learning
 
Using learning analytics to support formative assessment oln 20171111
Using learning analytics to support formative assessment oln 20171111Using learning analytics to support formative assessment oln 20171111
Using learning analytics to support formative assessment oln 20171111
Yi-Shan Tsai
 
2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...
2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...
2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...
eMadrid network
 
Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107
Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107
Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107
Yi-Shan Tsai
 

What's hot (20)

Learning analytics: Threats and opportunities
Learning analytics: Threats and opportunitiesLearning analytics: Threats and opportunities
Learning analytics: Threats and opportunities
 
Learning analytics: the way forward
Learning analytics: the way forwardLearning analytics: the way forward
Learning analytics: the way forward
 
Learning Analytics for online and on-campus education: experience and research
Learning Analytics for online and on-campus education: experience and researchLearning Analytics for online and on-campus education: experience and research
Learning Analytics for online and on-campus education: experience and research
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of Derby
 
LACE Masterclass Learning Analytics M&L Brussels 2014
LACE Masterclass Learning Analytics M&L Brussels 2014LACE Masterclass Learning Analytics M&L Brussels 2014
LACE Masterclass Learning Analytics M&L Brussels 2014
 
Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?
 
LAK15 panel - European Perspectives
 LAK15 panel - European Perspectives LAK15 panel - European Perspectives
LAK15 panel - European Perspectives
 
Higher Education & Game Principles: Context, Theory & Application - Daniel La...
Higher Education & Game Principles: Context, Theory & Application - Daniel La...Higher Education & Game Principles: Context, Theory & Application - Daniel La...
Higher Education & Game Principles: Context, Theory & Application - Daniel La...
 
Requirements for Learning Analytics
Requirements for Learning AnalyticsRequirements for Learning Analytics
Requirements for Learning Analytics
 
Leading with Data: Impacting Change on Scale featuring Belinda Tynan
Leading with Data: Impacting Change on Scale featuring Belinda TynanLeading with Data: Impacting Change on Scale featuring Belinda Tynan
Leading with Data: Impacting Change on Scale featuring Belinda Tynan
 
Learning Design
Learning DesignLearning Design
Learning Design
 
Is uw instelling klaar voor learning analytics?
Is uw instelling klaar voor learning analytics?Is uw instelling klaar voor learning analytics?
Is uw instelling klaar voor learning analytics?
 
Learning analytics, learning design and MOOCs
Learning analytics, learning design and MOOCsLearning analytics, learning design and MOOCs
Learning analytics, learning design and MOOCs
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?
 
Chec presentation daniela and viv
Chec presentation daniela and vivChec presentation daniela and viv
Chec presentation daniela and viv
 
The power of learning analytics to unpack learning and teaching: a critical p...
The power of learning analytics to unpack learning and teaching: a critical p...The power of learning analytics to unpack learning and teaching: a critical p...
The power of learning analytics to unpack learning and teaching: a critical p...
 
Using Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning NeedsUsing Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning Needs
 
Using learning analytics to support formative assessment oln 20171111
Using learning analytics to support formative assessment oln 20171111Using learning analytics to support formative assessment oln 20171111
Using learning analytics to support formative assessment oln 20171111
 
2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...
2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...
2020_12_11 «Open Educational Resources for Improving Teaching and Learning Ex...
 
Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107
Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107
Supporting Higher Education to Integrate Learning Analytics_EUNIS20171107
 

Similar to Scalable, Actionable, and Ethical Learning Dashboards: a reality check

Learning dashboards for actionable feedback: the (non)sense of chances of suc...
Learning dashboards for actionable feedback: the (non)sense of chances of suc...Learning dashboards for actionable feedback: the (non)sense of chances of suc...
Learning dashboards for actionable feedback: the (non)sense of chances of suc...
Tinne De Laet
 
Co-designing learning dashboards for scalable feedback
Co-designing learning dashboards for scalable feedbackCo-designing learning dashboards for scalable feedback
Co-designing learning dashboards for scalable feedback
Tinne De Laet
 
Data-based feedback through learning dashboards: does it support the first-ye...
Data-based feedback through learning dashboards: does it support the first-ye...Data-based feedback through learning dashboards: does it support the first-ye...
Data-based feedback through learning dashboards: does it support the first-ye...
Tinne De Laet
 
Using learning analytics to improve student transition into and support throu...
Using learning analytics to improve student transition into and support throu...Using learning analytics to improve student transition into and support throu...
Using learning analytics to improve student transition into and support throu...
Tinne De Laet
 
Can learning analytics improve engineering education in both MOOC and traditi...
Can learning analytics improve engineering education in both MOOC and traditi...Can learning analytics improve engineering education in both MOOC and traditi...
Can learning analytics improve engineering education in both MOOC and traditi...
Tinne De Laet
 
Designing Learning Analytics for Humans with Humans
Designing Learning Analytics for Humans with HumansDesigning Learning Analytics for Humans with Humans
Designing Learning Analytics for Humans with Humans
alywise
 
Preliminary PhD Defence - Student-facing Dashboards
Preliminary PhD Defence - Student-facing DashboardsPreliminary PhD Defence - Student-facing Dashboards
Preliminary PhD Defence - Student-facing Dashboards
Sven Charleer
 
ABLE project presentation EFYE 2017
ABLE project presentation EFYE 2017ABLE project presentation EFYE 2017
ABLE project presentation EFYE 2017
Tinne De Laet
 
2021 nyu-learn - ai for academic advising, a matter of trust
2021 nyu-learn - ai for academic advising, a matter of trust2021 nyu-learn - ai for academic advising, a matter of trust
2021 nyu-learn - ai for academic advising, a matter of trust
Tinne De Laet
 
Learning analytics research informed institutional practice
Learning analytics research informed institutional practiceLearning analytics research informed institutional practice
Learning analytics research informed institutional practice
Yi-Shan Tsai
 
Learning dashboard for supporting students: from first-year engineering to MO...
Learning dashboard for supporting students: from first-year engineering to MO...Learning dashboard for supporting students: from first-year engineering to MO...
Learning dashboard for supporting students: from first-year engineering to MO...
Tinne De Laet
 
Macfadyen usc tlt keynote 2015.pptx
Macfadyen usc tlt keynote 2015.pptxMacfadyen usc tlt keynote 2015.pptx
Macfadyen usc tlt keynote 2015.pptx
Leah Macfadyen
 
ISSOTL Presentation
ISSOTL PresentationISSOTL Presentation
ISSOTL Presentation
David Heath
 
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...
EUmoocs
 
Learning design and learning analytics
Learning design and learning analyticsLearning design and learning analytics
Learning design and learning analyticsRebecca Ferguson
 
WCOL2019: Learning analytics for learning design or learning design for learn...
WCOL2019: Learning analytics for learning design or learning design for learn...WCOL2019: Learning analytics for learning design or learning design for learn...
WCOL2019: Learning analytics for learning design or learning design for learn...
Marko Teräs
 
ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015
Ed Foster
 
VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...
VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...
VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...
eMadrid network
 
Designing Learning Analytics Dashboards: Lessons Learned
Designing Learning Analytics Dashboards: Lessons LearnedDesigning Learning Analytics Dashboards: Lessons Learned
Designing Learning Analytics Dashboards: Lessons Learned
Katrien Verbert
 
HEI Fellowship: Blog Post 1
HEI Fellowship: Blog Post 1HEI Fellowship: Blog Post 1
HEI Fellowship: Blog Post 1
Rebecca Ferriday
 

Similar to Scalable, Actionable, and Ethical Learning Dashboards: a reality check (20)

Learning dashboards for actionable feedback: the (non)sense of chances of suc...
Learning dashboards for actionable feedback: the (non)sense of chances of suc...Learning dashboards for actionable feedback: the (non)sense of chances of suc...
Learning dashboards for actionable feedback: the (non)sense of chances of suc...
 
Co-designing learning dashboards for scalable feedback
Co-designing learning dashboards for scalable feedbackCo-designing learning dashboards for scalable feedback
Co-designing learning dashboards for scalable feedback
 
Data-based feedback through learning dashboards: does it support the first-ye...
Data-based feedback through learning dashboards: does it support the first-ye...Data-based feedback through learning dashboards: does it support the first-ye...
Data-based feedback through learning dashboards: does it support the first-ye...
 
Using learning analytics to improve student transition into and support throu...
Using learning analytics to improve student transition into and support throu...Using learning analytics to improve student transition into and support throu...
Using learning analytics to improve student transition into and support throu...
 
Can learning analytics improve engineering education in both MOOC and traditi...
Can learning analytics improve engineering education in both MOOC and traditi...Can learning analytics improve engineering education in both MOOC and traditi...
Can learning analytics improve engineering education in both MOOC and traditi...
 
Designing Learning Analytics for Humans with Humans
Designing Learning Analytics for Humans with HumansDesigning Learning Analytics for Humans with Humans
Designing Learning Analytics for Humans with Humans
 
Preliminary PhD Defence - Student-facing Dashboards
Preliminary PhD Defence - Student-facing DashboardsPreliminary PhD Defence - Student-facing Dashboards
Preliminary PhD Defence - Student-facing Dashboards
 
ABLE project presentation EFYE 2017
ABLE project presentation EFYE 2017ABLE project presentation EFYE 2017
ABLE project presentation EFYE 2017
 
2021 nyu-learn - ai for academic advising, a matter of trust
2021 nyu-learn - ai for academic advising, a matter of trust2021 nyu-learn - ai for academic advising, a matter of trust
2021 nyu-learn - ai for academic advising, a matter of trust
 
Learning analytics research informed institutional practice
Learning analytics research informed institutional practiceLearning analytics research informed institutional practice
Learning analytics research informed institutional practice
 
Learning dashboard for supporting students: from first-year engineering to MO...
Learning dashboard for supporting students: from first-year engineering to MO...Learning dashboard for supporting students: from first-year engineering to MO...
Learning dashboard for supporting students: from first-year engineering to MO...
 
Macfadyen usc tlt keynote 2015.pptx
Macfadyen usc tlt keynote 2015.pptxMacfadyen usc tlt keynote 2015.pptx
Macfadyen usc tlt keynote 2015.pptx
 
ISSOTL Presentation
ISSOTL PresentationISSOTL Presentation
ISSOTL Presentation
 
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...
EMMA Summer School - Rebecca Ferguson - Learning design and learning analytic...
 
Learning design and learning analytics
Learning design and learning analyticsLearning design and learning analytics
Learning design and learning analytics
 
WCOL2019: Learning analytics for learning design or learning design for learn...
WCOL2019: Learning analytics for learning design or learning design for learn...WCOL2019: Learning analytics for learning design or learning design for learn...
WCOL2019: Learning analytics for learning design or learning design for learn...
 
ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015
 
VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...
VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...
VII Jornadas eMadrid "Education in exponential times". "Student-facing learni...
 
Designing Learning Analytics Dashboards: Lessons Learned
Designing Learning Analytics Dashboards: Lessons LearnedDesigning Learning Analytics Dashboards: Lessons Learned
Designing Learning Analytics Dashboards: Lessons Learned
 
HEI Fellowship: Blog Post 1
HEI Fellowship: Blog Post 1HEI Fellowship: Blog Post 1
HEI Fellowship: Blog Post 1
 

Recently uploaded

1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx
JosvitaDsouza2
 
How to Create Map Views in the Odoo 17 ERP
How to Create Map Views in the Odoo 17 ERPHow to Create Map Views in the Odoo 17 ERP
How to Create Map Views in the Odoo 17 ERP
Celine George
 
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptxMARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
bennyroshan06
 
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
MysoreMuleSoftMeetup
 
Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)
rosedainty
 
Sectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdfSectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdf
Vivekanand Anglo Vedic Academy
 
Basic phrases for greeting and assisting costumers
Basic phrases for greeting and assisting costumersBasic phrases for greeting and assisting costumers
Basic phrases for greeting and assisting costumers
PedroFerreira53928
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
Jisc
 
How to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleHow to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS Module
Celine George
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...
Sandy Millin
 
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
AzmatAli747758
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
TechSoup
 
Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......
Ashokrao Mane college of Pharmacy Peth-Vadgaon
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
GeoBlogs
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
Tamralipta Mahavidyalaya
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
Balvir Singh
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
Atul Kumar Singh
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
Jisc
 

Recently uploaded (20)

1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx
 
How to Create Map Views in the Odoo 17 ERP
How to Create Map Views in the Odoo 17 ERPHow to Create Map Views in the Odoo 17 ERP
How to Create Map Views in the Odoo 17 ERP
 
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptxMARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
 
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
 
Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)
 
Sectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdfSectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdf
 
Basic phrases for greeting and assisting costumers
Basic phrases for greeting and assisting costumersBasic phrases for greeting and assisting costumers
Basic phrases for greeting and assisting costumers
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
 
How to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleHow to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS Module
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...
 
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
 
Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
 

Scalable, Actionable, and Ethical Learning Dashboards: a reality check

  • 2. Scalable, Actionable, and Ethical Learning Dashboards a reality check Tinne De Laet Tinne.DeLaet@kuleuven.be @TinneDeLaet
  • 3. Who am I? Why I am here? woman phdrobotics Head Tutorial Services Engineering Science KU Leuven . associate professor mech.engine er
  • 4. founded 1425 research-intensive university 16 faculties → general university > 55 000 students 2016-2017-2018 Europe’s most innovative university (Reuters ranking)
  • 5. “Learning analytics is about collecting traces that learners leave behind and using those traces to improve learning.” - Erik Duval Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012, https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/ 5 Learning Analytics?
  • 6. Learning Dashboards? 6Dashboard Confusion, Stephen Few, Intelligent Enterprise, March 20, 2004 “A dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.” - Stephen Few
  • 7. Successful Transition from secondary to higher Education using Learning Analytics enhance a successful transition from secondary to higher education by means of learning analytics  design and build analytics dashboards,  dashboards that go beyond identifying at-risk students, allowing actionable feedback for all students on a large scale. Achieving Benefits from Learning Analytics research strategies and practices for using learning analytics to support students during their first year at university  developing the technological aspects of learning analytics,  focuses on how learning analytics can be used to support students. 7 www.stela-project.eu @STELA_project 2015-1-UK01-KA203-013767 www.ableproject.eu @ABLE_project_eu 562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD
  • 8. STELA ♥ ABLE 8 actionable feedback student-centered program level inclusive first-year experience institution-wide Learning Analytics actual implementation
  • 9. [!] Feedback must be “actionable”. 9 Warning! Male students have 10% less probability to be successful. You are male. Warning! Your online activity is lagging behind. action? ? action? ? 
  • 10. 10 awareness (self-)reflection sensemaking impact data questions answers behavior change new meaning Verbert K, Duval E, Klerkx J; Govaerts S, Santos JL (2013) Learning analytics dashboard applications. American Behavioural Scientist, 10 pages. Published online February 2013. [!] Feedback must be “actionable”.
  • 11. learning dashboards @KU Leuven interaction self-reflection LISSA STUDENT ADVISOR STUDENT LASSI – learning skills REX - scoresPOS – future students
  • 12. [!] Start with the available data. Lots of data may eventually become available in the future … …. already start with what is available 12 (*) (*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication. British Journal of Educational Technology, 44(4), 616-628.
  • 13. Case study dashboard interaction student – study advisor
  • 14. Study advisor – student conversations 14 Should I consider another program? Can I still finish the bachelor in 3 years? How should I compose my program for next year? What is the personal situation? How can I help? What is the best next step?
  • 15. [!] Use all available expertise. 15 visualization experts practitioners / end-users researchers LA researchers first-year study success Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology (http://ieeexplore.ieee.org/document/7959628/).
  • 17. [!] Wording matters. 17 73% chance of success 73% of students of earlier cohorts with the same study efficiency obtained the bachelor degree http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
  • 18. LISSA: status 18 26 programs >4500 students 114 student advisors training of study advisors • Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology • http://blog.associatie.kuleuven.be/tinnedelaet/lissa-learning-dashboard-supporting-student-advisers-in-traditional-higher-education/ • Millecamp M., Gutiérrez F., Charleer S., Verbert K., De Laet T.# (2018). A qualitative evaluation of a learning dashboard to support advisor-student dialogues. Proceedings of the 8th International Learning Analytics & Knowledge Conference. LAK. Sydney, 5-9 March 2018 (pp. 1-5) ACM. dashboards for three examination periods observations, interviews, questionnaires
  • 19. LISSA: evaluation – observations 19 15 observations insights (-) factual (+) interpretative (!) reflective Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology
  • 20. Evaluation – interviews “When students see the numbers, they are surprised, but now they believe me. Before, I used my gut feeling, now I feel more certain of what I say as well”. “It’s like a main thread guiding the conversation.” “I can talk about what to do with the results, instead of each time looking for the data and puzzling it together.” “Students don’t know where to look during the conversation, and avoid eye contact. The dashboard provides them a point of focus”. “A student changed her study method in June and could now see it paid off.” LISSA supports a personal dialogue.  the level of usage depends on the experience and style of the study advisors  fact-based evidence at the side  narrative thread  key moments and student path help to reconstruct personal track “I can focus on the student’s personal path, rather than on the facts.” “Now, I can blame the dashboard and focus on collaboratively looking for the next step to take.” 20
  • 21. LISSA: evaluation – student questionnaires 21 26 programs @KU Leuven 291 student questionnaires first examination period “Confronting, but useful” “I want to use this dashboard at home.” “Also show the sub-grades for labs, … ” “How can I know the data is trustworth?” “Can’t these visualizations be send to students?” “Crisp and clear.”
  • 22. 22 0 0 1 1 1 1 4 2 1 4 4 3 29 21 36 37 49 42 176 112 156 132 141 169 80 155 93 116 92 72 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1. The dashboard is clarifying and surveyable. 2. The shown information regarding my study situation is correct. 3. The shown position with respect to my fellow students (histograms per exam and global… 4. A conversation with my student advisors helped me to gain insight in my study trajectory. 5. The visualisation is of added value to the conversation with the student advisor. 6. The shown information provide me insight in my current situation. Student questionnaire January 2018 (N=291) Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree
  • 23. [!] Do not oversimplify. Show uncertainty. 23 • reality is complex • measurement is limited • individual circumstances • need for nuance • trigger reflection http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
  • 24. [!] Be careful with predictive algorithms. 24 http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/ • reality is complex • measurement is limited • individual circumstances • need for nuance • trigger reflection
  • 26. [!] Start with the available data. 26 data already available? administrative (examples) student records course grades systems (examples) LMS access logs advisor meetings ) Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Small data as a conversation starter for learning analytics: exam results dashboard for first-year students in higher education. Journal of Research in Innovative Teaching & Learning, , 1-14. demo: https://learninganalytics.set.kuleuven.be/static-demo-rex/ (en) or https://learninganalytics.set.kuleuven.be/demo/rex-1718jan-ir (nl)
  • 27. [!] Think beyond the obvious data. 27 • Don’t think too traditional. • Many institutions are collecting survey data for educational research.
  • 28. [!] Not all data is usable. 28 example data from a traditional course with “VLE as a file system” test scores activity/week (#days) weeks of the year
  • 29. [!] Not all data is usable. 29 example data from a course with flipped classroom & blended learning exam scores activity (# of modules used) Not a single student using less than 10 modules passed the course. Most of the successful students used 15 modules or more.
  • 30. [!] Keep Learning Analytics in mind when designing learning activities. 30 Learning Analytics Learning Design INFORM ENABLE If LA indeed contributes to improved learning design… … don’t make it an afterthought
  • 31. 31 Does my concentration matter? How is my time management? I feel uncertain. Is this normal? How can I improve my concentration?
  • 32. data already available? administrative (examples) student records course grades [!] Think beyond the obvious data. 32 systems (examples) LMS access logs advisor meetings surveys (examples) quality insurance LASSI
  • 33. ~ 30 LASSI questions (shortened version) “Learning Skills” Example: When preparing for an exam, I create questions that I think might be included. Example: I find it difficult to maintain my concentration while doing my coursework. Example: I find it hard to stick to a study schedule. raw scores (selected 5 out of 10) CONCENTRATION MOTIVATION FAILURE ANXIETY TEST STRATEGY TIME MANAGEMENT norm scores (in Flemish HE context) Example: STRONG Example: AVERAGE Example: LOW Example: VERY STRONG Example: VERY WEAK 33  Meta cognitive abilities Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017). readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo
  • 34. Dashboard learning skills 34 students complete LASSI questionnaire students received personalized email with invitation for dashboard 4367 students in 26 programs in 9 faculties @KU Leuven demo: https://learninganalytics.set.kuleuven.be/static-demo-lassi/ (en) of https://learninganalytics.set.kuleuven.be/demo/lassi-1718 (nl) 2 programs @TU Delft
  • 35. Feedback model 1. What is this about? 2. How am I doing? 3. How does this relates to others? 4. Why is this relevant? 5. What can I do about it? 35
  • 36. 36 3. How does this relates to others? 2. How am I doing? 1. What is this about? @studyProgram@ @yourScore@
  • 37. 4. Why is this relevant? 5. What can I do about it? 37
  • 38. 38 5. What can I do about it?
  • 41. Students that click through Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness. In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham. 41  better learning skills
  • 42. More intense users Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness. In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham. 42  worse learning skills
  • 43. What can we learn from dashboard usage? 43Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and Microinteractions. accepted for ECTEL 2018 p<1e-5 p<1e-9p-test
  • 44. [!] Give students “the key”. 44 • Student has the key to own data. • Student takes initiative to share/discuss own data. • GDPR as opportunity!
  • 45. [!] Acceptance precedes impact. 45 • Involve stakeholders from the start and value their input! COmmunication COoperation • Demonstrate usefulness. • Take care of ethics and privacy. • Best scenario: students & study advisors as ambassadors COCO
  • 46. Impact? survey before intervention  2nd year students 2016-2017  experiences first-year feedback  41 vragen, 5-point Likert scale  pen & paper dashboards  LISSA  LASSI (learning skills)  3 x REX (grades) Survey after intervention  2nd year students 2017-2018
  • 47. Impact? During the first year I received sufficient information regarding my academic achievements. 47 Engineering Science (p<0.001)
  • 48. Impact? The information I received helped to position myself with respect to my peers. 48 Engineering Science (p<0.001)
  • 49. Impact? 49 The information I received made me reflect. The information I received made me adapt my behaviour.
  • 50. [!] Context matters! • available data • national and institutional regulations and culture • educational vision • educational system, size of population .. • … Don’t just copy existing LA solutions! 50
  • 51. Summary case studies 11 findings/recommendations [!] Use all available expertise. [!] Start with the available data. [!] Look beyond the obvious data. [!] Not all data is usable. [!] Wording matters. [!] Don’t oversimplify. Show uncertainty. [!] Beware of predictive algorithms. [!] Keep Learning Analytics in mind when designing learning activities. [!] Give students “the key” to their data. [!] Acceptance precedes impact. [!] Context matters!  humble approach  small data  involvement of stakeholders, especially practitioners  actionable feedback  scalability  traditional university settings Is this Learning Analytics?
  • 52. Future? 52 Continue and extend dashboards @KU Leuven? Transfer to other universities? extension?
  • 53. Project team @ 53 Sven Charleer AugmentHCI, Computer Science department PhD researcher ABLE Katrien Verbert AugmentHCI, Computer Science department Copromotor of STELA & ABLE Carolien Van Soom Leuven Engineering and Science Education Center Head of Tutorial Services of Science Copromotor of STELA & ABLE Greet Langie Leuven Engineering and Science Education Center Vicedean (education) faculty of Engineering Technology Copromotor of STELA & ABLE Tinne De Laet Leuven Engineering and Science Education Center Head of Tutorial Services of Engineering Science Coordinator of STELA KU Leuven coordinator of ABLE Francisco Gutiérrez AugmentHCI, Computer Science department PhD researcher ABLE Tom Broos Leuven Engineering and Science Education Center AugmentHCI, Computer Science department PhD researcher STELA Martijn Millecamp AugmentHCI, Computer Science department PhD researcher ABLE Special thanks to study advisors for their cooperation, advice, feedback, and support! Jasper, Bart, Riet, Hilde, An, Katrien, … ♥
  • 54. Response - REX 54 571 76 245 220 353 258 274 1060 187 283 843 242 163 38 130 120 200 151 162 647 128 230 771 269 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% clicked non clicked
  • 55. Evaluation - REX 55 22 26 105 443 511 0 100 200 300 400 500 600 1 2 3 4 5 useful (stars) 26 29 134 441 429 0 100 200 300 400 500 600 1 2 3 4 5 clear (stars) 41 60 220 385 456 0 100 200 300 400 500 600 1 2 3 4 5 added value (stars)
  • 56. Micro-interactions - REX 56 defaultalternative ) Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and Microinteractions. accepted for ECTEL 2018
  • 57. Micro-interactions 57Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and Microinteractions. accepted for ECTEL 2018
  • 58. 58 2 3 1 7 2 1 10 44 23 36 3 11 64 97 81 74 36 29 150 115 126 110 119 91 61 30 56 59 128 157 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 7. The dashboard makes me more aware on my current study situation. 8. The dashboard makes me forecast the different possibilities in my future study trajectory. 9. The dashboard helps me to reflect on my past and current study behaviour or study trajectory. 10. The dashboards stimulates me to adapt my approach in my studies for the future (study behaviour or study… 11. If I will have a new conversation after one of the next examination periods, I hope that the visualisation will be… 12. I would like to consult the information on my own. Student questionnaire January 2018 (N=291) Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree
  • 59. How to determine thresholds for different groups? LISSA dashboard 59  upper and lower group: clear message  middle group as small as possible  Do not overfit! (nuance)

Editor's Notes

  1. e.g. previous GPA, attendance, running scores
  2. Meer continue feedback. Focus op welbevinden. Tijdig studenten opmerken die het moeilijk hebben met de aanpassing en die pro-actief feedback geven maar ook contacteren.