The current state of higher education has increasing pressure over academics to offer high quality experience at scale. But what could be the actions that can be deployed to achieve this increase? What would be a good guiding principle to decide these actions? In this talk we explore first the possibility of using feedback and a coach mentality to provide student support, and then how data can help us scale that technique. There are examples of potential scenarios to deploy this at the level of a course, program or overall student experience.

1. Using data to provide
personalised feedback at scale
Prof Abelardo Pardo
Division of IT, Engineering and the Environment
slideshare.net/abelardo_pardo Twitter: @abelardopardo
NickMacMillan

2. Abelardo Pardo Using data to provide personalised feedback at scale 2
JonnyCaspari
Under
Pressure
Can
Data
Help?
Which
Actions?
Examples

3. Abelardo Pardo Using data to provide personalised feedback at scale 3
JonnyCaspari
Under
Pressure

4. Abelardo Pardo Using data to provide personalised feedback at scale 4
Simple information transfer is not working
Mazur, E. (2009). Farewell, lecture. Science, 323(5910), 50-51.
Krugazorflickr.com

5. Abelardo Pardo Using data to provide personalised feedback at scale 5
SamAbrahamflickr.com
Blended Learning
Frontier between physical and virtual spaces is blurring

6. Abelardo Pardo Using data to provide personalised feedback at scale 6
JansonHewsflickr.com

7. Abelardo Pardo Using data to provide personalised feedback at scale 7
Use a
theoretical
model

8. Abelardo Pardo Using data to provide personalised feedback at scale 8
“… robust correlations between student
involvement in a subset of ‘educationally
purposive activities’, and positive outcomes of
student success and development, including
satisfaction, persistence, academic achievement
and social engagement”
Trowler, V. (2010). Student engagement literature review. York, UK: The Higher Education Academy.
shuaGandersonflickr.com

9. Abelardo Pardo Using data to provide personalised feedback at scale 9
JonnyCaspari
Under
Pressure
Which
actions?

10. Abelardo Pardo Using data to provide personalised feedback at scale 10
“… teaching in higher education will
necessarily shift the balance of its efforts
towards a greater investment in design as
a way of coping with otherwise intolerable
pressures on staff and resources.”
Goodyear, P. (2015). Teaching as Design. HERDSA Review of Higher Education, 2, 27-50.

11. Abelardo Pardo Using data to provide personalised feedback at scale 11
“There is no such thing as a neutral design”
JeremyBrooksflickr.com
Thaler, R. H., & Sunstein, C. R. (2008). Nudge. Great Britain: Yale University Press.

12. Abelardo Pardo Using data to provide personalised feedback at scale 12
Thaler, R. H., & Sunstein, C. R. (2008). Nudge. Great Britain: Yale University Press.
“People make good choices in contexts in
which they have experience, good
information, and prompt feedback”
DerekBruffflickr.com

13. Abelardo Pardo Using data to provide personalised feedback at scale 13
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: beliefs, techniques, and illusions. Annu Rev Psychol, 64,
417-444. doi:10.1146/annurev-psych-113011-143823
• Understand
human memory
and learning
• Know useful study
techniques
MatthewMaillet
• Know how to
monitor
• Understand
existing biases

14. Abelardo Pardo Using data to provide personalised feedback at scale 14
Hattie, J. A. (1999). Influences on student learning. Inaugural professorial address, University of Auckland, New Zealand
If You Could Choose One…
• More than 500 meta-analyses
of student achievements
• 100 factors with potential
influence
• Feedback in top five
• (74 meta-analyses) Most
effective form: video, audio,
computer-assisted
instructional feedback, and/or
related goals

15. Abelardo Pardo Using data to provide personalised feedback at scale 15
Pardo, A. (2017). A feedback model for data-rich learning experiences. Assessment & Evaluation in Higher Education, 1-11. doi:
10.1080/02602938.2017.1356905
FarukAteşflickr.com
“Feedback is a process to positively influence how students
engage with their work in a learning experience so that they can
improve its overall quality with respect to an appropriate reference
and increase their self-evaluative capacity”

16. Abelardo Pardo Using data to provide personalised feedback at scale 16
JacksonLavarnway

17. Abelardo Pardo Using data to provide personalised feedback at scale 17
JonnyCaspari
Under
Pressure
Can
Data
Help?
Can
Data
Help?

18. Abelardo Pardo Using data to provide personalised feedback at scale 18
Large number of events per user
2016-08-09 00:05:43.124199+00,989,129.78.56.144,"{""outcome"": ""incorrect"", ""assessment"":
""summative"", ""score"": 44.4444444444444, ""exercise"": ""/data2u/static/exco_exc/DRM/
problem_13_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196619,exco-answer
2016-08-09 00:05:44.140307+00,989,129.78.56.144,"{""score"": 44.4444444444444, ""exercise"": ""/data2u/
static/exco_exc/DRM/problem_03_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196620,exco-view
2016-08-09 00:05:47.76122+00,968,49.182.128.186,"{""question_id"":""COD-numberofwires-eqt_1"",""answer"":
1}",abelardopardo,196621,embedded-question
2016-08-09 00:05:48.861036+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":0}",abelardopardo,196622,embedded-question
2016-08-09 00:05:48.959791+00,989,129.78.56.144,"{""outcome"": ""correct"", ""assessment"":
""summative"", ""score"": 50.0, ""exercise"": ""/data2u/static/exco_exc/DRM/problem_03_c.html"",
""sequence"": ""DRM-exco-C""}",abelardopardo,196623,exco-answer
2016-08-09 00:05:49.539162+00,989,129.78.56.144,"{""score"": 50.0, ""exercise"": ""/data2u/static/
exco_exc/DRM/problem_14_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196624,exco-view
2016-08-09 00:05:49.571138+00,1311,129.78.56.159,"{""url"":""https://flip.ee.usyd.edu.au/elec1601/
Material/COD/index.html""}",abelardopardo,196625,resource-view
2016-08-09 00:05:50.050035+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":0}",abelardopardo,196626,embedded-question
2016-08-09 00:05:51.709295+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":0}",abelardopardo,196627,embedded-question
2016-08-09 00:05:51.962474+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":0}",abelardopardo,196628,embedded-question
2016-08-09 00:05:52.0819+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":""-1""}",abelardopardo,196629,embedded-question
2016-08-09 00:05:53.025356+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":1}",abelardopardo,196630,embedded-question
2016-08-09 00:05:54.756229+00,1311,129.78.56.159,"{""url"":""https://flip.ee.usyd.edu.au/elec1601/
Material/COD/COD_notes.html#range-accuracy-and-precision-of-the-floating-point-
representation""}",abelardopardo,196631,resource-view
2016-08-09 00:05:54.856333+00,989,129.78.56.144,"{""outcome"": ""incorrect"", ""assessment"":
""summative"", ""score"": 50.0, ""exercise"": “"/data2u/static/exco_exc/DRM/problem_03_c.html"",
""sequence"": ""DRM-exco-C""}",abelardopardo,196623,exco-answer

19. Abelardo Pardo Using data to provide personalised feedback at scale 19
4187,
2016-07-14 00:56:46.341946+00,
{“time”":0,"id":"xEJtdMQMcrs","event":"PLAY"},
https://mycourse.com/Material/HLP/HLP_notes.html,
embedded-video
4187, 2016-07-14 00:56:46.341946+00, VIDEO, xEJtdMQMcrs, PLAY, COD, W2
Combine logs with design
Learner played a video about topic COD from Week 2

20. Abelardo Pardo Using data to provide personalised feedback at scale 20
4187,
2016-07-14 00:56:46.341946+00,
{“time”":0,"id":"xEJtdMQMcrs","event":"PLAY"},
https://mycourse.com/Material/HLP/HLP_notes.html,
embedded-video
4187, 2016-07-14 00:56:46.341946+00, VIDEO, xEJtdMQMcrs, PLAY, COD, W2
Combine logs with design
Learner played video about COD from W2 during W6
For the first time during
the course — The
student is catching up
For the sixth time during
the course — The
student is reviewing

21. Abelardo Pardo Using data to provide personalised feedback at scale 21
Combine events to obtain tactics
Tactic 1
• Review videos and
immediately answer questions
(or annotate).
• Review suggested additional
material.
• Engage with some practice
exercises.
• Engage in discussion in forum
Tactic 2
• Try to solve the summative
assessment questions. Most
attempts are incorrect
• Review course structure
(assessment, requirements)
• Consult a small subset of
available resources

22. Abelardo Pardo Using data to provide personalised feedback at scale 22
Identify learning strategies
Strategy 1
• Numerous short sessions with
comprehensive levels of
engagement.
• Sessions spread across time.
• Frequent self-reflection
elements.
• Numerous engagements with
discussion forum.
Strategy 2
• Infrequent long sessions only
focused in one type of
resource.
• Sessions always close to
submission deadlines.
• Poor variety of tactics.

23. Abelardo Pardo Using data to provide personalised feedback at scale 23
JonnyCaspari
Under
Pressure
Can
Data
Help?
Which
Actions?
Examples

24. Abelardo Pardo Using data to provide personalised feedback at scale 24
MikeWilson
Course
Degree or program
Student experience

25. Abelardo Pardo Using data to provide personalised feedback at scale 25
Pardo, A., & Mirriahi, N. (2017). Design, Deployment and Evaluation of a Flipped Learning First-Year Engineering Course. In C.
Reidsema, L. Kavanagh, R. Hadgraft, & N. Smith (Eds.), Flipping the classroom. Practice and Practices in Higher Education (pp.
177-191). Singapore: Springer. doi:10.1007/978-981-10-3413-8_11
Blended Learning Experience
• 13 week experience
• 1 lecture (2hr), 1 tutorial (2hr), 1 laboratory session (3hr) per week
• Videos with immediate MCQ for recall (formative assessment)
• Resources with embedded formative assessment items
• Preparation exercises required (summative) before lectures, and then
available for practice (formative)
• Self-reflection and awareness elements available

26. Abelardo Pardo Using data to provide personalised feedback at scale 26
Fincham, E., Gašević , D., Jovanović, J., & Pardo, A. (In Press). From Study Tactics to Learning Strategies: An Analytical Method
for Extracting Interpretable Representations. IEEE Transactions on Learning Technologies.
Detecting Learning Tactics in Sessions
1. 60% video, 40% content
2. Almost exclusively summative assessment (90%)
3. Almost exclusively formative assessment or practice (90%)
4. Content browsing (90%)
5. Meta-cognitive actions (90%). Browsing the self-reflection elements, checking
pages about how the course is organised, suggested strategies, etc.
6. Content access and formative assessment (90%)
7. Access to content and meta-cognitive elements.
8. Formative assessment, summative assessment and content access.

27. Abelardo Pardo Using data to provide personalised feedback at scale 27
Fincham, E., Gašević , D., Jovanović, J., & Pardo, A. (In Press). From Study Tactics to Learning Strategies: An Analytical Method
for Extracting Interpretable Representations. IEEE Transactions on Learning Technologies.
Identifying Learning Strategies

28. Abelardo Pardo Using data to provide personalised feedback at scale 28
Jovanović, J., Gašević, D., Pardo, A., Mirriahi, N., & Dawson, S. (In Press). An analytics-based framework to support teaching
and learning in a flipped classroom. In J. M. Lodge, J. Cooney Horvath, & L. Corrin (Eds.), Learning analytics in the classroom:
translating learning analytics research for teachers. United Kingdom: Taylor & Francis.
Support teaching and learning in a flipped classroom

29. Abelardo Pardo Using data to provide personalised feedback at scale 29
Instructor — per task
Technology — per student

30. Abelardo Pardo Using data to provide personalised feedback at scale 30

31. Abelardo Pardo Using data to provide personalised feedback at scale 31
ontasklearning.org
Pardo et al. OnTask: Delivering Data-Informed, Personalised Learning Support Actions. Journal of Learning Analytics. In Press

32. Abelardo Pardo Using data to provide personalised feedback at scale 32

33. Abelardo Pardo Using data to provide personalised feedback at scale 33

34. Abelardo Pardo Using data to provide personalised feedback at scale 34
• Support instructors to
manage personalised
feedback processes
• Simple rule-base
knowledge encoding
• Provide appropriate
view of data sources
• Scale to large and highly
diverse cohorts
• Open-source project
• Pilots running in 2018
• Contact us if interested
ontasklearning.org

35. Abelardo Pardo Using data to provide personalised feedback at scale 35
Focus groups
• “It helps me to validate where I am; do I need to
freak out right now?”
• “…gives you a nudge — Stop procrastinating and
playing games!”
• A reminder to study “across the board” (flow-on
effect)
• “The wording makes you want to do it. Like an
encouragement.”

36. Abelardo Pardo Using data to provide personalised feedback at scale 36
Program
objective 1
Program
objective 2
Program
objective 3
Program
objective4
1st year course 1 Basic Basic
1st year course 2 Basic
2nd year course 1 Proficient Proficient Basic
2nd year course 2 Proficient Proficient
3rd year course 1 Expert Expert Expert Expert
Support at the Degree/Program Level

37. Abelardo Pardo Using data to provide personalised feedback at scale 37
Program
objective 1
Program
objective 2
Program
objective 3
Program
objective4
1st year course 1 Basic Basic
1st year course 2 Basic
2nd year course 1 Proficient Proficient Basic
2nd year course 2 Proficient Proficient
3rd year course 1 Expert Expert Expert Expert
Support at the Degree Level
• Indicators of student engagement with respect to program objectives
• Tactics adopted in courses assessing each objective
• Provide personalized feedback to support achievement

38. Abelardo Pardo Using data to provide personalised feedback at scale 38
Program
objective 1
Program
objective 2
Program
objective 3
Program
objective4
1st year course 1 Basic Basic
1st year course 2 Basic
2nd year course 1 Proficient Proficient Basic
2nd year course 2 Proficient Proficient
3rd year course 1 Expert Expert Expert Expert
Identifying Degree Trajectories
• Provide personalized feedback to support trajectory

39. Abelardo Pardo Using data to provide personalised feedback at scale 39
James, N., Kovanovic, V., Marshall, R., Joksimovic, S., & Pardo, A. (2018). Examining the value of learning analytics for
supporting work-integrated learning. Paper presented at the National Conference on Work Integrated Learning, Brisbane,
Queensland, Australia.
SalzburgGlobalSeminar
Preparing graduates for work
• Identify indicators of progress in: social integration, soft skills,
community awareness.
• Connect with institutional resources
• Provide suggestions on how to engage or increase effectiveness

40. Abelardo Pardo Using data to provide personalised feedback at scale 40
RyoYoshitake
• Institutions are under pressure to provide quality at scale.
• Knowledge must be connected with actions at course,
program overall student experience levels.
• Data captured through technology mediation can provide
valuable insights for decision making.
• Deliver personalise actions based on knowledge
extracted from data.
Conclusions

41. Using data to provide
personalised feedback at scale
Prof Abelardo Pardo
Division of IT, Engineering and the Environment
slideshare.net/abelardo_pardo Twitter: @abelardopardo
NickMacMillan