Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The University of Sydney Page 1
Bottom-up growth of learning analytics at two
Australian universities:
Empowering staff wi...
The University of Sydney Page 2
“We found that mature foundations for LA implementations were
identified in institutions t...
The University of Sydney Page 3
1. Macquarie: Empowering staff with actionable LMS data
2. Sydney: Learning analytics by s...
Empowering staff with actionable LMS data
THE MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
February 2016
Learning analytics in Moodle
MOTIVATIONS AND BENEFITS
• Large(ish) classes, failure
and retention issues
• Staff familiari...
Enhancing an existing plugin
6
MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
• Originally developed by Phillip Dawson, Adam Ol...
Involving staff
7
ACADEMICS AND STUDENT SUPPORT
Staff expectations around an
early alert system
Prototyping and developmen...
MEAP
8
CAPABILITIES AND APPLICATIONS
Stakeholder outcomes
9
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• 3400+ personalised emails sent, average ~46% opened
0
50
...
Stakeholder outcomes
10
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• Who: unit
convenors and
student support
staff
• What: ce...
Stakeholder outcomes
11
THE DARK SIDE
• Issues around
• Message
composition
• Suggestions
• Student contexts
• Being label...
Stakeholder outcomes
12
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• Was there an effect?
Online activity Online activityOnli...
Stakeholder outcomes
13
PERSONALISED, DATA-DRIVEN INTERVENTIONS
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
Forum Login
Changeinri...
Back to the users
14
FURTHER ITERATIVE DEVELOPMENT
Empowering staff with data
15
MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
Moodle
Report
Logins Forums Assessments
Parameters...
Lessons learned and next steps
16
LOOKING BACK AND LOOKING FORWARD
• Looking back
• Talk to the users
• Find champions
• S...
The University of Sydney Page 17
1. Macquarie: Empowering staff with actionable LMS data
2. Sydney: Learning analytics by ...
The University of Sydney Page 18
Learning analytics by
stealth
The Student Relationship Engagement System
(SRES)
Standing ...
The University of Sydney Page 19
The contexts of learning analytics
Common barriers to adoption
– Policy and ethical
chall...
The University of Sydney Page 20
The Student Relationship Engagement System
Attendance
Interim
grades
LMS
metrics
Third pa...
The University of Sydney Page 21
Personalising connections with students
– Empowering staff
– Flexible & intuitive
– Targe...
The University of Sydney Page 22
Stakeholder outcomes
Discontinued
Failed
Passed
F
P
C
D
HD
1st year arts unit ~500 enrolm...
The University of Sydney Page 23
Co-evolution of the SRES
– Organic adoption by academics
– Co-evolution of capabilities
0...
The University of Sydney Page 24
Learning analytics by stealth?
– Co-evolving capabilities and competencies around data-dr...
The University of Sydney Page 25
Learning analytics by stealth?
– Co-evolving capabilities and competencies around data-dr...
The University of Sydney Page 26
The contexts of learning analytics
Common barriers to adoption
– Policy and ethical
chall...
The University of Sydney Page 27
Lessons learned and next steps
– Looking back
– It was ugly but it worked
– Ease of use i...
The University of Sydney Page 28
1. Macquarie: Empowering staff with actionable LMS data
2. Sydney: Learning analytics by ...
Building institutional readiness
through openness
THE MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)
February 2016
Our approach
30
CONNECTING USERS WITH DATA THROUGH ANALYTICS
Macquarie
Open
Analytics
Toolkit
Data
Users
Analytics
LMS
Vid...
Bringing data together
31
NUANCES OF LEARNING DATA
LMS
Video
Classrooms
Mobile
Business
systems
External
courses
Learning
...
Empowering staff with data
32
MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)
Empowering staff with data
33
MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)
Lessons learned and next steps
34
LOOKING BACK AND LOOKING FORWARD
• Looking back
• Multi-disciplinary, multi-level teams
...
The University of Sydney Page 35
Lessons learned and
issues raised
The University of Sydney Page 36
Lessons learned and issues raised
– Give them what they want vs. build it and they will c...
The University of Sydney Page 37
Adoption pipeline
Colvin et al. (2015) Student retention and learning analytics: A snapsh...
The University of Sydney Page 38
Learning analytics is not an elixir for ineffective teaching,
nor does it reveal an ideal...
The University of Sydney Page 39
1. MEAP Empowering staff with actionable LMS data
Chris Froissard, Deborah Richards, Amar...
Upcoming SlideShare
Loading in …5
×

Bottom-up growth of learning analytics at two Australian universities: Empowering staff with actionable intelligence to improve student outcomes

408 views

Published on

University of Otago, February 2016

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Bottom-up growth of learning analytics at two Australian universities: Empowering staff with actionable intelligence to improve student outcomes

  1. 1. The University of Sydney Page 1 Bottom-up growth of learning analytics at two Australian universities: Empowering staff with actionable intelligence to improve student outcomes @dannydotliu danny.liu@mq.edu.au danny.liu@sydney.edu.au
  2. 2. The University of Sydney Page 2 “We found that mature foundations for LA implementations were identified in institutions that adopted a rapid innovation cycle whereby small scale projects are initiated and outcomes quickly assessed within short time frames. The successful projects in this cycle are then further promoted for scale-ability and mainstream adoption. In the context of LA, this small-scale seeded approach appeared more effective in terms of organisational acceptance and adoption than a whole of institution model attempting to roll out a single encompassing program.” Learning Analytics in Australia: Office for Learning & Teaching 2016
  3. 3. The University of Sydney Page 3 1. Macquarie: Empowering staff with actionable LMS data 2. Sydney: Learning analytics by stealth 3. Macquarie: Building institutional readiness through openness
  4. 4. Empowering staff with actionable LMS data THE MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP) February 2016
  5. 5. Learning analytics in Moodle MOTIVATIONS AND BENEFITS • Large(ish) classes, failure and retention issues • Staff familiarity with Moodle, single point of access • Learning experience data already centralised • No available learning analytics tool with actionable data Log viewer Statistics report
  6. 6. Enhancing an existing plugin 6 MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP) • Originally developed by Phillip Dawson, Adam Olley, and Ashley Holman Moodle Report Logins Forums Assessments Parameters Traffic lights BYO Moodle Engagement Analytics Plugin Indicators Action
  7. 7. Involving staff 7 ACADEMICS AND STUDENT SUPPORT Staff expectations around an early alert system Prototyping and development User testing Piloting Feedback and further development
  8. 8. MEAP 8 CAPABILITIES AND APPLICATIONS
  9. 9. Stakeholder outcomes 9 PERSONALISED, DATA-DRIVEN INTERVENTIONS • 3400+ personalised emails sent, average ~46% opened 0 50 100 150 200 250 300 350 01Aug 08Aug 15Aug 22Aug 29Aug 05Sep 12Sep 19Sep 26Sep 03Oct Numberofstudents 0 20 40 60 80 100 AreyouhavingproblemsaccessingiLearn? ACCG250-Ifyousnoozeyoulose! AreyouhavingtroubleloggingintoiLearn? AreyouhavingtroubleloggingintoiLearninACCG100? MECH201:EngineeringDynamics-iLearnActivity ENGG150quizfeedback ECON110iLearnLogin AHIS140MythintheAncientWorld AHIS140MythintheAncientWorld AreyouhavingtroubleloggingintoECON111 AHIS140Session2Participation AHIS140MythintheAncientWorld Quiz1 Quiz1Result-Afin253 Ilearn-Afin253 ACST101-Quiz1 ECON110TutorialSubmissions ECON111-Tutorialscontributetoyourfinalgrade ECON111-Tutorialscontributetoyourfinalgrade MECH202Progressthusfar MECH203Progressthusfar ISYS114 ACST101-Lastdaytowithdrawwithoutfailchargeis… ISYS114Performance ISYS114weeklytutorialsubmissions ISYS114AttendanceandPerformance ISYS114WorkshopAttendanceandPerformance ISYS114Workshopattendance Tutorialfeedback Tutorialparticipation TutorialparticipationforEUL101SocietiesofEurope ACST101-Mondayisthelastdaytowithdrawwithout… ACST101-Mondayisthelastdaytowithdrawwithout… ACST101-Mondayisthelastdaytowithdrawwithout… Itsnottoolatetogethelp! ACST101-Mondayisthelastdaytowithdrawwithout… ECON111-Pleasecheckyourtutorialrecords ECON111-pleasecheckyourtutorialrecords Percentageopened 1 hour 1 day 1 weekEmail opened within:
  10. 10. Stakeholder outcomes 10 PERSONALISED, DATA-DRIVEN INTERVENTIONS • Who: unit convenors and student support staff • What: census, updates, reminders • Why: predominantly for at-risk students • How: logins, assessment submissions, grades, attendance I was surprised someone cared/was actually monitoring, kind of a weird, I don't know totalitarian/'people are watching you' feeling? But in this situation I was happy. He gave me specific advice and encouraged me and it made me feel much better. The email basically kicked me into gear and I completed all my assessments post-email to a high level. Very useful. I wouldn't have been able to do such a large scale analysis and identify so many students without MEAP. I wouldn't have been able to send them such tailored, structured and consistent messages.
  11. 11. Stakeholder outcomes 11 THE DARK SIDE • Issues around • Message composition • Suggestions • Student contexts • Being labelled I was contacted but in a manner that suggest I should drop out of the course and not waste the convenor's time. She didn't ask whether I was experiencing any issues outside of university, but simply that I should transfer course if I can't handle the workload. Being labelled as lazy when you're doing your best and don't have any other choice is quite sad. Being aware and then being told of my own inadequacies is confronting and, yes, does make me feel worse about life in general. It's something I need to be told and is still that extra bit of motivation. The email was worded in a way that it the unit [convenor] was trying to tell me I was doing horrible and should drop out and didn't refer any help.
  12. 12. Stakeholder outcomes 12 PERSONALISED, DATA-DRIVEN INTERVENTIONS • Was there an effect? Online activity Online activityOnline activityOnline activity Online activity Risk rating Risk rating Change in risk rating No email sent Emailed but not opened Opened email
  13. 13. Stakeholder outcomes 13 PERSONALISED, DATA-DRIVEN INTERVENTIONS -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 Forum Login Changeinriskrating * Missed online quizzes & tutorial submissions No email sent Emailed but not opened Opened email
  14. 14. Back to the users 14 FURTHER ITERATIVE DEVELOPMENT
  15. 15. Empowering staff with data 15 MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP) Moodle Report Logins Forums Assessments Parameters Traffic lights Gradebook Moodle Engagement Analytics Plugin Action Attendance
  16. 16. Lessons learned and next steps 16 LOOKING BACK AND LOOKING FORWARD • Looking back • Talk to the users • Find champions • Staff have varying levels of error tolerance • LA is a political football • Looking forward • Further evaluating impact • Production and wider trial • Source code release • New Moodle LA spec https://docs.moodle.org/dev/Learning_Analytics_Specification
  17. 17. The University of Sydney Page 17 1. Macquarie: Empowering staff with actionable LMS data 2. Sydney: Learning analytics by stealth 3. Macquarie: Building institutional readiness through openness
  18. 18. The University of Sydney Page 18 Learning analytics by stealth The Student Relationship Engagement System (SRES) Standing out from a Crowd SumAll CC BY-NC-ND 2.0 https://flic.kr/p/kYbv4C
  19. 19. The University of Sydney Page 19 The contexts of learning analytics Common barriers to adoption – Policy and ethical challenges – Culture of resistance to change – Vendor solutions – Data accuracy – One-size-fits-all Pressing institutional needs – ~$7 million/year lost to attrition – Larger class sizes – More disconnected students – Feedback very generalised
  20. 20. The University of Sydney Page 20 The Student Relationship Engagement System Attendance Interim grades LMS metrics Third party tools Other data as needed Student Relationship Engagement System
  21. 21. The University of Sydney Page 21 Personalising connections with students – Empowering staff – Flexible & intuitive – Targeted and personalised – Multi-channel – Benefits – Highly customisable – Efficient – key data in one place, operating at scale – Connect staff and all students (not just at-risk) Student Relationship Engagement System
  22. 22. The University of Sydney Page 22 Stakeholder outcomes Discontinued Failed Passed F P C D HD 1st year arts unit ~500 enrolment 1st year science unit ~1000 enrolment “Many thanks Adam. Yes, things are going much better this semester. I really appreciate how you keep in contact and keep an eye on us. It's such a big class, I don't know how you do it.” “Just to let you know that your emails really helped me survive last semester. I never realised how big a change it would be from school.”
  23. 23. The University of Sydney Page 23 Co-evolution of the SRES – Organic adoption by academics – Co-evolution of capabilities 0 5000 10000 15000 20000 25000 0 10 20 30 40 50 60 70 2012 2013 2014 2015 Numberofstudents Numberofunitsorschools Number of units Number of schools Number of students Pilot EWS introduced EWS integrated New analyses More data types Data import
  24. 24. The University of Sydney Page 24 Learning analytics by stealth? – Co-evolving capabilities and competencies around data-driven pedagogy and curriculum Student Relationship Engagement System
  25. 25. The University of Sydney Page 25 Learning analytics by stealth? – Co-evolving capabilities and competencies around data-driven pedagogy and curriculum Student Relationship Engagement System
  26. 26. The University of Sydney Page 26 The contexts of learning analytics Common barriers to adoption – Policy and ethical challenges – Culture of resistance to change – Vendor solutions – Data accuracy – One-size-fits-all Pressing institutional needs – ~$7 million/year lost to attrition – Larger class sizes – More disconnected students – Feedback very generalised
  27. 27. The University of Sydney Page 27 Lessons learned and next steps – Looking back – It was ugly but it worked – Ease of use is important – does it save time? – Attendance was a (surprisingly) popular metric – Everyone uses a customisable system differently – Personalisation at scale – Looking forward – Facilitate wider roll-out – Further (re)developments – ML, student view, sub-messages, etc – Research & evaluation of impact on students and staff
  28. 28. The University of Sydney Page 28 1. Macquarie: Empowering staff with actionable LMS data 2. Sydney: Learning analytics by stealth 3. Macquarie: Building institutional readiness through openness
  29. 29. Building institutional readiness through openness THE MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT) February 2016
  30. 30. Our approach 30 CONNECTING USERS WITH DATA THROUGH ANALYTICS Macquarie Open Analytics Toolkit Data Users Analytics LMS Video Classrooms Mobile Business systems External courses Understand students Identify and predict Code of practice “LAMP Lighters”
  31. 31. Bringing data together 31 NUANCES OF LEARNING DATA LMS Video Classrooms Mobile Business systems External courses Learning Record Store (LRS) Custom analytics engine
  32. 32. Empowering staff with data 32 MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)
  33. 33. Empowering staff with data 33 MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)
  34. 34. Lessons learned and next steps 34 LOOKING BACK AND LOOKING FORWARD • Looking back • Multi-disciplinary, multi-level teams • System architecture choice is important • Students are very open about data (unless it’s identifiable) • Staff and students have a (surprisingly) good idea of what they want • Looking forward • LRS to production • Piloting ‘dashboards’* with staff and students • Working with xAPI community • Open sourcing analytics tools
  35. 35. The University of Sydney Page 35 Lessons learned and issues raised
  36. 36. The University of Sydney Page 36 Lessons learned and issues raised – Give them what they want vs. build it and they will come – Customisability is key – Utility (eventually) trumps aesthetics (to an extent) – But people still like shiny things – Data are not enough – connect with pedagogical, pastoral – Surprisingly little kickback about privacy & ethics – Tension between research ethics & general ethics – It’s tricky to measure impact – Iterate – capabilities, implementation – Focus on the human
  37. 37. The University of Sydney Page 37 Adoption pipeline Colvin et al. (2015) Student retention and learning analytics: A snapshot of Australian practices and a framework for advancement. Office of Learning and Teaching, Sydney. First, implementers require an analytic tool or combination of tools that manage data inputs and generate outputs in the form of actionable feedback… As these increasingly meet the real needs of learners and educators the organisational uptake is accelerated.
  38. 38. The University of Sydney Page 38 Learning analytics is not an elixir for ineffective teaching, nor does it reveal an ideal pedagogy; instead, it provides data-driven tools or suggestions to help instructors make changes that can be measured in terms of student outcomes. Pistilli, M. D., Willis III, J. E., & Campbell, J. P. (2014). Analytics Through an Institutional Lens: Definition, Theory, Design, and Impact. In Learning Analytics (pp. 79-102). Springer New York.
  39. 39. The University of Sydney Page 39 1. MEAP Empowering staff with actionable LMS data Chris Froissard, Deborah Richards, Amara Atif 2. MOAT Building institutional readiness through openness James Hamilton, Ed Moore, Yvonne Breyer et al. 3. SRES Learning analytics by stealth Charlotte Taylor, Adam Bridgeman, Kathryn Bartimote-Aufflick et al. @dannydotliu danny.liu@mq.edu.au danny.liu@sydney.edu.au

×