Learning Analytics @ The Open University
•Download as PPTX, PDF•
4 likes•742 views
The document discusses learning analytics at The Open University. It describes how the university uses predictive analytics and indicators to identify at-risk students early and trigger interventions. A key focus is developing early alert indicators using statistical modeling of student data to predict assignment submissions and module completion probabilities. The university also implemented a policy on ethical use of student data, developed an evaluation framework for analytics-driven actions, and examined the impact of learning design on student outcomes and satisfaction. The overall aim is to retain more students and help them achieve their study goals through strategic and targeted use of analytics.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Learning Analytics @ The Open University
Similar to Learning Analytics @ The Open University (20)
Recently uploaded
Recently uploaded (20)
Learning Analytics @ The Open University
- 1. Learning Analytics @ The Open University JISC Networking Event 11th May 2016 Kevin Mayles, Head of Analytics, The Open University
- 2. kevin.mayles@open.ac.uk | @kevinmayles Where are you from? Learning Analytics @ The Open University ● PVC Learning & Teaching ● CIO / IT ● Planning Office ● Student Support ● Faculty Learning and Teaching Centre Institute of Educational Technology Faculties and Schools Learning and Teaching Solutions Academic Professional Services Information Technology Strategy and Information Office Academic Services Marketing Student Registration and Fees Business Performance Improvement Library Services
- 3. kevin.mayles@open.ac.uk | @kevinmayles Where are you from? Learning Analytics @ The Open University ● PVC Learning & Teaching ● CIO / IT ● Planning Office ● Student Support ● Faculty Learning and Teaching Centre Institute of Educational Technology Faculties and Schools Learning and Teaching Solutions Academic Professional Services Information Technology Strategy and Information Office Academic Services Marketing Student Registration and Fees Business Performance Improvement Library Services ©TransportforLondon
- 4. kevin.mayles@open.ac.uk | @kevinmayles OU Context 2014/15 174k students The average age of our new undergraduate students is 29 40% new undergraduates have 1 A-Level or lower on entry Over 21,000 OU students have disabilities 868k assessments submitted, 395k phone calls and 176k emails received from students
- 5. kevin.mayles@open.ac.uk | @kevinmayles p.5 A clear vision statement was developed to galvanise effort across the institution on the focused use of analytics Analytics for student success vision Vision To use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals Mission This needs to be achieved at : ● a macro level to aggregate information about the student learning experience at an institutional level to inform strategic priorities that will improve student retention and progression ● a micro level to use analytics to drive short, medium and long-term interventions
- 6. kevin.mayles@open.ac.uk | @kevinmayles Vision in action
- 7. kevin.mayles@open.ac.uk | @kevinmayles The OU recognises that three equally important strengths are required for the effective deployment of analytics Analytics enhancement strategy Adapted from Barton and Court (2012)
- 8. kevin.mayles@open.ac.uk | @kevinmayles Analytics enhancement strategy Early alert indicators using predictive analytics Policy on the ethical use of student data for learning analytics Analytics for action evaluation framework Impact of learning design on outcomes
- 9. kevin.mayles@open.ac.uk | @kevinmayles Analytics enhancement strategy Early alert indicators using predictive analytics Policy on the ethical use of student data for learning analytics Analytics for action evaluation framework Impact of learning design on outcomes
- 10. kevin.mayles@open.ac.uk | @kevinmayles 10 Development of early alert indicators Application of a student number forecasting model to trigger interventions with vulnerable students Calvert (2014)
- 11. kevin.mayles@open.ac.uk | @kevinmayles 11 Open University: data + analysis Statistical modelling 2015 cohort ‘Training’ dataset Predictions for 2016 cohort Output dataset Factors Factors Logistic regression
- 12. kevin.mayles@open.ac.uk | @kevinmayles 12 Development of early alert indicators The 30 variables identified associated with success vary in their importance at each milestone Student (Demographic) Student – previous study/motivation Student progress in previous OU study Student – module Qualification / module of study Calvert (2014)
- 13. kevin.mayles@open.ac.uk | @kevinmayles 13 Current indicators Module probabilities Integrated into the Student Support Intervention Tool Predicts the probability of a student completing and passing the module
- 14. kevin.mayles@open.ac.uk | @kevinmayles 14 OU Analyse FailPass No submit Time(weeks) student engagement with learning activities
- 15. kevin.mayles@open.ac.uk | @kevinmayles 15 OU Analyse Module fingerprint time Assessment 1
- 16. kevin.mayles@open.ac.uk | @kevinmayles 16 Current indicators OU Analyse Predicts the submission of next assignment weekly Deployed through OU Analyse Dashboard
- 17. kevin.mayles@open.ac.uk | @kevinmayles 17 Outcomes of current pilots Summary of the interim evaluation of piloting as at March 2016 ● There is a mixed picture in the quantitative analysis on the impact in the pilot tutor groups on withdrawal rates and assignment submissions (note that tutors are self selected and the expectations to intervene are not consistent across the module piloting) ● It is a useful tool for understanding students and their participation ● Predictions generally agree with tutors' experience and intuitions of which students might potentially be at risk ● A (potential) USP of OU Analyse was the information provided between the assignment submission in relation to students' engagement with learning materials ● Overall, all tutors interviewed were positive about the affordances of OUA, and are keen to use it again (for a range of reasons) in their next module
- 18. kevin.mayles@open.ac.uk | @kevinmayles 18 Case studies and vignettes “I love it it’s brilliant. It brings together things I already do […] it’s an easy way to find information without researching around such as in the forums and look for students to see what they do when I have no contact with them […] if they do not answer emails or phones there is not much I can do. OUA tells me whether they are engaged and gives me an early indicator rather than waiting for the day they submit”
- 19. kevin.mayles@open.ac.uk | @kevinmayles Analytics enhancement strategy Early alert indicators using predictive analytics Policy on the ethical use of student data for learning analytics Analytics for action evaluation framework Impact of learning design on outcomes
- 21. kevin.mayles@open.ac.uk | @kevinmayles Information for students 21
- 22. kevin.mayles@open.ac.uk | @kevinmayles Analytics enhancement strategy Early alert indicators using predictive analytics Policy on the ethical use of student data for learning analytics Analytics for action evaluation framework Impact of learning design on outcomes
- 23. kevin.mayles@open.ac.uk | @kevinmayles Scaffolding action Analytics for Action Evaluation Framework and Toolkit 23
- 24. kevin.mayles@open.ac.uk | @kevinmayles Real-time progression reports 24
- 25. 25
- 26. 26
- 27. 27
- 28. kevin.mayles@open.ac.uk | @kevinmayles Supporting module teams ● Module teams work with support to identify actions that can be taken for current and future presentations ● The Analytics project have developed a ‘costed’ menu of response actions that can be taken ‘in-presentation’ or during the next presentation ● Budgetary considerations ● Resource considerations Removing ‘Blockers’ 28 LTS enabled Module Team SST enabled AL enabled Library enabledSupporting evaluation methods
- 29. kevin.mayles@open.ac.uk | @kevinmayles Analytics enhancement strategy Early alert indicators using predictive analytics Policy on the ethical use of student data for learning analytics Analytics for action evaluation framework Impact of learning design on outcomes
- 30. kevin.mayles@open.ac.uk | @kevinmayles 30 Learning design link to success
- 31. kevin.mayles@open.ac.uk | @kevinmayles 31 Learning design link to success
- 32. kevin.mayles@open.ac.uk | @kevinmayles Constructivist Learning Design Assessment Learning Design Balanced- variety Learning Design Socio-construct. Learning Design Learning Design 150+ modules Rienties, B. and Toetenel, L. (2016)
- 33. kevin.mayles@open.ac.uk | @kevinmayles Constructivist Learning Design Assessment Learning Design Balanced- variety Learning Design Socio-construct. Learning Design Student Satisfaction Student retention Learning Design 150+ modules VLE Engagement Week 1 Week 2 Week 30+ Rienties, B. and Toetenel, L. (2016) Communication
- 34. kevin.mayles@open.ac.uk | @kevinmayles Analytics enhancement strategy Early alert indicators using predictive analytics Policy on the ethical use of student data for learning analytics Analytics for action evaluation framework Impact of learning design on outcomes
- 35. kevin.mayles@open.ac.uk | @kevinmayles Are there any questions? For further details please contact: ● Kevin Mayles – kevin.mayles@open.ac.uk ● @kevinmayles ● Slideshare: http://www.slideshare.net/KevinMayles ● OU Analyse: https://analyse.kmi.open.ac.uk/ References: BARTON, D. and COURT, D., 2012. Making Advanced Analytics Work For You. Harvard business review, 90(10), pp. 78-83. CALVERT, C.E., 2014. Developing a model and applications for probabilities of student success: a case study of predictive analytics. Open Learning: The Journal of Open, Distance and e-Learning. KUZILEK, J., HLOSTA, M., HERRMANNOVA, D., ZDRAHAL, Z. and WOLFF, A., 2015. OU Analyse: Analysing At-Risk Students at The Open University. Learning Analytics Review, no. LAK15-1, March 2015, ISSN: 2057-7494 RIENTIES, B. and TOETENEL, L., 2016. The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 pp. 333– 341. SCHÖN, D.A., 1987. Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco, CA, US: Jossey-Bass.
Editor's Notes
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.
- Belinda Analytics is at the heart of the university’s strategic priority to deliver an outstanding student experience. We’ve developed this vision that drives our development of the use of analytics for both short term action and long term strategic decision making.
- Belinda Our strategy is based around the 3 key underpinning strengths we need to develop as an institution. Each equally important.
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.
- Needing to improve our longer-term forecasting of student numbers we are now able to predict a students chances of success by mapping their situation/characteristics to 30 of the most signification factors associated with student success: These include student characteristics or factors such as previous educational attainment occupational status previous study or reasons for study the number of credits already achieved the total number of previous passes or withdrawals the number of credits being studied in the year latest assignment scores what modules or qualification they are studying. We then map this data to historical records. Essentially we model one cohort of students based on the activates and outcomes of previous cohorts. This enables us to correlate those student characteristics to likely student outcome. This aggregated data is very accurate – providing us with a 2% tolerance. This data can be used to predict student progression and therefore income. As well as being applied to a cohort these predictions can also be used to trigger interventions with individual students who may have a low probability of success.
- Mention Zdenek Zdrahal and how session… In the second approach we have developed a tool called OU Analyse which enables us to undertake a week by week analysis of student engagement including study engagement with their learning activities. From this data we can analyse previous student cohorts and correlate learning activities against outcome. This enables us to identify, on a module by module basis, which activities carry the greatest value in terms of student success. To note: the predictive model assesses a student using 4 different algorithms, each of the algorithms takes into account different factors and the historic data we hold about previously successful students. The model will then give a student a vote for each of the algorithms that shows the student is at risk, therefore the more votes you have the higher the likelihood you are at risk. On this slide, the diagram represents one of the algorithms that takes into account the learning activities, which, are represented by the circles. Blue circles are activities that combine in different ways, actions such as forum use, interaction or downloading of resources, or online content. The pink circle shows a week of non-engagement with the VLE. Here we see a stylised pattern of weekly activity of a student who successfully achieved a pass for this module. Contrast this with the activity of a student who didn’t submit. You can see that the pattern is very different. In this example, the student who didn’t submit spend periods not engaging with the VLE. By modelling the pattern of behaviour of students who pass, fail or don’t submit, we are able to get a predict what successful and not successful VLE engagment looks like, and get a ‘finger-print’ for each module. Armed with this knowledge we are then able to monitor activity within the current cohort and quickly identify individual students who are not engaging with those high value activities. These students are flagged as ‘at risk’ and can be offered targeted interventions. Finally, we have also applied analytics to the exploration of the impact of learning design on learning performance. Takes demographic data + presentation-related data (aggregated VLE data available weekly) For each module, identifying important VLE actions, e.g., access forum, access or download a resource, access OU content. Learning activities will include a combination of these actions Identify students at risk of failing the module as early as possible so that OU intervention is efficient and meaningful. Finally, we have also applied analytics to the exploration of the impact of learning design on learning performance. Takes demographic data + presentation-related data (aggregated VLE data available weekly) For each module, identifying important VLE actions, e.g., access forum, access or download a resource, access OU content. Learning activities will include a combination of these actions Identify students at risk of failing the module as early as possible so that OU intervention is efficient and meaningful.
- Based on demographic data Based on VLE activities
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.
- Cluster analysis of 40 modules (>19k students) indicate that module teams design four different types of modules: constructivist, assessment driven, balanced, or socio-constructivist. The LAK paper by Rienties and colleagues indicates that learning design and learning design activities in particular strongly influence how students are engaging in our LMS. VLE engagement is higher in modules with socio-constructivist or balanced variety learning designs, and lower for constructivist designs. In terms of learning outcomes, learning design seems to have an impact on learning performance. Students rate constructivist modules higher, and socio-constructivist modules lower. However, in terms of student retention (% of students passed) constructivist modules have lower retention, while socio-constructivist have higher. In particular, modules with a heavy reliance on content and cognition (assimilative activities) seemed to lead to lower completion and pass rates. Thus, learning design strongly influences behaviour, experience and performance. (and we believe we are the first to have mapped this with such a large cohort). Red lines represent –ve effect Green lines represent +ve effect Blue lines represent no significant effect. Eventually with the availability of yet more data, we may soon be better able to understand the complex relationship between learning design and learning processes and outcomes with resultant sharing of best practice evidenced by such analyses. If this could then be mapped to student characteristics, there is the potential to personalise learning designs. (Learning performance was calculated by the number of learners who completed and passed the module relative to the number of learners who registered for each module.)
- Cluster analysis of 40 modules (>19k students) indicate that module teams design four different types of modules: constructivist, assessment driven, balanced, or socio-constructivist. The LAK paper by Rienties and colleagues indicates that learning design and learning design activities in particular strongly influence how students are engaging in our LMS. VLE engagement is higher in modules with socio-constructivist or balanced variety learning designs, and lower for constructivist designs. In terms of learning outcomes, learning design seems to have an impact on learning performance. Students rate constructivist modules higher, and socio-constructivist modules lower. However, in terms of student retention (% of students passed) constructivist modules have lower retention, while socio-constructivist have higher. In particular, modules with a heavy reliance on content and cognition (assimilative activities) seemed to lead to lower completion and pass rates. Thus, learning design strongly influences behaviour, experience and performance. (and we believe we are the first to have mapped this with such a large cohort). Red lines represent –ve effect Green lines represent +ve effect Blue lines represent no significant effect. Eventually with the availability of yet more data, we may soon be better able to understand the complex relationship between learning design and learning processes and outcomes with resultant sharing of best practice evidenced by such analyses. If this could then be mapped to student characteristics, there is the potential to personalise learning designs. (Learning performance was calculated by the number of learners who completed and passed the module relative to the number of learners who registered for each module.)
- Belinda This encapsulates our strategy which is moving forward on all fronts. Kevin will now demonstrate an operation tool available at scale and one of our latest experimental prototypes.