The power of learning analytics to unpack learning and teaching: a critical perspective Ludwig-Maximilians-Universität München Fakultät für Psychologie und Pädagogik

1. @DrBartRienties
Professor of Learning Analytics
The power of learning analytics
to unpack learning and teaching:
a critical perspective
30th of October 2019
Ludwig-Maximilians-Universität
München
Fakultät für Psychologie und
Pädagogik

2. First an apology
Adeniji, B. (2019). A Bibliometric Study on Learning Analytics. Long Island University. Retrieved from https://digitalcommons.liu.edu/post_fultext_dis/16/
There is too much
exciting stuff happening
at the OU!!!

3. Agenda
1. What is learning analytics?
2. Exemplar 1: How is the OU implementing LA on a large scale?
3. Exemplar 2: Can we track how good learners are in search-
skills?
4. Exemplar 3: Can we predict what is a good learning design?
5. What are the main affordances and limitations of Learning
Analytics in terms of data?

4. (Social) Learning Analytics
“LA is the measurement, collection, analysis and reporting of data about learners
and their contexts, for purposes of understanding and optimising learning and the
environments in which it occurs” (LAK 2011)
Social LA “focuses on how learners build knowledge together in their cultural
and social settings” (Ferguson & Buckingham Shum, 2012)

5. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.

6. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.

7. It’s everywhere
7

8. Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, 1-16.

9. Prof Paul Kirschner (OU NL)
“Learning analytics: Utopia or dystopia”, LAK 2016 conference

10. 1. Increased availability of learning data
2. Increased availability of learner data
3. Increased ubiquitous presence of technology
4. Formal and informal learning increasingly blurred
5. Increased interest of non-educationalists to understand
learning (Educational Data Mining, 4profit companies)
6. Personalisation and flexibility as standard

11. Exemplar 1: How is the OU implementing LA on a large scale:
Leading global distance learning, delivering high-quality education to anyone, anywhere, anytime
The Open University
Largest
University
in Europe
No formal
entry
requirements
enter with one
A-level or less
33%
38%
of part-time
undergraduates
taught by OU in UK
173,927 formal
students
55%
of students are
'disadvantaged'
FTSE 100 have
sponsored staff on OU
courses in 2017/8
60%
66%
of new
undergraduates
are 25+ 1,300
Open University students
has a disability (23,630)
1 in 8
Students are
already in work
3 in 4
employers use
OU learning
solutions to
develop
workforce

12. 13,206
learners
6,000
learners
123,000
learners
• A pioneering Moodle-based award winning site providing free access to learning – 1,000+ courses, plus articles and videos.
• The OU publishes curriculum as short courses onto OpenLearn and creates free Badged Open Courses (BOCs) which
reward informal learners with a badge / certificate of achievement.
• Links from BBC broadcasting with themed pages
developed to support the content of many series. In addition,
print resources can be ordered such as posters.
7.8m
new learners
each year
60m
since launch
in 2006
Opening Access: OpenLearn

13. Innovative and engaging content for
broadcast audiences and rich resources for
students. 2016/17: the OU co-produced 35
series:
• Generated 250 million viewing and
listening events across channels and
platforms.
• Directed 1.2 million viewers to
OpenLearn.
Dedicated channel on YouTube with bite-
sized learning. It is the largest educational
presence on YouTube in EU with:
• Over 1,700 public videos with 50 million
views.
• Over 166,000 subscribers to our channel,
more than any other UK educational
institution
• Reaching over 3 million learners per year.
The OU now engages with learners
on Facebook:
• Viewed 6.2 million times by 2.5
million users.
• We run Facebook live sessions to
engage around topical issues.
• We will be doing more with
Facebook on the DfE Flexible
Learning Fund in 2019.
Reaching out to millions more
250m
views
2016/17
3m
learners
per year
2.5m
in
2017

14. • 2m transactions a day
• 6m quiz questions answered a year
• 4500 tutors
• Groups of 15-20
• 8-16 hours a week
• Home, work, trains, prisons, submarines
• 130 degrees and other qualifications
• From 450+ modules
Study Experience Overview

15. STUDENT SUCCESS ANALYTICS
15
O R G AN I S AT I O N AL C APAB I LT I E S
Productionised
output and MI
Strategic
analysis
Modelling /
AI
Data
collection
Data storage
and access
Technology
architecture
Learning
design and
delivery
Student
lifecycle
managemen
t
Continuous
improvemen
t and
innovation
Creation of
actionable
insight
Availability
of data
Impact the
student
experience
Adapted from Barton and Court (2012) - https://hbr.org/2012/10/making-advanced-analytics-work-for-you

16. HOW ANALYTICS SUPPORTS STUDENT SUCCESS
STUDENT SUCCESS
ANALYTICS KPIs
STUDENT SUCCESS
PRIORITIES DATA & ANALYTICS WILL SUPPORT, ALIGN TO, AND PROACTIVELY DRIVE THE RECRUITMENT
AND STUDENT SUCCESS STRATEGY THROUGH CONTINUOUS INNOVATION
CONVERSION
STUDY
CONTINUATION
EMPLOYABILITY
STUDY
ENGAGEMENT
WEBSITE APP ALVLECHATBO
T
PHONECOMMUNICATION
CHANNELS
DATA
ANALYTICS
CAPABILITY
CRM & DIGITAL (ANALYTICS) INTEGRATION LAYER
ENTERPRIS
E DATA
HUB
BI
ENABLEMENT
DATA SCIENCE
ENABLEMENT
ANALYTICS
INDUSTRIALISATION
DATA SECURITY, PRIVACY & GOVERNANCE
Build Your Own / Self-serveStrategic Analysis Canned Reporting
BUSINESS INTELLIGENCE
DIGITAL EXPERIENCE OFFLINE CHANNELS
Statutory Reporting
ANALYTICS & DATA SCIENCE
Modelling, AI &
Machine
Learning
Test & LearnKnowledge
Graph
Survey Analytics
Sentiment
Analysis
Trigger Based
Interventions
Digital Journey
Optimisation
Personalised
Prospect Targeting
Personalised
Recommendations
ANALYTICS CAPABILITY
16
SUPPORTED
OPEN ENTRY
IMPROVED
COMMUNICATIONS
FLEXIBLE STUDY
INTENSITY
INCREASED RETURN
RATES
INCREASED YR3
CONTINUATION RATES
INCREASED
SATISFACTION
IMPROVED
PERSONAL
OUTCOMES
INCREASED PASS
RATES
IMPROVED CAREER
OUTCOMES
D ATA & AN ALY T I C S V I S I O N

17. Using Predictive Learning Analytics
A quick history of OU Analyse
Pres. Scope Delivery
2014B 2 modules, selected course members Excel spreadsheet, sent by email,
manually generated
2014J 12 modules, selected course members Automated pred., excel spreadsheet
2015J Made available to tutors in selected modules Dashboard, within OU network
2016J ~1000 users (338 accessed) , 785 tutors (305 accessed) Dashboard on Internet, no VPN
needed, grade predictions
2017J,18
B
37 modules, 375 users (240 accessed), 323 tutors (204
accessed)
Mostly STEM pilots, user
acceptance
2018J 250 modules in OUA, ~3500 users, including mostly tutors,
module chairs, staff tutors, cluster managers, SSTs
Dashboard combining OUA
predictions and SIO Student
Probabilities

18. 18
What does it do?
It produces predictions as to
whether students are at risk
of failing their studies.
The model predicts on a
weekly basis whether or not
a given student will submit
their TMA.
It uses a traffic light system
to pinpoint in red students at
risk, in amber those with a
moderate probability of failing
and in green those who are
unlikely to fail.
OU ANALYSE
https://analyse.kmi.open.ac.uk/#dashboard

19. Herodotou, C., Hlosta, M., Boroowa, A., Rienties, B., Zdrahal, Z., Mangafa, C. (2019). Empowering online teachers through predictive learning analytics. British
Journal of Educational Technology. 50(6), 3064-3079. Impact factor: 3.142.

20. Herodotou, C., Hlosta, M., Boroowa, A., Rienties, B., Zdrahal, Z., Mangafa, C. (2019). Empowering online teachers through predictive learning analytics. British Journal of Educational Technology. 50(6), 3064-3079. Impact factor: 3.142.
“Teachers in the “average” OUA frequency group had
significantly better average student’s performance in course
presentations in which they used OUA than in presentations
without OUA. These findings may be explained by existing non-
PLA-related approaches and thus used to monitor students’
engagement with the course material. As shown in other studies
(Herodotou et al., 2017), OUA helped teachers in monitoring their
students’ behavior online in an effective manner; rather than
having to search for disperse student’s engagement information
in, for example, forums and the VLE, teachers had all relevant
information gathered together in a single space (OUA).”

21. 21
2017/18: Use of OUA

22. In summary
• Uptake of LA a long and non-linear
process
• Use of OUA by teachers significantly
influences students’ performance
• Large differences in practices (i.e.,
discipline, module, teacher
characteristics)

23. Exemplar 2: Are students well skilled in
searching the internet?
● Across the globe people are assumed to have good internet searching skills
● However, recently there is a debate whether this is actually the case?
● In particular, some have raised concerns about a widely used self-report instrument called Internet-Specific
Epistemic Questionnaire (ISEQ)?
● Are students well skilled in searching the internet?
● (How would you set up a design to test this?)
23

24. • Lab study whereby 269
students worked in dyads on
complex red yeast rice case
• We monitored which websites
they visited (and which they did
not)
• We analysed chat data and
final dyad answer to
government advice
Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D. T., Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the
internet. Computers in Human Behavior, 73, August 2017, 507–518

25. Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D. T., Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the
internet. Computers in Human Behavior, 73, August 2017, 507–518

26. ● No relation between ISEQ and what students actually do online 
Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D. T., Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the internet.
Computers in Human Behavior, 73, August 2017, 507–518

27. Exemplar 3: linking existing datasets
• Learning design data (>300 modules mapped)
• VLE data
• >140 modules aggregated individual data weekly
• >37 modules individual fine-grained data daily
• Student feedback data (>140)
• Academic Performance (>140)
• Predictive analytics data (>40)
• Data sets merged and cleaned
• 111,256 students undertook these modules

28. Learning Design is described as “a methodology for enabling
teachers/designers to make more informed decisions in how they go about
designing learning activities and interventions, which is pedagogically
informed and makes effective use of appropriate resources and
technologies” (Conole, 2012).

29. Assimilative Finding and
handling
information
Communication Productive Experiential Interactive/
Adaptive
Assessment
Type of activity Attending to
information
Searching for
and processing
information
Discussing
module related
content with at
least one other
person (student
or tutor)
Actively
constructing an
artefact
Applying
learning in a
real-world
setting
Applying
learning in a
simulated
setting
All forms of
assessment,
whether
continuous, end
of module, or
formative
(assessment for
learning)
Examples of
activity
Read, Watch,
Listen, Think
about, Access,
Observe,
Review, Study
List, Analyse,
Collate, Plot,
Find, Discover,
Access, Use,
Gather, Order,
Classify, Select,
Assess,
Manipulate
Communicate,
Debate, Discuss,
Argue, Share,
Report,
Collaborate,
Present,
Describe,
Question
Create, Build,
Make, Design,
Construct,
Contribute,
Complete,
Produce, Write,
Draw, Refine,
Compose,
Synthesise,
Remix
Practice, Apply,
Mimic,
Experience,
Explore,
Investigate,
Perform,
Engage
Explore,
Experiment,
Trial, Improve,
Model, Simulate
Write, Present,
Report,
Demonstrate,
Critique
Conole, G. (2012). Designing for Learning in an Open World. Dordrecht: Springer.
Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151
modules. Computers in Human Behavior, 60 (2016), 333-341
Open University Learning Design Initiative (OULDI)

30. Toetenel, L., Rienties, B. (2016). Analysing 157 Learning Designs using Learning Analytic approaches as a means to evaluate the impact of pedagogical
decision-making. British Journal of Educational Technology, 47(5), 981–992.

33. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student
engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
69% of what students are
doing in a week is
determined by us, teachers!

34. Constructivist
Learning Design
Assessment
Learning Design
Productive
Learning Design
Socio-construct.
Learning Design
VLE Engagement
Student
Satisfaction
Student
retention
150+ modules
Week 1 Week 2 Week30
+
Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151
modules. Computers in Human Behavior, 60 (2016), 333-341
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student
engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
Communication

35. So what happens when you give
learning design visualisations to
teachers?
Toetenel, L., Rienties, B. (2016) Learning Design – creative design to visualise learning activities. Open Learning: The Journal of Open and Distance Learning,
31(3), 233-244.

36. Toetenel, L., Rienties, B. (2016) Learning Design – creative design to visualise learning activities. Open Learning: The Journal of Open and Distance Learning,
31(3), 233-244.

37. Conclusions I
1. A lot of data is coming into (and out of)
education
2. A lot of “semi-standardised” data is
gathered within and across institutions
3. Great opportunities to harvest fine-
grained and longitudinal data

38. Conclusions II
1. What about the ethics?
2. What can be standardised (and what not)?
3. Are we optimising the record player?

39. The power of learning analytics to visualise
evidence of learning
T: drBartRienties
E: bart.rienties@open.ac.uk
W: www.bartrienties.nl
W: https://www.organdonation.nhs.uk/
W: https://www.sportentransplantatie.nl/