This document discusses the complexity challenges faced by the Open University in implementing an institutional strategy for learning analytics. It recognizes that three key strengths are required: data infrastructure and processes, data science capabilities, and integrating analytics into business processes. The OU is developing capabilities across 10 areas including predictive indicators, learning design, and implementation approaches. While complexity cannot be controlled, effective project management, agile methods, communication, revisiting benefits and change control can help address structural, socio-political and emergent complexities faced in strategic analytics projects.

1. Riding the tiger: dealing with complexity in
the implementation of institutional strategy
for learning analytics
Kevin Mayles, Head of Analytics, Open University

2. The Open University Mission

3. 3

4. Student profile
Nearly 30% of new OU
undergraduates are under 25
The average age of our new
undergraduate students is 30
Only 9% of our new students are
over 50
42% new undergraduates have 1
A-Level or lower on entry
Over 17,400 OU students have
disabilities
11,000 OU students are studying at
postgraduate level

5. p.5
A clear vision statement has been developed to galvanise effort across
the institution on the focused use of analytics
Analytics for student success vision
Vision
To use and apply information strategically (through specified indicators) to retain
students and progress them to complete their study goals
Mission
This needs to be achieved at :
● a macro level to aggregate information about the student learning experience at an
institutional level to inform strategic priorities that will improve student retention and
progression
● a micro level to use analytics to drive short, medium and long-term interventions

6. Vision in action

7. The OU recognises that three equally important strengths are required
for the effective deployment of analytics
Underpinning organisational strengths
Adapted from Barton and Court (2012)

8. The OU recognised three equally important strengths are required for
the effective deployment of analytics
Underpinning organisational strengths
We need to ensure we have
the right architecture and
processes for collecting the
right data and making them
accessible for analytics – we
need a ‘big data’ mind-set

9. The OU recognised three equally important strengths are required for
the effective deployment of analytics
Underpinning organisational strengths
The university needs world class
capability in data science to continually
mine the data and build rapid prototypes
of simple tools, and a clear pipeline for
the outputs to be mainstreamed into
operations

10. The OU recognises that three equally important strengths are required
for the effective deployment of analytics
Underpinning organisational strengths
Benefits will be realised through existing
business processes impacting on
students directly and through
enhancement of the student learning
experience – we will develop an
‘analytics mind-set’ in
these areas
For/in/on-action adapted from Schön (1987)

11. The OU is developing its
capabilities in 10 key areas that
build the underpinning strengths
required for the effective
deployment of analytics
Analytics enhancement strategy

12. 12
Development of predictive indicators
Application of a student number forecasting model to trigger
interventions with vulnerable students
Calvert (2014)

13. 13
Development of predictive indicators
The 30 variables identified associated with success vary in their
importance at each milestone
Student
(Demographic)
Student – previous
study/motivation
Student progress
in previous OU
study
Student – module
Qualification /
module of study
Calvert (2014)

14. 14
Application of predictive indicators

15. 15
Application of predictive indicators

16. 16
Application of predictive indicators
Technology
showcase
4.30pm
Thursday

17. 17
Learning design link to success
Rienties et al (2015)

18. 18
Learning design link to success
Rienties et al (2015)
Concurrent
session 6C
2.45pm
Thursday


19. 19
Implementation at the OU
© Transport for London

20. p.20
The complexity challenge
What is project complexity?
● “Complicated”: e.g. a Swiss watch
● “Complex”: from the Latin ‘complexus’ (braided together). Nonlinear and
unpredictable.
●Like quality – it is hard to quantify and is something that is experienced
● Language: an analogy – not based in complexity science / complex adaptive systems
theory
● Subjective not objective
● Complexity is art not science
Maylor et al (2013)

21. p.21
Complexities
● Structural complexity
●Number, size, financial scale, interdependencies, variety, pace, technology, breadth
of scope, number of specialties, multiple locations/time zones
● Socio-political complexity
●People, politics, stakeholder / sponsor commitment, resistance, shared
understanding, fit, hidden agendas, conflicting priorities, transparency
● Emergent complexity
●Technology and commercial maturity and novelty, clarity of vision / goals, clear
success criteria / benefits, previous experience, availability of information,
unidentified stakeholders
● Assessed through the ‘Complexity Assessment Tool’
Maylor et al (2013)

22. p.22
How complex is the OU Analytics project?
Structural
Socio-politicalEmergent
OU Analytics Project Complexity
H
M
L

23. p.23
Responding to complexities
Complexity 
Response

Structural Socio-political Emergent
Plan and control

Plan comms (inc. clear
visualisation); isolate key
tasks; create project
board of stakeholders
Co-location; use PMO as
point of control; scenario
planning; change control
Relational
Prioritise communication
with stakeholders; reach
out to others 
Socialise changes; revisit
assumptions; increase
formal communication
Flexibility
(Risk and change)
Anticipate refinement
and testing; change
control; parallel
developments
Manage expectations of
change; revisit benefits
regularly; ‘look-ahead’
with client

Maylor et al (2013)

24. p.24
Complexities faced at the OU
Structural Socio-political Emergent
Benefits - clarity
Unfamiliar technology
Supply chain not in place
Skills shortage
Integration of technical
disciplines
Dependencies
Pace
Experience of staff
Culture change needed
Impact of organisational
change
External stakeholder
alignment and
understanding
Benefits and success
measures will become
clear
Technology will become
familiar and change
Scope, schedule and
resource availability likely
to change
Stakeholder engagement
will improve

25. p.25
What have we done, what have we learned?
Structural Socio-political Emergent
Effective project
management controls in
place
Agile method – early
delivery and iterate
You can never do enough
communicating
Revisited benefits
regularly
Project board – wide
representation – including
the doubters
High profile amongst
senior leadership
Spend time on key (loud)
stakeholders
Direct control of resources
– small dedicated team
leading the way
Get small pilots going and
people come on board
Change control – use it!

26. p.26
You cannot control the complexity…
Thank you…

27. Are there any questions?
For further details please contact:
● Kevin Mayles – kevin.mayles@open.ac.uk
● @kevinmayles
References:
BARTON, D. and COURT, D., 2012. Making Advanced Analytics Work For You. Harvard business review, 90(10), pp.
78-83.
CALVERT, C.E., 2014. Developing a model and applications for probabilities of student success: a case study of
predictive analytics. Open Learning: The Journal of Open, Distance and e-Learning.
MAYLOR, H.R., TURNER, N.W. and MURRAY-WEBSTER, R., 2013. How Hard Can It Be? Research Technology
Management, 56(4), pp. 45-51.
RIENTIES, B., TOETENEL, L. and BRYAN, A., 2015. “Scaling up” learning design: impact of learning design activities
on LMS behaviour and performance. Proceedings of the 5th Learning Analytics and Knowledge Conference 2015.
SCHÖN, D.A., 1987. Educating the reflective practitioner: Toward a new design for teaching and learning in the
professions. San Francisco, CA, US: Jossey-Bass.