Speakers: David Lewis, senior analytics consultant, Jisc Martin Lynch, learning systems manager, University of South Wales An opportunity to find out about how an institution has been implementing learning analytics to support the student journey with and opportunity to discuss issues and possibilities that the use of learning analytics may create.

1. Learning Analytics Implementation at USW - 2017/18 activity
Playing the hand you’re dealt
Learning Analytics Implementation at USW 2017/18
Martin Lynch
Learning Systems Manager, CELT

2. Learning Analytics Implementation at USW
• What is happening regarding Learning Analytics implementation at USW;
• The cards in our deck
• Timeline of events
• What are we doing in the project – 4 use-cases
• Ethics and GDPR
• Challenges and successes

3. Learning Analytics Implementation at USW
The cards in the deck
• JISC
• Tribal
• Blackboard
• GDPR
• PVC
• Strategic alignment
• Professional Services – Library, Registry,
IT Services Development
• Governance
• Learning Technologist
• Developer
• Project manager
• PACs
• Students
STUDENTS
STUDENTS
External
InternalStakeholders

4. Learning Analytics Implementation at USW
June 2015 PVC Student Retention and Success Dr Ben Calvert promotes internal conversation
regarding Student Retention and Success – USW now looking for an analytics solution
USW put forward
for the last place on
the JISC discovery
phase - consultation
stage begins with Bb
Bb consulting run
Discovery phase at
USW in order to
produce a “Readiness
Assessment”
Memorandum of
understanding signed
between USW and JISC
Readiness
assessment
report published
University agrees to set up
Learning Analytics Project – 2
year duration 2 posts identified,
with 0.6FTE project management
Project board
initiated at USW
Recruitment process
setting up for learning
technologist and developer
Begin extraction
of historical data
from Blackboard
Assessment data
coming from Bb
Progression data from Student
Record system now in UDD
Appointment of
learning technologist
Student enrolment
form amended in
light of LA project
Module marks for
current session held
in Student Record
System showing in
Data Explorer
Decision taken that Data
explorer does not meet
minimum requirements
for use as a pilot tool –
data and usability issues
Senior faculty staff and data users
meet with Tribal to consider
predictive modelling requirements
– select 3 key models
Staff questionnaire
sent out to baseline
expectation on
functionality of Data
explorer 50 staff agree
to pilot use of Data
Explorer from Jan
Appointment of
IT Developer
RUHere
attendance data
surfacing in DX
Tribal develop
3 new models
for USW –
data still has
gaps
USW invited to
join the ‘group
of five’
Workshop conducted for
student-facing dashboards
for 2019
All marks and
grades from Bb
and Quercus
appearing in DX
Library ‘recipe’
simplified by JISC
– test data
extract can begin
Historical grade data
from Student Record
System and
Blackboard (3 years)
now in warehouse –
Tribal can review
their models
Project timeline May 2015 - May 2018
PROJECT SET-UP = 18 MONTHS
PROCESS AND PRODUCT DEVELOPMENT = CONTINUOUS
DATA = 14 MONTHS

5. Learning Analytics Implementation at USW
What we are doing in the project?
Use-case#1
Data Explorer for PAC’s

6. Learning Analytics Implementation at USW
• Personal Academic Coaches (PAC’s) 300+ at USW
• Every student allocated a PAC on enrolment
• Senior teaching staff from course team – understand delivery
• Follows student through their studies – 3X20 min meetings every year, focus on
academic progress
• Need to document the event and have data to inform conversation
• Data Explorer replaces accessing multiple data sources but more than the sum of
the parts

7. Learning Analytics Implementation at USW

8. Learning Analytics Implementation at USW

9. Learning Analytics Implementation at USW

10. Learning Analytics Implementation at USW

11. Learning Analytics Implementation at USW

12. Learning Analytics Implementation at USW

13. Learning Analytics Implementation at USW

14. Learning Analytics Implementation at USW

15. Learning Analytics Implementation at USW

16. Learning Analytics Implementation at USW

17. Learning Analytics Implementation at USW
Use-case#2
Data Explorer for Curriculum Design

18. Learning Analytics Implementation at USW

19. Learning Analytics Implementation at USW

20. Learning Analytics Implementation at USW

21. Learning Analytics Implementation at USW

22. Learning Analytics Implementation at USW

23. Learning Analytics Implementation at USW

24. Learning Analytics Implementation at USW
Use-case#3 – the value of “Predictive
Analytics” to the institution

25. Learning Analytics Implementation at USW
• Predictions are made with very large data set (3 years)
• Machine learning algorithm so refining factors (regression analysis) in each
context (module or course) to establish significance
• Applying pattern recognition from training set to current students to find close
matches
• Predictive models are questions you want to ask of your data
• Tribal our current data science partner and models are almost there…

26. Learning Analytics Implementation at USW

27. Learning Analytics Implementation at USW
• First models are retention-based and deal with risk
• Intention is to use intelligence to better target support
But..
• Are these models any good?
• What services to build - how should we use this data?
• What value is this to the institution?

28. Learning Analytics Implementation at USW
Use-case#4 – student-facing dashboards

29. Learning Analytics Implementation at USW
• Unexplored in the sector so breaking new ground in the UK
• Small-scale studies – ref. Dr Liz Bennett University of Huddersfield, Ed de Quincey
Keele University
• Understanding this as a type of feedback literacy – ref. Paul Sutton (2012) notions
of feedback literacy
Principles are;
• Access should be mediated in the first instance
• Motivation is personal so dashboards should be customisable to fit user needs
• Dispositional dimension is critical

30. Learning Analytics Implementation at USW

31. Learning Analytics Implementation at USW

32. Learning Analytics Implementation at USW

33. Learning Analytics Implementation at USW
Ethics and GDPR
• USW implementation of Learning Analytics GDPR compliant
• Enrolment process amended
• Consent on protected characteristics
• Clearly described data processing activity
• Student-facing guide
• Student involvement in project
• Guided by ICO guidance and JISC
• Using ‘legitimate business interest’ to process data

34. Learning Analytics Implementation at USW
Successes
• Good resourcing has driven success
• Good appointments and effective project management
• Effective governance – interested (but not too involved) executive sponsor and a good board that turn up
and ask questions
• Staff see the value already but important to wait until its ready before showing it
Challenges
• Supplier pilot processes – delayed deployment for a year
• Some developments and data wrangling too slow (predictor, library recipe)
• Developers are hard to find – took us 10 months
• Unclear internal business processes – who looks after wobbling students?
• Inconsistent use of data for decision making
Project successes and challenges

35. Learning Analytics Implementation at USW
• DX – full roll-out and use for Personal Academic Coaches (PAC’s)
• Use of DX by other roles – faculty management, course and module teams to assess engagement and to
contribute to course design, side-cases (eg to evidence non-payment of fees)
• Study Goal – pilot use as a simple attendance monitoring tool
• Use of predictive data to drive early-warning alerts (Tribal)
• Assess value to institution of predictive business intelligence (Tribal)
• Compare and contrast Tribal and JISC predictive offers
• Student-facing dashboards – prototype mash-ups – refactor to something more useful
• Attract research and development funding to incubate this
• Full project evaluation and analysis of impact>options appraisal >recommendations
Project Vision and plans for 2018/19

36. Learning Analytics Implementation at USW
Questions?
martin.lynch@southwales.ac.uk