The Open University Library Data Project
Is there a statistically significant relationship
between library resource accesses and
student attainment at the Open University, a
distance learning institution?
Anne Gambles, Project Manager
The Northern Collaboration Conference 2016-09
Open University Facts and Figures
1.89 million people worldwide have studied with the OU
2
173,899
UK students
>21,000
people with
disabilities studied
with the OU in
2014/15.
76%
of OU students
work full or
part-time during
their studies
8353
Overseas
students
30%
of new OU
undergraduates
are under 25
21%
of OU UK
undergraduates
live in the 25%
most deprived
areas
3
85%
surveyed in the 2016
National Student
Survey agree library
resources and
services are good
enough for their
needs
1,295,001
million
downloads from Open
Research Online. The
Open University's open
access repository of
research publications
51k
Unique visitors to
the OU Library
website per
month
Award
winning
Digital and information
literacy tools for
students
1st
UK HE to provide
a 24/7 webchat
service
Resource
richness
600+ databases &
collections,
134k+ electronic
journals and over
200k ebooks
Open University Library
Facts and Figures
OutlineProject
Background LibraryDataStrategy
LibraryData
Project
overview
LibraryData
Progress
and findings
4
Early steps
Exploring the potential of library data 2010-
5
• RISE Jisc-funded project in 2010 investigating the
potential of using library activity data from EZproxy to
create an amazon-like recommender service for a
discovery system. Part of the Jisc Activity Data
programme
• Working with Jisc Library Analytics and Metrics Project
(LAMP) and the Jisc Learning Analytics library community
panel
• Keeping up-to-date with work on projects in other HEIs
exploring the relationship between library use and student
success
Background
Exploring the potential of library data 2013-2014
6
Needham, Gill; Parker, Jo; Nurse, Richard;
Scantlebury, Non and Dick, Sam (2013).
Can an excellent distance learning library service
support student retention and how can we find out?
Open Learning, 28(2) pp. 135–140.
http://dx.doi.org/10.1080/02680513.2013.847364
Article in Open Learning 2013
and
Poster at Design4Learning conference
2014
Background
Exploring the potential of library data 2014-2015
7
• Working with institutional Learning Analytics project
• Providing library data to module teams
• Menu of potential ‘data-driven’ interventions include library
activities
Why
University of Huddersfield 2013
‘There is a statistically significant relationship between
library resource use and level of degree result’
University of Minnesota 2014
‘Students who utilize academic libraries within their first year have
higher cumulative grade point averages and retention, on average,
than their peers who did not use library services.’
Library Analytics
“Using data about student engagement with library services and
content to help institutions and students understand and improve
library services to learners.”
Other UK HEIs report a correlation
between student success /
retention and library use – is this
the case at the OU?
8
9
Strategy
Project overview
Strategy
Evidence
impact
Integration Skills
Data on library use by
students not
integrated into
institutional data
spaces
No strong evidence base
of relationship between
library use and
retention/achievement at
OU
Library
Services lack
data analysis
skills
Challenges
Focus areas
10
11
How can library data benefit
students, faculties and the
University as a whole?
• Better understand student
success factors; it may also help
to improve the accuracy of
student success predictions
• Provide evidence and understand
the impact of students’
engagement with independent
study
• Inform improvements to library
services and trigger interventions
for disengaged students
Analysing
OU
Library
data
alongside
OU data
will help
to:
Intended outcomes
We can
demonstrate that
library
engagement
enhances student
achievement and
retention
Library engagement
data is available for
use across the OU
and routinely being
used where key
decisions are made
about learning
Library Staff are
confident with using
data to show how
library use contributes
to student success
and retention
Faculty can be
confident that library
engagement improves
student achievement
and retention
Faculty can access
real-time library data to
identify student
engagement
Faculty can be
confident that library
staff can use data to
support retention &
achievement
Contribute to improved
student success: increased
retention and completion
12
Progress and findings
13
Skills (library staff)
Readiness assessment
14
Strongly
Disagree
NeutralDisagree Agree
Strongly
Agree
Strongly
Disagree
NeutralDisagree Agree
Strongly
Agree
Strongly
Disagree
NeutralDisagree Agree
Strongly
Agree
Strongly
Disagree
NeutralDisagree Agree
Strongly
Agree
Strongly
Disagree
NeutralDisagree Agree
Strongly
Agree
I am confident that I know what
analytics data the library has and I
know how to access it
I am confident that I know what
analytics data the university has
and I know how to access it
I am confident in my ability to use
simple analytics tools
I am confident in my ability to
analyse data
I am confident in my ability to use
data to support decision making
Strongly
Disagree
NeutralDisagree Agree
Strongly
Agree
OVERALL
High
Feb
2016
SDH
Confidence levelLow
Overall confidence in
finding and using data is
low.
Library staff will need more
help to find and use library
and university data.
Once data tools are in
place then they will need
help to build skills in
analysing and using data.
Online library resource accesses
Data on online library resource accesses is taken from
EZproxy and OpenAthens systems
15
Online library resource accesses
16
• Starting point URL
only
• Data files include
students who are
studying more
than one module
• Data files include
data from students
who didn’t
complete their
module
7.23
13.22
23.25
33.44
0
5
10
15
20
25
30
35
40
Level 1 Level 2 Level 3 PG
AVERAGENUMBEROFACCESSESOFLIBRARY
RESOURCESPERSTUDENT(MEAN)
Online library resource access by level
(all faculties, 2014 presentations)
Mean per student
Online library accesses and student achievement
17
n = [CELLRANGE]
n = [CELLRANGE]
n = [CELLRANGE]
n =
[CELLRANGE]
0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0
Deferral
Fail
Pass
Distinction
Number of library resource accesses per student (mean)
Mean number of library resources used per student -
3 level 1 modules (Arts, Law, Technology)
2014-15 presentations
Library Data project 2015/16
n=11,501
n=11,406
Pilot study:
Pilot Study
Library use and Age
18
n = [CELLRANGE]
n = [CELLRANGE]
n = [CELLRANGE]
n = [CELLRANGE]
n = [CELLRANGE]
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
i Under 25 ii 26-35 iii 36-45 iv 46-55 v 56 and over
AVERAGENUMBEROFLIBRARYRESOURCES
ACCESSED
Age groups - average use of online library resources - 3 level 1 modules
(Arts, Law, Technology) 2014-15
Pilot Study
Library use and gender
19
n =
[CELLRANGE]
n =
[CELLRANGE]
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Female Male
AVERAGENUMBEROFLIBRARYRESOURCES
ACCESSED
Average use of online library resources by
gender - 3 level 1 modules (Arts, Law,
Technology) 2014-15
Pilot Study
Library use and level of educational qualification on entry
20
n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE]
n = [CELLRANGE]
n = [CELLRANGE]
n = [CELLRANGE]
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
1 No Formal
Qualifications
2 Less than A
Levels
3 A Levels or
equivalent
4 HE Qualification 5 PG Qualification 6 Not known
AVERAGENUMBEROFLIBRARYRESOURCESACCESSED
Average use of online library resources by previous education - 3
level 1 modules (Arts, Law, Technology) 2014-15
Library Data Project 2016
n=11,501
21
Online library resource accesses and
student achievement
Wider study:
n = [CELLRANGE]
n = [CELLRANGE]
n = [CELLRANGE]
0.00 5.00 10.00 15.00 20.00 25.00 30.00
Fail
Pass
Distinction
Mean library resource accesses per student - 2014 and 2015
presentations (all modules)
Mean resource accesses per student
n = 315,460
22
Online library resource accesses
and student achievement
Wider study:
23
Online library resource accesses
and student achievement
Wider study:
The Wollongong methodology
24
University of Wollongong Library
Cox, B. L. and Jantti, M. (2012) Capturing business intelligence
required for targeted marketing, demonstrating value, and driving
process improvement, Library & Information Science Research. 34,
PP308-316 doi:10.1016/j.lisr.2012.06.002
• Looked at whether there is a relationship between library resource
accesses and student success
• Aimed to demonstrate Library impact
• The approach and methodology was tested with OU data with some
small adjustments.
• UoW used time accessing resources (from the full ezproxy logfiles),
OU has used count of accesses ezproxy and openathens for 2014
students
• UoW used Weighted Average Marks – OU used Overall Continuous
Assessment Score (OCAS)
Replicating the Wollongong study
25
Replicating the Wollongong study
26
The average continuous
assessment score for students
who did not access OU Library
online resources is 64.
The average continuous
assessment score for students
that accessed one resource
during the year was 68.
There is a non-linear correlation
between average resource
accesses and average student
Score (R squared = 0.54)
Replicating the Wollongong study
27
Findings so far
● The percentage of students accessing library resources and the number of library
resources they access on average increases the higher the level of study
● There is quite a wide variance between different faculties, levels and modules in
access to library resources
● There seems to be a similar relationship between library access and student
success at the Open University, despite students’ mainly online library access
● Early indications are that the relationship is statistically significant
Key indications from our early studies are:
28
29
What’s next?
Questions?
Project Blog: www.open.ac.uk/blogs/LibraryData/
30

Is there a statistically significant relationship between library resource access and student attainment at The Open University, a distance learning institution

  • 1.
    The Open UniversityLibrary Data Project Is there a statistically significant relationship between library resource accesses and student attainment at the Open University, a distance learning institution? Anne Gambles, Project Manager The Northern Collaboration Conference 2016-09
  • 2.
    Open University Factsand Figures 1.89 million people worldwide have studied with the OU 2 173,899 UK students >21,000 people with disabilities studied with the OU in 2014/15. 76% of OU students work full or part-time during their studies 8353 Overseas students 30% of new OU undergraduates are under 25 21% of OU UK undergraduates live in the 25% most deprived areas
  • 3.
    3 85% surveyed in the2016 National Student Survey agree library resources and services are good enough for their needs 1,295,001 million downloads from Open Research Online. The Open University's open access repository of research publications 51k Unique visitors to the OU Library website per month Award winning Digital and information literacy tools for students 1st UK HE to provide a 24/7 webchat service Resource richness 600+ databases & collections, 134k+ electronic journals and over 200k ebooks Open University Library Facts and Figures
  • 4.
  • 5.
    Early steps Exploring thepotential of library data 2010- 5 • RISE Jisc-funded project in 2010 investigating the potential of using library activity data from EZproxy to create an amazon-like recommender service for a discovery system. Part of the Jisc Activity Data programme • Working with Jisc Library Analytics and Metrics Project (LAMP) and the Jisc Learning Analytics library community panel • Keeping up-to-date with work on projects in other HEIs exploring the relationship between library use and student success
  • 6.
    Background Exploring the potentialof library data 2013-2014 6 Needham, Gill; Parker, Jo; Nurse, Richard; Scantlebury, Non and Dick, Sam (2013). Can an excellent distance learning library service support student retention and how can we find out? Open Learning, 28(2) pp. 135–140. http://dx.doi.org/10.1080/02680513.2013.847364 Article in Open Learning 2013 and Poster at Design4Learning conference 2014
  • 7.
    Background Exploring the potentialof library data 2014-2015 7 • Working with institutional Learning Analytics project • Providing library data to module teams • Menu of potential ‘data-driven’ interventions include library activities
  • 8.
    Why University of Huddersfield2013 ‘There is a statistically significant relationship between library resource use and level of degree result’ University of Minnesota 2014 ‘Students who utilize academic libraries within their first year have higher cumulative grade point averages and retention, on average, than their peers who did not use library services.’ Library Analytics “Using data about student engagement with library services and content to help institutions and students understand and improve library services to learners.” Other UK HEIs report a correlation between student success / retention and library use – is this the case at the OU? 8
  • 9.
  • 10.
    Project overview Strategy Evidence impact Integration Skills Dataon library use by students not integrated into institutional data spaces No strong evidence base of relationship between library use and retention/achievement at OU Library Services lack data analysis skills Challenges Focus areas 10
  • 11.
    11 How can librarydata benefit students, faculties and the University as a whole? • Better understand student success factors; it may also help to improve the accuracy of student success predictions • Provide evidence and understand the impact of students’ engagement with independent study • Inform improvements to library services and trigger interventions for disengaged students Analysing OU Library data alongside OU data will help to:
  • 12.
    Intended outcomes We can demonstratethat library engagement enhances student achievement and retention Library engagement data is available for use across the OU and routinely being used where key decisions are made about learning Library Staff are confident with using data to show how library use contributes to student success and retention Faculty can be confident that library engagement improves student achievement and retention Faculty can access real-time library data to identify student engagement Faculty can be confident that library staff can use data to support retention & achievement Contribute to improved student success: increased retention and completion 12
  • 13.
  • 14.
    Skills (library staff) Readinessassessment 14 Strongly Disagree NeutralDisagree Agree Strongly Agree Strongly Disagree NeutralDisagree Agree Strongly Agree Strongly Disagree NeutralDisagree Agree Strongly Agree Strongly Disagree NeutralDisagree Agree Strongly Agree Strongly Disagree NeutralDisagree Agree Strongly Agree I am confident that I know what analytics data the library has and I know how to access it I am confident that I know what analytics data the university has and I know how to access it I am confident in my ability to use simple analytics tools I am confident in my ability to analyse data I am confident in my ability to use data to support decision making Strongly Disagree NeutralDisagree Agree Strongly Agree OVERALL High Feb 2016 SDH Confidence levelLow Overall confidence in finding and using data is low. Library staff will need more help to find and use library and university data. Once data tools are in place then they will need help to build skills in analysing and using data.
  • 15.
    Online library resourceaccesses Data on online library resource accesses is taken from EZproxy and OpenAthens systems 15
  • 16.
    Online library resourceaccesses 16 • Starting point URL only • Data files include students who are studying more than one module • Data files include data from students who didn’t complete their module 7.23 13.22 23.25 33.44 0 5 10 15 20 25 30 35 40 Level 1 Level 2 Level 3 PG AVERAGENUMBEROFACCESSESOFLIBRARY RESOURCESPERSTUDENT(MEAN) Online library resource access by level (all faculties, 2014 presentations) Mean per student
  • 17.
    Online library accessesand student achievement 17 n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0 Deferral Fail Pass Distinction Number of library resource accesses per student (mean) Mean number of library resources used per student - 3 level 1 modules (Arts, Law, Technology) 2014-15 presentations Library Data project 2015/16 n=11,501 n=11,406 Pilot study:
  • 18.
    Pilot Study Library useand Age 18 n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] 0.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 i Under 25 ii 26-35 iii 36-45 iv 46-55 v 56 and over AVERAGENUMBEROFLIBRARYRESOURCES ACCESSED Age groups - average use of online library resources - 3 level 1 modules (Arts, Law, Technology) 2014-15
  • 19.
    Pilot Study Library useand gender 19 n = [CELLRANGE] n = [CELLRANGE] 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 Female Male AVERAGENUMBEROFLIBRARYRESOURCES ACCESSED Average use of online library resources by gender - 3 level 1 modules (Arts, Law, Technology) 2014-15
  • 20.
    Pilot Study Library useand level of educational qualification on entry 20 n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] 0.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00 1 No Formal Qualifications 2 Less than A Levels 3 A Levels or equivalent 4 HE Qualification 5 PG Qualification 6 Not known AVERAGENUMBEROFLIBRARYRESOURCESACCESSED Average use of online library resources by previous education - 3 level 1 modules (Arts, Law, Technology) 2014-15 Library Data Project 2016 n=11,501
  • 21.
    21 Online library resourceaccesses and student achievement Wider study: n = [CELLRANGE] n = [CELLRANGE] n = [CELLRANGE] 0.00 5.00 10.00 15.00 20.00 25.00 30.00 Fail Pass Distinction Mean library resource accesses per student - 2014 and 2015 presentations (all modules) Mean resource accesses per student n = 315,460
  • 22.
    22 Online library resourceaccesses and student achievement Wider study:
  • 23.
    23 Online library resourceaccesses and student achievement Wider study:
  • 24.
    The Wollongong methodology 24 Universityof Wollongong Library Cox, B. L. and Jantti, M. (2012) Capturing business intelligence required for targeted marketing, demonstrating value, and driving process improvement, Library & Information Science Research. 34, PP308-316 doi:10.1016/j.lisr.2012.06.002 • Looked at whether there is a relationship between library resource accesses and student success • Aimed to demonstrate Library impact • The approach and methodology was tested with OU data with some small adjustments. • UoW used time accessing resources (from the full ezproxy logfiles), OU has used count of accesses ezproxy and openathens for 2014 students • UoW used Weighted Average Marks – OU used Overall Continuous Assessment Score (OCAS)
  • 25.
  • 26.
    Replicating the Wollongongstudy 26 The average continuous assessment score for students who did not access OU Library online resources is 64. The average continuous assessment score for students that accessed one resource during the year was 68. There is a non-linear correlation between average resource accesses and average student Score (R squared = 0.54)
  • 27.
  • 28.
    Findings so far ●The percentage of students accessing library resources and the number of library resources they access on average increases the higher the level of study ● There is quite a wide variance between different faculties, levels and modules in access to library resources ● There seems to be a similar relationship between library access and student success at the Open University, despite students’ mainly online library access ● Early indications are that the relationship is statistically significant Key indications from our early studies are: 28
  • 29.
  • 30.