Talk for ALTC 2018 using Jisc student insight survey data, including factor analysis to identify most important factors in explaining the student digital experience.

1. What organisational variables support a positive student
digital experience?
Helen Beetham &Tabetha Newman
ALT-C
11/07/2018

2. Overview of today’s session
Intro Quant Qual Summary
What
next?

3. The vision: understanding the ‘student digital experience’

4. »Build capacity for student-centred research
»Iterative, participative development of research instrument(s)
and approaches
»Community of practice: shared process, findings, outcomes
»Cross-sector (HE / FE)
»Co-ownership of data
»Meaningful stakeholder engagement, including teaching staff
and students
»Local process and benefits first: national process second
The tracker / Insights vision

5. The research instrument (student survey)
› Four themes
› 70 questions
› Quantitative and
qualitative data
› 2 x rating scales
(infrastructure, digitalT&L)

6. Institutional variables
»External metrics: e.g. NSS,TEF …
»Ten questions to institutional leads, hypothesised to correlate
with student digital experience (lit review, consultation)

7. The data set : students & institutions
»43 Universities and 36 FE Colleges
»Approx. 30% UK Universities and 16% of UK Colleges
» 37,000 university and college students
»Av. 444 responses per institution

8. Methods

9. Research methods 1
» 70 questions, 37,000 UK students, +2 million
cells of data
» Quantitative analysis (SPSS):
› Q by Q summary
actor Analysis: could we identify clusters of
the most important Qs (‘factors’)
» Qualitative analysis (Nvivo):
› Free text feedback to further explore the
data

10. Research methods 2
» 70 questions, 37,000 UK students, +2 million
cells of data
» Quantitative analysis (SPSS):
› Q by Q summary
› Factor Analysis: could we identify clusters
of the most important Qs (‘factors’)
» Qualitative analysis (Nvivo):
› Free text feedback to further explore the
data
› Do student comments support factor
analysis themes?

11. Factor Analysis
What?
› Identify which Qs had the largest impact on student opinion
› Identify whether / how they fit into thematic clusters (‘factors’)
› Explore whether they relate to our institutional variables
Why use FA?
› Describes ”student digital experience” from student perspective
› Models most important factors relative to one another
› Validates and refines survey instrument
› Supports evidence-based advice to institutions
13/06/2018 https://digitalstudent.jisc.ac.uk #digitalstudent

12. What is factor analysis?
»The first factor tries to explain
as much variation in student
opinion as it can
-20
0
20
-20 0 20

13. What is factor analysis?
-20
0
20
-20 0 20
F1
»The first factor tries to explain
as much variation in student
opinion as it can

14. -20
0
20
-20 0 20
What is factor analysis?
F2
F1
»The first factor tries to explain
as much variation in student
opinion as it can
» Then when it cannot go on,
the next one starts to explain
the remaining variation

15. -20
0
20
-20 0 20
What is factor analysis?
F2
F3
F1

16. Quantitative results – Factor Analysis

17. Quantitative results
»FactorAnalysis (EFA) reduced 70 questions to just 8 factors
»Analysed University (n=23,000) and College (n=14,000) data
separately and …
»… got the SAME eight factors!
»We have a robust model of “digital experience” from student
perspective
»The 8 factors describe 51% of all variation in the dataset
»Map well onto survey instrument (validates design)
»Uni student data: 16 statistically important questions
»College student data: 16 + 11 (27) …

18. The model
F1
My digital development
(20%, 3Qs)
F2
My digital course
experience (6%, 4Qs)
F3
Frequency of digital
activities on course (6%,
2Qs)
F4 Anytime
access to online
resources (4%,1Q)
F5 My VLE
experience (4% 1Q)
F6 Ind. digital
learning (3%,
2Qs) F8 Digital
investment
(3%, 2Qs)
F7 My digital
wellbeing (3%,
1Q)
8 factors of student
digital experience
(Uni)

19. All 8 Factors & the University Qs
Factor The 16 questions contributing to each factor (scoring >0.65)
F1: My digital development
• I have regular opportunities to review and update my digital skills
• Before I started my course I was told what digital skills I would need
• Learners are given the chance to be involved in decisions about digital services
F2: My digital course experience
When digital technologies are used on my course …
• ...I enjoy learning more
• ...I understand things better
• ...I can fit learning into my life more easily
• ...I am more independent in my learning
F3: Frequency of digital activities
on my course
As part of your course, how often do you do the following digital activities?
• Use a polling device or online quiz to give answers in class
• Use an educational game or simulation for learning
F4: Anytime access to valued
online resources
Which of these do you have access to at institution whenever you need them?
• … ebooks and ejournals
F5: My VLE experience • I regularly access the VLE on a mobile device
F6: My independent digital
learning
In your own learning time, how often do you use digital tools or apps to:
• ...manage links
• ... organise your study time
F7: My digital wellbeing • I can participate in student union / club / society activities online
F8: Institutional digital investment
• Teaching rooms are well designed for the technologies we use
• The software used on my course is industry standard and up-to-date

20. All 8 Factors & the College Qs
Factor The 27 questions contributing to each factor (scoring >0.65)
F1: My digital development
• I have regular opportunities to review and update my digital skills
• Before I started my course I was told what digital skills I would need
• Learners are given the chance to be involved in decisions about digital services
F2: My independent digital learning
In your own learning time, how often do you use digital tools or apps to:
• ...manage links
• ... organise your study time
• … look for additional resources not recommended by your tutor
• … access lecture notes or recorded lectures
• … make notes or recordings
F3: My digital course experience
When digital technologies are used on my course …
• ...I enjoy learning more
• ...I understand things better
• ...I can fit learning into my life more easily
• ...I am more independent in my learning
F4: Anytime access to valued online
resources
Which of these do you have access to at institution whenever you need them?
• … ebooks and ejournals
• … File storage and backup
• … online course materials
F5: My VLE experience
• I regularly access the VLE on a mobile device
• I rely on it to do my coursework
• I would like it to be used more by my tutors
• I enjoy using the collaborative features
F6: Frequency of digital activities on my
course
As part of your course, how often do you do the following digital activities?
• Use a polling device or online quiz to give answers in class
• Use an educational game or simulation for learning
F7: My digital wellbeing
• I can participate in student union / club / society activities online
• The institution helps me stay safe online
• I can access institution health & wellbeing services online
• The institution protects my data privacy
F8: Institutional digital investment • Teaching rooms are well designed for the technologies we use
• The software used on my course is industry standard and up-to-date

21. »Factor 1 explains one fifth (20%) of all variation in opinion
»We called it “my digital development”
»It is made up of three statements from Q17:
• I have regular opportunities to review and update my digital skills
• Before I started my course I was told what digital skills I would
need
• Learners are given the chance to be involved in decisions about
digital services
The importance of Factor 1

22. »Factor 1 explains one fifth (20%) of all variation in opinion
»We called it “my digital development”
»It is made up of three statements from Q17:
»What can institutions do to ensure these issues are addressed?
• I have regular opportunities to review and update my digital skills
• Before I started my course I was told what digital skills I would
need
• Learners are given the chance to be involved in decisions about
digital services
The importance of Factor 1

23. »Factor 8 explains 3% of all variation in opinion
»The one factor that shows a direct relationship between
investment in infrastructure and the student digital experience
»We called it “institutional digital investment”
»It is made up of two statements from Q16:
• Teaching rooms are well designed for the technologies we use
• The software used on my course is industry standard and up-to-
date
The importance of Factor 8

24. »Calculated average factor
score per University (n=24 with
+370 responses)
»University’s overall NSS score
correlates* with Factors 2, 4, 6,
7 and nearly with Factors 1 and
8
»Conclusion: Our model
measures some of what NSS
measures
* Pearson’s Correlation: NSS versus F1 (0.39, p=0.07); F2 (0.53, p=0.009); F3 (0.39, p=0.07); F4 (-0.56, p=0.006, F6 (0.67, p<001); F7 (0.56, p=0.006); F8 (-0.38, p=0.008)
Correlation with external institutional metrics
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
78 80 82 84 86 88 90 92 94 96
NSS (2017) score
NSS score v Factor 2, 4, 6 and 7

25. Relationship with institutional Qs
»Only one significant relationship*
»Investment in student digital skills
relates to students’ average score
for factor 7 (‘digital wellbeing’)
»Conclusion: Investing in more forms
of digital support = more positive
digital experience
13/06/2018 https://digitalstudent.jisc.ac.uk #digitalstudent

26. Qualitative results – free text analysis

27. Qualitative approach
» Questions analysed:
› Please give an example of:
– a digital tool or app you find really useful for learning
– a digital course activity you have found really useful
› To improve your experience of digital teaching and learning:
– what one thing should we DO / should we NOT DO?

28. Qualitative approach
» Questions analysed:
› Please give an example of:
– a digital tool or app you find really useful for learning
– a digital course activity you have found really useful
› To improve your experience of digital teaching and learning:
– what one thing should we DO / should we NOT DO?
» Overall approach (neutral, then hypothesis-led by quant findings)
› Word frequency analysis
› Text searches of high frequency/significant words to explore context
› Coding of selected groups of responses

29. Qualitative approach: use of Nvivo

30. How do student comments support FA themes?
F1
My digital development
F2
My digital course
experience
F3
Frequency of digital
activities on course
F4 Anytime
access to online
resources
F5 My VLE
experience
F6 Ind. digital
learning
F8 Digital
investment
F7 My digital
wellbeing

31. Do student comments support factor analysis themes?
»Yes – students talk about all the
themes (factors) we identified,…
‘Use different apps to
improve our skills.’
‘Organise workshops …
on how to get the most
out of digital learning’
‘Teach skills we need for
the future as well as now’
‘Employability digital
fundamentals’

32. Do student comments support factor analysis themes?
»Yes – students talk about all the
themes (factors) we identified,…
»But practical (transactional) issues
came to the fore when students were
asked for ‘just one’ improvement.
Computer rooms are “crowded”,
“noisy”, “booked out”, “filthy”, “full of
desk hogs” and “a street market”!
‘Use different apps to
improve our skills.’
‘Organise workshops …
on how to get the most
out of digital learning’
‘Teach skills we need for
the future as well as now’
‘Employability digital
fundamentals’

33. How do student comments enrich EFA?
»Factor 1: ‘my digital development’
»When students talk about their skills
and development, and the kinds of
support they value, they talk about:
› ICT/IT and study/learning skills (FE)
› Digital/e-learning, IT/computer and
learning/academic skills (HE)
› But also: practical skills, software
development, skills for work,
creativity/productivity, library skills,
research, memory and revision…

34. › Factor 8: ‘institutional investment’
› FE students mainly concerned with issues of access:
reliable wifi, online services, usable devices, locations to study
› HE students more concerned with quality of provision:
especially high-value academic resources such as e-books and
journals, software, and facilities in lectures
› Both are concerned about cost, value for money, and equity
› Both are struggling to integrate their personal devices and
practices with the institutional digital environment
How do student comments enrich EFA?

35. What can institutions do to improve the
student digital experience?

36. Focus on students’ digital development
»From Factor 1:
› Support students to review and update their digital skills
› Tell students what digital skills they need before they start, and
how their course will prepare them for the future
› Involve students in decisions about their digital environment
»Review the other factors (hand-out)
»Read the report, including:
› Student personas
› Analysis of over 100k student comments
»But most importantly…

37. … engage your students!
»Run the survey (now the Jisc ‘Insights’
service)
»Follow our guidance on engaging
students…
»… and on participative approaches
(reflect and explore)
»Have confidence that you can
work with qual and quant data
and in partnership with your students!

38. Find out more
»Launched today:
› Full report on UK student data + ‘At a glance’ report
»Reports in progress:
› Teaching staff (N=1,900 lecturers from 15 institutions)
› International students (n=23,000 fromAustralia / New Zealand)
»Briefings/resources in progress:
› Briefings for teaching staff, students and leaders
› Briefings on key topics (e.g. well-being, transition to work)
› Toolkit for arriving students
› New student benchmarking tool (with the NUS)

39. Get in touch…
Digital experience insights
https://digitalinsights.jisc.ac.uk
Except where otherwise noted, this
work is licensed under CC-BY-NC-ND
https://digitalstudent.jisc.ac.uk #digitalstudent13/06/2018

40. How do student comments support…?
»F8: ‘institutional investment’: typical comments
The computers are
broken and old
Purchase more
computers as
there are not
enough for every
student in my
class.
A faster and
more open
internet service
The wifi is atrocious,
constantly dropping
out
Provide modern
learning tools and
resources and
update your
software
University
subscriptions do
not provide access
to some of the
online journals I
need
More places on
the main campus
with mains
sockets, wifi and
seating
Ensure consistent
use of lecture
recording facilities