The document discusses the limitations of learning analytics in understanding student interactions and outcomes, particularly highlighting the distinction between visible and hidden learning environments. It emphasizes the importance of acknowledging the 'hidden learning environment' where students engage with various resources beyond the formal virtual learning environment (VLE). The authors recommend a multi-faceted approach to data analysis to improve curriculum design and student outcomes.

1. bournemouth.ac.uk
Hidden learning spaces: What learning
analytics cannot tell us
Andrew Kitchenham and David Biggins
4 September 2019

2. bournemouth.ac.uk 2
Introduction
Andrew Kitchenham David Biggins

3. bournemouth.ac.uk 3
Scene setting
• Sector-wide interest in learning from data
• BU invested in a new VLE in 2017
• Exploratory learning analytics
• Piloting a project to use data to:
• Understand student interaction and staff VLE use
• Improve student experience, retention and outcomes
• Develop the use of the VLE
• Inform decision-making
• Informed by student input (questionnaire n=26)

4. bournemouth.ac.uk 4
Questions
1. Is your institution collecting
learning data?
2. What is the primary source
of that data?

5. bournemouth.ac.uk 5
Terminology
Core learning
materials
eg
lecture slides,
assignment briefs
Supplementary
learning materials
eg
External learning
tools,
ebooks
Independent
learning materials
eg
YouTube,
Google search

6. bournemouth.ac.uk 6
Two student journeys
Student A
Interaction
Outcome
Student B
VLE

7. bournemouth.ac.uk 7
Two student journeys
Student A
Interaction
Outcome
Student B
VLE
Visible
?
Hidden

8. bournemouth.ac.uk 8
Hidden learning environment
• Use of the VLE (frequently or very frequently)
• Core materials 80%
• Assignment briefs 90%
• Supplementary material 27%
VLE
Visible
HLE
Hidden

9. bournemouth.ac.uk 9
Hidden learning environment
VLE
Visible
HLE
Hidden
• Use of the HLE (frequently or very frequently)
• Core materials from peers 34%
• Assignment briefs 15%
• Supplementary material 15%
• Independent learning materials 80%

10. bournemouth.ac.uk 10
Hidden learning environment
VLE
Visible
HLE
Hidden
• What is in the HLE?
Social media
Internet search
iTunesU
YouTube
• Why the HLE?
• Has always existed
• Connectivist learning and generational characteristics
• Oversight aversion
• Lack of confidence in using the VLE
Internet:
Expert: 66%
VLE:
Expert: 13%

11. bournemouth.ac.uk 11
Hidden learning environment
‘we must understand the
underlying structure of the
phenomenon we seek to
explore with analytics
prior to digging into the data’
(Knight 2014)

12. bournemouth.ac.uk 12
Hidden learning environment
VLE
Visible
• Taking the HLE into account
• A twin-track approach
• Accept the limitations on what VLE data tells us
• Only part of student learning comes from the VLE
• Avoid reliance/decision-making on visible data
• Move beyond VLE data ..

13. bournemouth.ac.uk 13
Beyond VLE engagement
• Alternate indicators
• Attendance
• Formative assessment
• Demographics
• Awareness can lead to
changes in behaviour
• Changes in levels, not
absolutes
• Issues
• Institutional acceptance
• Policy and guidance
• GDPR
• Copyright
• Staff appreciation
• Learning design

14. bournemouth.ac.uk 14
Back to the HLE: Next steps
• Move from exploration to explanation
• What HLEs are being used and why?
• Why students don’t engage in all VLE components?
• What is the optimal recipe?
• Exploit the available data to
• Inform curriculum design
• Improve student outcomes
• Develop staff confidence and capability
• Mature appreciation
• Nuances of data interpretation

15. bournemouth.ac.uk 15
Questions
• akitchenham / dbiggins @ bournemouth.ac.uk

16. bournemouth.ac.uk 16
References
• Educause Horizon Report 2019. https://library.educause.edu/-
/media/files/library/2019/4/2019horizonreport.pdf?la=en&hash=C8E8D444AF372E705FA1BF9D4
FF0DD4CC6F0FDD1
• Educause Review. 2017. Learning Analytics: Avoiding failure.
https://er.educause.edu/articles/2017/7/learning-analytics-avoiding-failure
• Jayaprakash, S.M., Moody, E.W., Lauría, E.J., Regan, J.R. and Baron, J.D., 2014. Early alert of
academically at-risk students: An open source analytics initiative. Journal of Learning
Analytics, 1(1), pp.6-47.
• Jisc briefing 2019. Learning analytics and student success – assessing the evidence.
http://repository.jisc.ac.uk/6560/1/learning-analytics_and_student_success.pdf
• Knight, S., A. F. Wise, G. Arastoopour, D. Williamson Shaffer, S. Buckingham Shum, and P. A.
Kirschner. 2014. “Analytics for Learning and Becoming in Practice.” International Conference of
the Learning Sciences (ICLS 2014), 23–27 June 2014, Boulder, Colorado, 1680–1683.
• Shoufan, A., 2019. Estimating the cognitive value of YouTube's educational videos: A learning
analytics approach. Computers in Human Behavior, 92, pp.450-458.
• Society for Learning Analytics research (SOLAR). https://solaresearch.org/
• Wilson, A., Watson, C., Thompson, T.L., Drew, V. and Doyle, S., 2017. Learning analytics:
challenges and limitations. Teaching in Higher Education, 22(8), pp.991-1007.