A reflection on where we are with learning analytics as a new multi-discipline research area. Reflections from the Learning Analytics Conference 2013 with respect to Assessment.

1. Learning Analytics and Student Feedback
Professor Denise Whitelock
The Open University, Walton Hall,
Milton Keynes MK7 6AA, UK
denise.whitelock@open.ac.uk

2. Learning Analytics and Student Feedback
• What is Learning
Analytics?
• Origins
• Early work
• Learning Analytics
and Assessment
DMW UOC May 2013

3. Definition
“Learning Analytics are concerned with the measurement,
collection, analysis and reporting of data about learners
and their contexts, for purposes of understanding and
optimising learning and the environments in which it
occurs”.
Reference
SoLAR, Open Learning Analytics: An Integrated &
Modularised Platform, White Paper, Society for Learning
Analytics Research, 2011.
DMW UOC May 2013

4. Interdisciplinary Research
• Computer Sciences
• Learning Sciences
• AIED
• Data Mining
specialists
DMW UOC May 2013

5. Middle Space
• 3rd
LAK Conference
• Learning explicit
• New analytic
methods
• Computational
• Representational
• Statistical
• Visualisation
DMW UOC May 2013

6. Can LAK hold together for long?
• Challenges
• Different
methodologies
• Different theories
• Different predjucies
• Agreement on topics 3rd
LAK 2013
• Visualisation, social network
analysis, communication
and collaboration, discourse
analytics, predictive
analytics, sequence
analytics, assessment
After Suthers & Verbert (2013)
DMW UOC May 2013

7. Political and economic drivers
• Educause Review (2007)
• Academic Analytics,
Campbell & Oblinger (2007)
• Large data and stats =
predictive modelling
• Improve number of
graduates in US
• US finding now with school
exam data
• Hand code
• M.L.
• Apply whote set
• Make predictions
DMW UOC May 2013

8. Formative Research towards separate field
• Open Learner models (Bull & Kay, 2007)
• Social Network analysis (De Laat et al, 2007)
• Networks Adapting Pedagogical Practice (SNAPP),
(Dawson et al, 2010)
• Visualisation of large data sets, Honeycomb (van Ham
et al, 2009)
• Gephi: open source tool (Bastian et al, 2009)
• Signals (Arnold, 2010)
DMW UOC May 2013

9. Signals: Flagship Project
• Moves data from VLE
• Combines with prediction
models
• Real time red/amber/green
traffic lights
• Pilot study (Arnold, 2010)
showed
• Students sought help
earlier
• 12% more B/C grades
• 14% less D/F grades
DMW UOC May 2013

10. Formative Assessment and Learning
Analytics (1)
Tempelaar et al, 2013
• 1st
year Math & Stats
undergraduates,
Maastricht
• Reason text book
• Online questions
• Practice and
performance tests
• 92% higher practice
pass
• 51% lower practice pass
DMW UOC May 2013

11. Formative Assessment and Learning
Analytics (2)
Important for SAFeSEA?
• Learning styles (Vermunt, 1996)
• Self regulation for deep learning
• Practice for stepwise learning
• Motivation and engagement wheel (Martin,
2007)
• Learning emotions
• Pekrun’s control-value theory of learning
emotions
DMW UOC May 2013

12. Performance of video lectures
• Findings from Mirriahi &
Dawson (2013)
• Correlations between
quizzes and lectures
• Shows misalignment
between Assessment
and Teaching
materials?
DMW UOC May 2013

13. HOU2LEARN
• PLE from Hellenic Open
University
• Social network analysis
and final grades
• Online collaboration is
not a predictor for final
grade
• Koulocheri & Xenos,
2013
DMW UOC May 2013

14. OpenEssayist: SAFeSEA Web application for
summarisation-based formative feedback
DMW UOC May 2013

15. Key words and phrases visualized in the essay context. Sentences in
light-grey (green) background are key sentences as extracted by the
EssayAnalyser (the number at the start of the sentence indicates its
key-ness ranking); bigrams are indicated in bold (red) and boxed.
DMW UOC May 2013

16. The structural elements of the essay can be used jointly with
the key word extraction to highlight relevant information within
specific parts of the essay, here the introduction (and the
assignment question)
DMW UOC May 2013

17. Key words and phrases as separate lists
DMW UOC May 2013

18. Dispersion of key words across the essay
http://www.open.ac.uk/iet/main/research-scholarship/research-projec
DMW UOC May 2013

19. Can we find ways of using graph visualization
techniques on the key words and key sentences, to
make them helpful and meaningful to students?
DMW UOC May 2013

20. Final Thoughts
• Instant machine
feedback not prevalent
• Artificial Intelligence
analysis to tutors
leading to Wizard of Oz
responses (Shaffer,
2013)
• Just in time feedback is
the ultimate goal
DMW UOC May 2013

21. References (1)
Arnold, K.E. (2010). Signals: applying academic analytics, Educause
Quarterly, 33(1), p10.
http://www.educause.edu/ero/article/signals-applying-academic-analytics
(Accessed 30 April 2013)
Bastien, M., Heymann, S. & Jacomy, M. (2009). Gephi: an open source
software for exploring and manipulating networks. Paper presented at
the International AAAI Conference on Weblogs and Social Media.
Bull, S & Kay, J. (2007). Student models that invite the learner in: the
SMILI:-) open learner modelling framework, International Journal of
Artificial Intelligence in Education, 17(2).
Campbell, J.P. & Oblinger, D.G. (2007). Academic Analytics,
Educause.
Dawson, S., Bakharia, A. & Heathcote, E. (2010). Snapp: Realising the
affordances of real-time SNA within networked learning environments.
Paper presented at The 7th
International Conference on Networked
Learning, Aalborg, Denmark (3-4 May).DMW UOC May 2013

22. References (2)
De Laat, M., Lally, V., Lipponen, L. & Simons, R.-J. (2007).
Investigating patterns of interaction in networked learning and
computer-supported collaborative learning: a role for social network
analysis. International Journal of Computer Supported Collaborative
Learning, 2, pp.87-103.
Koulocheri, E. & Xenos, M. (2013). Considering Formal Assessment in
Learning Analytics within a PLE: The HOU2LEARN Case. Paper
presented at The International Learning Analytics & Knowledge (LAK)
Conference, Leuven, Belgium, (8-12 April 2013)
© 2013 ACM 978-1-4503-1785-6/13/04
Martin, A.J. (2007). Examining a multidimensional model of student
motivation and engagement using a construct validation approach.
British Journal of Educational Psychology, 77, 413-440.
DMW UOC May 2013

23. References (3)
Mirriahi, N. & Dawson, S. (2013) The pairing of lecture recording data
with assessment scores: A method of discovering pedagogical impact.
Paper presented at The International Learning Analytics & Knowledge
(LAK) Conference, Leuven, Belgium, (8-12 April 2013)
© 2013 ACM 978-1-4503-1785-6/13/04
Pekrun, R. (2006). The control-value theory of achievement emotions:
assumptions, corollaries and implications for educational research and
practice. Educational Psychology Review, 18, 315-34.
SoLAR, Open Learning Analytics: An Integrated & Modularised
Platform, White Paper, Society for Learning Analytics Research, 2011.
Suthers, D. & Verbert, K. (2013) Learning Analytics as a “Middle
Space”. Paper presented at The International Learning Analytics &
Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013)
© 2013 ACM 978-1-4503-1785-6/13/04
DMW UOC May 2013

24. References (4)
Tempelaar, D.T., Cuypers, H., van de Vrie, E., Heck, A. & van der
Kooij, H. (2013). Formative Assessment and Learning Analytics. Paper
presented at The International Learning Analytics & Knowledge (LAK)
Conference, Leuven, Belgium, (8-12 April 2013)
© 2013 ACM 978-1-4503-1785-6/13/04
Van Ham, F., Schulz, H.-J. & Dimicco, J.M. (2009). Honeycomb: visual
analysis of large scale social networks, Ifip International Federation for
Information Processing, 429-442.
Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J.
(2013) ‘OpenEssayist: Extractive Summarisation & Formative
Assessment of Free-Text Essays’. Workshop on Discourse-Centric
Learning Analytics, 3rd
Conference on Learning Analytics and
Knowledge (LAK 2013), Leuven, Belgium
Vermunt, J.D. (1996). Leerstijlen en sturen van leerprocessen in het
Hoger Onderwijs. Amsterdam/Lisse: Swets & Zeitlinger.
DMW UOC May 2013