1. Learning Analytics:
Welcome to the future of assessment?
Simon Buckingham Shum
Knowledge Media Institute, The Open University
Visiting Fellow, University of Bristol
(From August, University of Technology Sydney)
simon.buckinghamshum.net
twitter @sbskmi #LearningAnalytics #edmedia See the question at #edmediakeynote
Keynote
address,
EdMedia
2014,
25th
June,
Tampere,
Finland
1

2. learning objective: leave with
an expanded vision of analytics
better questions to ask in your next
analytics conversation
2

3. Big Data status report:
3
“Big data is like teenage sex: everyone
talks about it, nobody really knows
how to do it, everyone thinks
everyone else is doing it, so everyone
claims they are doing it...”
https://www.facebook.com/dan.ariely/posts/904383595868

4. When the Chancellor announces the adoption
of a new economic modelling technique…
4
…we query the
limitations
of the model

5. 5
…we query the
limitations
of the model

6. Siri is smart…
6
I was speaking at this event:
http://codeactsineducation.wordpress.com

7. Siri is smart…
“Find code acts in
education”
7

8. Siri is smart…
“Find code acts in
education”
8

9. Similarly, when we are confronted with
new learning analytics…
LAK13 Panel: Educational Data Scientists: A Scarce Breed
http://people.kmi.open.ac.uk/sbs/2013/03/lak13-edu-data-scientists-scarce-breed
John Behrens
(Pearson)
9

10. LAK13 Panel: Educational Data Scientists: A Scarce Breed
http://people.kmi.open.ac.uk/sbs/2013/03/lak13-edu-data-scientists-scarce-breed
John Behrens
(Pearson)
10
…we should query the limitations of the model

11. h=ps://twi=er.com/Wiswijzer2/status/414055472451575808
“Note:
check
the
huge
difference
between
knowing
and
measuring…”
11

12. a
few
quick
examples
of
learning
analy5cs
12

13. It’s out of the labs and into products: every learning
tool now has an “analytics dashboard” (a Google image search)
13

14. Intelligent tutoring for skills mastery (CMU)
Lovett M, Meyer O and Thille C. (2008) The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student
learning. Journal of Interactive Media in Education 14. http://jime.open.ac.uk/article/2008-14/352
“In this study, results showed that
OLI-Statistics students [blended
learning] learned a full semester’s
worth of material in half as much
time and performed as well or
better than students learning from
traditional instruction over a full
semester.”

15. Purdue University Signals: real time traffic-lights for
students based on predictive model
15
Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE
Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x
Validate a statistical model from:
• ACT or SAT score
• Overall grade-point average
• CMS usage composite
• CMS assessment composite
• CMS assignment composite
• CMS calendar composite
Predicted 66%-80% of struggling
students who needed help

16. Purdue University Signals: real time traffic-lights for
students based on predictive model
16
Pistilli, M. D., Arnold, K. and Bethune, M., Signals: Using Academic Analytics to
Promote Student Success. EDUCAUSE Review Online, July/Aug., (2012).
http://www.educause.edu/ero/article/signals-using-academic-analytics-
promote-student-success
“Results thus far show that students
who have engaged with Course Signals
have higher average grades and seek
out help resources at a higher rate
than other students.”

17. Spatial clustering algorithm to provoke reflection
17

18. Posture analysis of fieldwork students
18

19. …and many more examples including
discourse analytics language technologies to assess the quality
of online postings and debate
social network analytics graph analytics to assess strength
and topics of interpersonal ties
epistemic game analytics assessing the degree of
professional engagement in authentic project scenarios
visualizations to reveal important patterns of tool use over time
(see other presentations and tutorials)
19

20. but
before
we
get
carried
away,
let’s
just
pause…
20

21. Selwyn, N. (2014). Data entry: towards the critical study of digital data and education. Learning, Media and Technology. http://dx.doi.org/
10.1080/17439884.2014.921628
“observing, measuring, describing,
categorising, classifying, sorting, ordering
and ranking). […] these processes of meaning-making are never
wholly neutral, objective and ‘automated’ but are fraught with
problems and compromises, biases and
omissions.
21

22. For Morozov, analytics is
where technological
solutionism hits education:
22
“This flight from thinking
and the urge to replace
human judgments with
timeless truths produced by
algorithms is the underlying
driving force of
solutionism.”

23. Could analytics help us shift from the calculating
mind to the contemplative mind?
23
See also:
Complexity, Computing, Contemplation, Learning?
http://learningemergence.net/2011/05/04/cccl
http://www.contemplativecomputing.org/2011/03/first-draft-of-a-contemplative-computing-article.html
Alex Pang: “A contemplative stance can help people be
more creative; deal with complex problems that
require months or years to solve […]
Contemplation promotes both self-sufficiency and
close, questioning observation of the world, and both
are particularly valuable in this moment in the history
of technology.”
Calculating Mind, Contemplative Mind
http://people.kmi.open.ac.uk/sbs/2008/09/calculating-contemplative-mind

24. unpacking
the
core
ques5on
for
learning
analy5cs
24

25. can
we
tell
from
your
digital
profile
if
you’re
learning?
25

26. can
we
tell
from
your
digital
profile
if
you’re
learning?
26
Who?

27. can
we
tell
from
your
digital
profile
if
you’re
learning?
27
Who?
How? With what confidence?
After what kinds of training?

28. can
we
tell
from
your
digital
profile
if
you’re
learning?
28
Who?
How? With what confidence?
After what kinds of training?
Sourcing which data,
with what integrity?

29. can
we
tell
from
your
digital
profile
if
you’re
learning?
29
Who?
How? With what confidence?
After what kinds of training?
Sourcing which data,
with what integrity?
What kind of learning?
What kind of learner?

30. Accounting tools are not neutral
Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life. Sage, London. pp. 12-13
“accounting tools...do not simply
aid the measurement of economic
activity, they shape the
reality they measure”

31. In
what
senses
do
analy5cs
“shape
the
reality
they
measure”?
31

32. How
do
analyQcs
shape
educaQon?
Analytics reports at the
organisational and national
levels come with
consequences at different
scales — sometimes punitive,
often impacting millions of
people.
PoliQcally
32

33. How
do
analyQcs
shape
educaQon?
What data, concepts
and relationships do
the analytics
designers seek to
model?
Ontologically
33

34. Bowker, G. C. and Star, L. S. (1999). Sorting Things Out: Classification and Its Consequences. MIT Press, Cambridge, MA, pp. 277, 278, 281
“Classification systems provide both a
warrant and a tool for forgetting [...]
what to forget and how to forget it [...]
The argument comes down to asking not
only what gets coded in but what gets
coded out of a given scheme.”
34

35. Visualising “attainment” and “progress”
35

36. Which analytics could reflect the progress that ‘Joe’
has made on so many other fronts other than his SATS?
36

37. Key modelling issue: unit of analysis
! Discourse analysis: how do machines and humans differ in the
way they segment a transcript to make sense of it?
! Rosé, C. P., & Tovares A. (in press). What Sociolinguistics and Machine Learning Have to Say to One Another about Interaction
Analysis. In L. Resnick, Asterhan C., & Clarke S. (Eds.), Socializing Intelligence Through Academic Talk and Dialogue. Washington,
D.C.: American Educational Research Association
! Collective intelligence: If we are shifting from a sole focus on
individual accomplishment, to that of group knowledge
construction and performance, how do analytics assess
changes in a group’s knowledge and processes?
! Chen, B., & Resendes, M. (2014). Uncovering what matters: Analyzing transitional relations among contribution types in
knowledge-building discourse. In Proceedins of the Fourth International Conference on Learning Analytics And
Knowledge - LAK ’14 (pp. 226–230). New York, New York, USA: ACM Press. doi:10.1145/2567574.2567606
37

38. How
do
analyQcs
shape
educaQon?
What thresholds, samples,
relationships, patterns, etc. do
the algorithms encode and
seek?
On what basis is a
recommendation engine
proposing interventions?
Algorithmically
38

39. gov