Analytics: as if learning mattered Presentation from 'In Focus: Learner analytics and big data', a CDE technology symposium held at Senate House on 10 December 2013. Conducted by Adam Cooper (Co-Director, Cetis) Audio of the session and more details can be found at www.cde.london.ac.uk.

1. Analytics
(as if learning mattered)
Adam Cooper,
In Focus: Learner analytics and big data symposium,
University of London, December 10th 2013

2. Structure of the Talk
• My Perspective
• Threats
• Avoiding the threats
Image: cba Jaskirat Singh Bawa

3. PERSPECTIVE

4. Perspective of this Talk
Where am I coming from?
UK
Higher Ed
Institution
Cetis is based in the Institute for
Educational Cybernetics at the
University of Bolton.
“Our mission is to develop a
better understanding of how
information and communications
technologies affect the
organisation of education from
individual learning to the global
system.”

5. Defining “Analytics”
Analytics is the process of developing
actionable insights
through
problem definition
and the application of statistical models and analysis
against existing and/or simulated future data.

6. Learning vs Business Venn
Insights to
support
educational
aims and
objectives
Insights to
support
operational
and
strategic
activity.

7. Learning vs Business Venn
Retention
Student Satisfaction
Employability
Achievement
Applications
Easiest Case for
Investment

8. Structure to Manage Complexity
The SMT – “what makes a viable organisation?”
Subject/Discipline Member - “what makes a good chemist?”
(knowledge, approach to problems, creativity, attitudes, values...)
Course Team – “what curriculum?” (topics, suitable LOs and assessment
methods to match our typical student career etc)
Class Teacher – “where are my students collectively and
individually?” (intellectually, emotionally, meta-cognitively...)
Student/Learner – time, place and method of study
Self-organising and self-regulating

9. The Intersection Doppelgänger
Values
Epistemological
Retention
Student Satisfaction
Employability
Cause & Effect Models
Achievement
Moral Imperatives
Applications

10. The “Accommodation”
“The introduction of these techniques [Learning Analytics] cannot be
understood in isolation from the methods of educational management as
they have grown up over the past two centuries. These methods are
conditioned by the fact that educational managers are limited in their
capability to monitor and act upon the range of states which are taken
up by teachers and learners in their learning activities. ... Over the
years, an accommodation has developed between regulatory
authorities, management and teaching professionals: educational managers
indicate the goals which teachers and learners should work towards, provide
a framework for them to act within, and ensure that the results of their
activity meet some minimum standards. The rest is left up to the professional
skills of teachers and the ethical integrity of both teachers and learners.”
Prof. Dai Griffiths, “Implications of Analytics for Teaching Practice in Higher Education”

11. TWO THREATS
(+1 AND A BIT)

12. Threat #1: Interventionism
Image: via Wikimedia Commons p

13. Threat #2: BI Tradition
Image: via Wikimedia Commons p Hhultgren
• Centralised projects
• IT + senior leader
• Management reports
• KPIs
Image: Paul Hodge

14. So, you want to optimise student
success?
• Are the following well-specified?
• “optimise student success”
• “maximise the probability of success”
• What does success look like?
• Really?
• Says who?
• Is a “good education” indicated by student success?

15. Does everyone understand?
Attainment: an academic standard, typically shown by test
and examination results
Progress: how far a learner has moved between
assessment events
Achievement: takes into account the standards of
attainment reached by learners and the progress they
have made to reach those standards
Enrichment: the extent to which a learner gains
professional attitudes, meta-cognition, technical or artistic
creativity, delight, intellectual flexibility, ...

16. +1: Technological Solutionism
„Recasting all complex social situations either as neatly defined problems with
definite, computable solutions or as transparent and self-evident processes that can
be easily optimized—if only the right algorithms are in place!—this quest is likely to
have unexpected consequences that could eventually cause more damage than the
problems they seek to address.
I call the ideology that legitimizes and sanctions such aspirations “solutionism.” I borrow this
unabashedly pejorative term from the world of architecture and urban planning, where it has
come to refer to an unhealthy preoccupation with sexy, monumental, and narrow-minded
solutions—the kind of stuff that wows audiences at TED Conferences—to problems that are
extremely complex, fluid, and contentious … solutionism presumes rather than
investigates the problems that it is trying to solve, reaching “for the answer before the
questions have been fully asked.” How problems are composed matters every bit as
much as how problems are resolved.‟
Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism”, 2013

17. 3M and Six-Sigma
“McNerney axed 8,000 workers (about 11% of the workforce), intensified the performancereview process, and tightened the purse strings at a company that had become a profligate
spender. He also imported GE's vaunted Six Sigma program—a series of management
techniques designed to decrease production defects and increase efficiency. Thousands of
staffers became trained as Six Sigma „black belts‟.
...
Now his successors face a challenging question: whether the relentless emphasis on efficiency
had made 3M a less creative company. ... Those results are not coincidental. Efficiency
programs such as Six Sigma are designed to identify problems in work processes—and
then use rigorous measurement to reduce variation and eliminate defects.
...
Indeed, the very factors that make Six Sigma effective in one context can make it
ineffective in another. Traditionally, it uses rigorous statistical analysis to produce
unambiguous data that help produce better quality, lower costs, and more efficiency.
That all sounds great when you know what outcomes you'd like to control. But what
about when there are few facts to go on—or you don't even know the nature of the
problem you're trying to define?”
Source : Bloomberg Business Week, June 2007.

18. The Bit
Data:
it may be big
but its not clever

19. The Bit: Net Scale Fascination (delusion?)
Data:
it may be big
but its not clever

20. • Hendrik‟s photo
Source unknown – thanks to Hendrik Drachsler for passing it on.

21. Threat Rating=UBI*(I+D)UTS
Where:
UBI = Uncritical BI tradition factor
I
= Interventionist tendencies
D
= Big data fascination
UTS = Uncritical technological solutionism
This is not serious...

22. From: “Analytics in Higher Education:
Benefits, Barriers, Progress, and Recommendations” (ECAR
Report)
Why am I Worried?

23. NEUTRALISING THREATS

24. Data Is Useless Without the Skills to
Analyze It
„...employees need to become:
Ready and willing to experiment: Managers and business analysts
must be able to apply the principles of scientific experimentation to
their business. They must know how to construct intelligent
hypotheses. They also need to understand the principles of
experimental testing and design, including population selection and
sampling, in order to evaluate the validity of data analyses.
...
Adept at mathematical reasoning: How many of your managers
today are really “numerate” — competent in the interpretation and
use of numeric data?‟
Jeanne Harris, HBR Blog, Sept 2012

25. From: “Analytics in Higher Education:
Benefits, Barriers, Progress, and Recommendations” (ECAR
Report)
Is This a Real Problem?

26. Make Some Progress By:
• Directly investing in specialists
• E.g. OU, statisticians and data wranglers
• ECAR Survey Report 2012: “invest in people over tools... hiring
and/or training... analysis”
• Operating a “shared service”
• Higher Education Statistics Agency (HESA)
• Universities and Colleges Admissions Service (UCAS)
• Jisc

27. Data Is Useless Without the Skills to
Analyze It
„...employees need to become:
Ready and willing to experiment: Managers and business analysts
must be able to apply the principles of scientific experimentation to
their business. They must know how to construct intelligent
hypotheses. They also need to understand the principles of
experimental testing and design, including population selection and
sampling, in order to evaluate the validity of data analyses.
...
Adept at mathematical reasoning: How many of your managers
today are really “numerate” — competent in the interpretation and
use of numeric data?‟
Jeanne Harris, HBR Blog, Sept 2012

28. We Get Further with “Soft Capability”
• Appreciation of the limitations of analytics technologies
• Expertise to comprehend, critique, and converse
(as opposed to originate or implement)

29. Visualisations are the Opium
of the People
(c)2008, Ryan Block/Engadget

30. Sparklines: Tufte vs ...

31. Significance and Sample Size
“Flagged for action because 74% of students felt they were
well supported by their supervisor compared to 80% in
similar institutions (difference >5%)”
“We need to talk to Mr. Benn; the drop-out rate doubled in
his class this year.”
Correlation is not Causation
“The predictive model indicates that students with attributes
{X, Y, Z} are only 50% likely to complete.”

32. What Does This Give Us?
(aside from people less likely
to cut themselves on sharp tools)

34. Absorptive Capacity
“...the ability of a firm to recognize the value of new, external
information, assimilate it, and apply it to commercial ends is
critical to its innovative capabilities. We label this capability a
firm‟s absorptive capacity and suggest that it is largely a function of
the firm‟s level of prior related knowledge. [...] We formulate a model
of firm investment in research and development (R&D), in which
R&D contributes to a firm’s absorptive capacity, and test
predictions relating a firm‟s investment in R&D to the knowledge
underlying technical change within an industry.”
Cohen and Levinthal, Admin. Science Quarterly 35(1), 1990

35. How to do LA as if learning mattered
1. Build soft capability.
2. Engage in participatory (collaborative) design.
3. Develop & prototype technology (& models) organically.
4. Reflect on the effect LA innovation has on practice.
5. Design and develop with the educator IN the system.
6. Optimise the environment, not only the outcome.
7. Be realistic about scale/quality of data and its meaning.
Personal Learning Analytics?
• Changed dyamic
• Less “big brother”

36. Participatory
Design
Learning
Analytics
Image: cbn Jesse Millan

37. Thank-you for your attention
These slides are at: http://is.gd/AaiLM

38. Reading and References
Jacqueline Bichsel, “Analytics in Higher Education: Benefits, Barriers, Progress, and
Recommendations”
Available from http://www.educause.edu/ecar
Cohen and Levinthal (1990), “Absorptive Capacity: A New Perspective on Learning and
Innovation” Admin. Science Quarterly 35(1)
http://faculty.fuqua.duke.edu/%7Echarlesw/s591/Bocconi-Duke/Papers/C10/CohenLevinthalASQ.pdf
Adam Cooper, “What is Analytics? Definition and Essential Characteristics”
http://publications.cetis.ac.uk/2012/521
Jeanne Harris, “Data is Useless Without the Skills to Analyze it”
http://blogs.hbr.org/2012/09/data-is-useless-without-the-skills/
Bloomberg Business Week, “At 3M, A Struggle Between Efficiency And Creativity”
http://www.businessweek.com/stories/2007-06-10/at-3m-a-struggle-between-efficiency-and-creativity
Dai Griffiths, “CETIS Analytics Series: The impact of analytics in Higher Education on academic
practice”
http://publications.cetis.ac.uk/2012/532
Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism”
ISBN-10: 9781610391382
Phil Winne, “Improving Education” (lecture)
http://www.youtube.com/watch?v=dBxMZoMTIU4

39. Licence
“Analytics (as if learning mattered)” by Adam Cooper, Cetis,
is licensed under the Creative Commons Attribution 3.0 Unported Licence
http://creativecommons.org/licenses/by/3.0/
Learning Analytics Community Exchange
http://www.laceproject.info
Case studies and white papers:
Cetis Analytics Series, http://publications.cetis.ac.uk/c/analytics
Me:
http://blogs.cetis.ac.uk/adam
a.r.cooper@bolton.ac.uk
Cetis:
http://www.cetis.ac.uk