Analytics
(as if learning mattered)
Adam Cooper,
In Focus: Learner analytics and big data symposium,
University of London,...
Structure of the Talk

• My Perspective

• Threats
• Avoiding the threats
Image: cba Jaskirat Singh Bawa
PERSPECTIVE
Perspective of this Talk
Where am I coming from?
UK

Higher Ed

Institution

Cetis is based in the Institute for
Education...
Defining “Analytics”

Analytics is the process of developing

actionable insights
through
problem definition
and the appli...
Learning vs Business Venn

Insights to
support
educational
aims and
objectives

Insights to
support
operational
and
strate...
Learning vs Business Venn
Retention
Student Satisfaction

Employability
Achievement
Applications
Easiest Case for
Investme...
Structure to Manage Complexity
The SMT – “what makes a viable organisation?”
Subject/Discipline Member - “what makes a goo...
The Intersection Doppelgänger

Values
Epistemological

Retention
Student Satisfaction

Employability
Cause & Effect Models...
The “Accommodation”
“The introduction of these techniques [Learning Analytics] cannot be
understood in isolation from the ...
TWO THREATS
(+1 AND A BIT)
Threat #1: Interventionism

Image: via Wikimedia Commons p
Threat #2: BI Tradition
Image: via Wikimedia Commons p Hhultgren

• Centralised projects
• IT + senior leader
• Management...
So, you want to optimise student
success?

• Are the following well-specified?
• “optimise student success”
• “maximise th...
Does everyone understand?
Attainment: an academic standard, typically shown by test
and examination results
Progress: how ...
+1: Technological Solutionism
„Recasting all complex social situations either as neatly defined problems with
definite, co...
3M and Six-Sigma
“McNerney axed 8,000 workers (about 11% of the workforce), intensified the performancereview process, and...
The Bit

Data:
it may be big
but its not clever
The Bit: Net Scale Fascination (delusion?)

Data:
it may be big

but its not clever
• Hendrik‟s photo

Source unknown – thanks to Hendrik Drachsler for passing it on.
Threat Rating=UBI*(I+D)UTS
Where:
UBI = Uncritical BI tradition factor
I
= Interventionist tendencies
D
= Big data fascina...
From: “Analytics in Higher Education:
Benefits, Barriers, Progress, and Recommendations” (ECAR
Report)

Why am I Worried?
NEUTRALISING THREATS
Data Is Useless Without the Skills to
Analyze It
„...employees need to become:

Ready and willing to experiment: Managers ...
From: “Analytics in Higher Education:
Benefits, Barriers, Progress, and Recommendations” (ECAR
Report)

Is This a Real Pro...
Make Some Progress By:

• Directly investing in specialists
• E.g. OU, statisticians and data wranglers
• ECAR Survey Repo...
Data Is Useless Without the Skills to
Analyze It
„...employees need to become:
Ready and willing to experiment: Managers a...
We Get Further with “Soft Capability”

• Appreciation of the limitations of analytics technologies
• Expertise to comprehe...
Visualisations are the Opium
of the People

(c)2008, Ryan Block/Engadget
Sparklines: Tufte vs ...
Significance and Sample Size
“Flagged for action because 74% of students felt they were
well supported by their supervisor...
What Does This Give Us?
(aside from people less likely
to cut themselves on sharp tools)
Absorptive Capacity
“...the ability of a firm to recognize the value of new, external

information, assimilate it, and app...
How to do LA as if learning mattered

1. Build soft capability.
2. Engage in participatory (collaborative) design.
3. Deve...
Participatory
Design

Learning
Analytics

Image: cbn Jesse Millan
Thank-you for your attention

These slides are at: http://is.gd/AaiLM
Reading and References
Jacqueline Bichsel, “Analytics in Higher Education: Benefits, Barriers, Progress, and
Recommendatio...
Licence

“Analytics (as if learning mattered)” by Adam Cooper, Cetis,
is licensed under the Creative Commons Attribution 3...
In Focus presentation: Analytics: as if learning mattered
Upcoming SlideShare
Loading in...5
×

In Focus presentation: Analytics: as if learning mattered

593

Published on

Analytics: as if learning mattered

Presentation from 'In Focus: Learner analytics and big data', a CDE technology symposium held at Senate House on 10 December 2013. Conducted by Adam Cooper (Co-Director, Cetis)

Audio of the session and more details can be found at www.cde.london.ac.uk.

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
593
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • title slideadd title into box shown: add presenters
  • body copy no bullets
  • In Focus presentation: Analytics: as if learning mattered

    1. 1. Analytics (as if learning mattered) Adam Cooper, In Focus: Learner analytics and big data symposium, University of London, December 10th 2013
    2. 2. Structure of the Talk • My Perspective • Threats • Avoiding the threats Image: cba Jaskirat Singh Bawa
    3. 3. PERSPECTIVE
    4. 4. Perspective of this Talk Where am I coming from? UK Higher Ed Institution Cetis is based in the Institute for Educational Cybernetics at the University of Bolton. “Our mission is to develop a better understanding of how information and communications technologies affect the organisation of education from individual learning to the global system.”
    5. 5. Defining “Analytics” Analytics is the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data.
    6. 6. Learning vs Business Venn Insights to support educational aims and objectives Insights to support operational and strategic activity.
    7. 7. Learning vs Business Venn Retention Student Satisfaction Employability Achievement Applications Easiest Case for Investment
    8. 8. Structure to Manage Complexity The SMT – “what makes a viable organisation?” Subject/Discipline Member - “what makes a good chemist?” (knowledge, approach to problems, creativity, attitudes, values...) Course Team – “what curriculum?” (topics, suitable LOs and assessment methods to match our typical student career etc) Class Teacher – “where are my students collectively and individually?” (intellectually, emotionally, meta-cognitively...) Student/Learner – time, place and method of study Self-organising and self-regulating
    9. 9. The Intersection Doppelgänger Values Epistemological Retention Student Satisfaction Employability Cause & Effect Models Achievement Moral Imperatives Applications
    10. 10. The “Accommodation” “The introduction of these techniques [Learning Analytics] cannot be understood in isolation from the methods of educational management as they have grown up over the past two centuries. These methods are conditioned by the fact that educational managers are limited in their capability to monitor and act upon the range of states which are taken up by teachers and learners in their learning activities. ... Over the years, an accommodation has developed between regulatory authorities, management and teaching professionals: educational managers indicate the goals which teachers and learners should work towards, provide a framework for them to act within, and ensure that the results of their activity meet some minimum standards. The rest is left up to the professional skills of teachers and the ethical integrity of both teachers and learners.” Prof. Dai Griffiths, “Implications of Analytics for Teaching Practice in Higher Education”
    11. 11. TWO THREATS (+1 AND A BIT)
    12. 12. Threat #1: Interventionism Image: via Wikimedia Commons p
    13. 13. Threat #2: BI Tradition Image: via Wikimedia Commons p Hhultgren • Centralised projects • IT + senior leader • Management reports • KPIs Image: Paul Hodge
    14. 14. So, you want to optimise student success? • Are the following well-specified? • “optimise student success” • “maximise the probability of success” • What does success look like? • Really? • Says who? • Is a “good education” indicated by student success?
    15. 15. Does everyone understand? Attainment: an academic standard, typically shown by test and examination results Progress: how far a learner has moved between assessment events Achievement: takes into account the standards of attainment reached by learners and the progress they have made to reach those standards Enrichment: the extent to which a learner gains professional attitudes, meta-cognition, technical or artistic creativity, delight, intellectual flexibility, ...
    16. 16. +1: Technological Solutionism „Recasting all complex social situations either as neatly defined problems with definite, computable solutions or as transparent and self-evident processes that can be easily optimized—if only the right algorithms are in place!—this quest is likely to have unexpected consequences that could eventually cause more damage than the problems they seek to address. I call the ideology that legitimizes and sanctions such aspirations “solutionism.” I borrow this unabashedly pejorative term from the world of architecture and urban planning, where it has come to refer to an unhealthy preoccupation with sexy, monumental, and narrow-minded solutions—the kind of stuff that wows audiences at TED Conferences—to problems that are extremely complex, fluid, and contentious … solutionism presumes rather than investigates the problems that it is trying to solve, reaching “for the answer before the questions have been fully asked.” How problems are composed matters every bit as much as how problems are resolved.‟ Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism”, 2013
    17. 17. 3M and Six-Sigma “McNerney axed 8,000 workers (about 11% of the workforce), intensified the performancereview process, and tightened the purse strings at a company that had become a profligate spender. He also imported GE's vaunted Six Sigma program—a series of management techniques designed to decrease production defects and increase efficiency. Thousands of staffers became trained as Six Sigma „black belts‟. ... Now his successors face a challenging question: whether the relentless emphasis on efficiency had made 3M a less creative company. ... Those results are not coincidental. Efficiency programs such as Six Sigma are designed to identify problems in work processes—and then use rigorous measurement to reduce variation and eliminate defects. ... Indeed, the very factors that make Six Sigma effective in one context can make it ineffective in another. Traditionally, it uses rigorous statistical analysis to produce unambiguous data that help produce better quality, lower costs, and more efficiency. That all sounds great when you know what outcomes you'd like to control. But what about when there are few facts to go on—or you don't even know the nature of the problem you're trying to define?” Source : Bloomberg Business Week, June 2007.
    18. 18. The Bit Data: it may be big but its not clever
    19. 19. The Bit: Net Scale Fascination (delusion?) Data: it may be big but its not clever
    20. 20. • Hendrik‟s photo Source unknown – thanks to Hendrik Drachsler for passing it on.
    21. 21. Threat Rating=UBI*(I+D)UTS Where: UBI = Uncritical BI tradition factor I = Interventionist tendencies D = Big data fascination UTS = Uncritical technological solutionism This is not serious...
    22. 22. From: “Analytics in Higher Education: Benefits, Barriers, Progress, and Recommendations” (ECAR Report) Why am I Worried?
    23. 23. NEUTRALISING THREATS
    24. 24. Data Is Useless Without the Skills to Analyze It „...employees need to become: Ready and willing to experiment: Managers and business analysts must be able to apply the principles of scientific experimentation to their business. They must know how to construct intelligent hypotheses. They also need to understand the principles of experimental testing and design, including population selection and sampling, in order to evaluate the validity of data analyses. ... Adept at mathematical reasoning: How many of your managers today are really “numerate” — competent in the interpretation and use of numeric data?‟ Jeanne Harris, HBR Blog, Sept 2012
    25. 25. From: “Analytics in Higher Education: Benefits, Barriers, Progress, and Recommendations” (ECAR Report) Is This a Real Problem?
    26. 26. Make Some Progress By: • Directly investing in specialists • E.g. OU, statisticians and data wranglers • ECAR Survey Report 2012: “invest in people over tools... hiring and/or training... analysis” • Operating a “shared service” • Higher Education Statistics Agency (HESA) • Universities and Colleges Admissions Service (UCAS) • Jisc
    27. 27. Data Is Useless Without the Skills to Analyze It „...employees need to become: Ready and willing to experiment: Managers and business analysts must be able to apply the principles of scientific experimentation to their business. They must know how to construct intelligent hypotheses. They also need to understand the principles of experimental testing and design, including population selection and sampling, in order to evaluate the validity of data analyses. ... Adept at mathematical reasoning: How many of your managers today are really “numerate” — competent in the interpretation and use of numeric data?‟ Jeanne Harris, HBR Blog, Sept 2012
    28. 28. We Get Further with “Soft Capability” • Appreciation of the limitations of analytics technologies • Expertise to comprehend, critique, and converse (as opposed to originate or implement)
    29. 29. Visualisations are the Opium of the People (c)2008, Ryan Block/Engadget
    30. 30. Sparklines: Tufte vs ...
    31. 31. Significance and Sample Size “Flagged for action because 74% of students felt they were well supported by their supervisor compared to 80% in similar institutions (difference >5%)” “We need to talk to Mr. Benn; the drop-out rate doubled in his class this year.” Correlation is not Causation “The predictive model indicates that students with attributes {X, Y, Z} are only 50% likely to complete.”
    32. 32. What Does This Give Us? (aside from people less likely to cut themselves on sharp tools)
    33. 33. Absorptive Capacity “...the ability of a firm to recognize the value of new, external information, assimilate it, and apply it to commercial ends is critical to its innovative capabilities. We label this capability a firm‟s absorptive capacity and suggest that it is largely a function of the firm‟s level of prior related knowledge. [...] We formulate a model of firm investment in research and development (R&D), in which R&D contributes to a firm’s absorptive capacity, and test predictions relating a firm‟s investment in R&D to the knowledge underlying technical change within an industry.” Cohen and Levinthal, Admin. Science Quarterly 35(1), 1990
    34. 34. How to do LA as if learning mattered 1. Build soft capability. 2. Engage in participatory (collaborative) design. 3. Develop & prototype technology (& models) organically. 4. Reflect on the effect LA innovation has on practice. 5. Design and develop with the educator IN the system. 6. Optimise the environment, not only the outcome. 7. Be realistic about scale/quality of data and its meaning. Personal Learning Analytics? • Changed dyamic • Less “big brother”
    35. 35. Participatory Design Learning Analytics Image: cbn Jesse Millan
    36. 36. Thank-you for your attention These slides are at: http://is.gd/AaiLM
    37. 37. Reading and References Jacqueline Bichsel, “Analytics in Higher Education: Benefits, Barriers, Progress, and Recommendations” Available from http://www.educause.edu/ecar Cohen and Levinthal (1990), “Absorptive Capacity: A New Perspective on Learning and Innovation” Admin. Science Quarterly 35(1) http://faculty.fuqua.duke.edu/%7Echarlesw/s591/Bocconi-Duke/Papers/C10/CohenLevinthalASQ.pdf Adam Cooper, “What is Analytics? Definition and Essential Characteristics” http://publications.cetis.ac.uk/2012/521 Jeanne Harris, “Data is Useless Without the Skills to Analyze it” http://blogs.hbr.org/2012/09/data-is-useless-without-the-skills/ Bloomberg Business Week, “At 3M, A Struggle Between Efficiency And Creativity” http://www.businessweek.com/stories/2007-06-10/at-3m-a-struggle-between-efficiency-and-creativity Dai Griffiths, “CETIS Analytics Series: The impact of analytics in Higher Education on academic practice” http://publications.cetis.ac.uk/2012/532 Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism” ISBN-10: 9781610391382 Phil Winne, “Improving Education” (lecture) http://www.youtube.com/watch?v=dBxMZoMTIU4
    38. 38. Licence “Analytics (as if learning mattered)” by Adam Cooper, Cetis, is licensed under the Creative Commons Attribution 3.0 Unported Licence http://creativecommons.org/licenses/by/3.0/ Learning Analytics Community Exchange http://www.laceproject.info Case studies and white papers: Cetis Analytics Series, http://publications.cetis.ac.uk/c/analytics Me: http://blogs.cetis.ac.uk/adam a.r.cooper@bolton.ac.uk Cetis: http://www.cetis.ac.uk
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×