University of Technology, Sydney, Dec 2013

Learning Analytics
Should we be concerned about the
digital “quantification” o...
aim
leave with a better sense of the

analytics design space
+
better questions
than you can ask right now
2
AI &
Argumentation

KMi, OU

Learning
Technology

Semantic Scientific
Publishing
Dialogue / Issue /
Argument Mapping
Learn...
70-strong lab prototyping next generation
learning / collaboration / social media
analytics / future internet

4
The Hypermedia Discourse Group
Knowledge Media Institute, Open University:

http://kmi.open.ac.uk/projects/member/simon-bu...
make the invisible visible

Scholarship
and
Research

make the opaque permeable
make the ephemeral persistent

by using di...
make the invisible visible

Learning
Analytics

make the opaque permeable
make the ephemeral persistent

by using data abo...
From an analytics product review…

8
From an analytics product review…

“Some have tried to argue that
this technology doesn't work out
cost effectively when c...
Aquarium Analytics!

10
11
How is your aquatic ecosystem?
“This means that the keeper can be notified before water
conditions directly harm the fish—...
How is your learning ecosystem?
This means that the teacher can be notified before
learning conditions directly harm the s...
The rise of analytics…
NMC Horizon 2011 Report:
Learning Analytics
(4-5yrs adoption)

L. Johnson, R. Smith, H. Willis, A. ...
The rise of analytics…
Ed-Tech startups explosive growth

Audrey Waters: http://hackeducation.com/2012/11/19/top-ed-tech-t...
The rise of analytics…
LMS/VLEs + Analytics

Publishers + Analytics

16
Back to Aquarium Analytics…

17
Back to Aquarium Analytics…

fish
aquarium science

learners?
learning science
instructional design

18
Purdue University Signals: real time trafficlights for students based on predictive model

19
Purdue University Signals: real time trafficlights for students based on predictive model
MODEL:
•  ACT or SAT score
•  Ov...
Purdue University Signals: real time trafficlights for students based on predictive model

“Results thus far show that
stu...
Predictive analytics @open.edu
Demo-­‐
graphics	
  

VLE	
  
interac)on	
  

Registra)on	
  
Pa.ern	
  

Library	
  
inter...
Predictive analytics @open.edu
Adding in user interaction data from the VLE

Test a range of
predictive models:
final resu...
Hmmm…

no learning sciences/design
underpinning these predictive models of student success

models based on a mix of

inst...
the opportunity for the

learning sciences
to combine with your university’s

collective
intelligence
25
predictive models
are exciting
but there are many other
kinds of analytics
26
Analytics 101
Elaborated version of figure from Doug Clow:
h.p://www.slideshare.net/dougclow/the-­‐learning-­‐analy)cs-­‐c...
Analytics coming to a VLE near you:
Blackboard basic summary stats
http://www.blackboard.com/Platforms/Analytics/Products/...
Student Activity Dashboard (Erik Duval)

Duval E. (2011) Attention please!: learning analytics for visualization and recom...
Khan Academy has extended great instructional
movies with a tutoring platform with detailed analytics
http://www.youtube.c...
Adaptive platforms generate fine-grained
analytics on individuals’ curriculum mastery
https://grockit.com/research

31
Intelligent tutoring for skills mastery (CMU)
http://oli.cmu.edu

“In this study, results showed
that OLI-Statistics stude...
Track learner activity with a virtual machine
(Abelardo Pardo, LAK13 Conference Keynote)

33
http://www.slideshare.net/abe...
Track learner activity with a virtual machine
(Abelardo Pardo, LAK13 Conference Keynote)

http://www.slideshare.net/abelar...
macro
meso
micro
analytics
35
Macro/Meso/Micro Learning Analytics

League Tables

Macro:
region/state/national/international

Data
Interoperability
Init...
Macro/Meso/Micro Learning Analytics

Macro:
region/state/national/international
Meso:
institution-wide

Univ. Student
info...
Macro/Meso/Micro Learning Analytics

Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual u...
Hard distinctions between Learning +
Academic analytics may dissolve
…as they get joined up, each level enriches the other...
Hard distinctions between Learning +
Academic analytics may dissolve
…as they get joined up, each level enriches the other...
Analytics are not the end, but a means
The goal is to optimize the whole system

learners

Intent

curriculum
theories
ped...
Webinar replay on OU analytics strategy

Belinda Tynan & Simon Buckingham Shum (2013). Designing Systemic Learning Analyti...
Optimize the system
for what?

43
design analytics to achieve
your university’s strategic
goals
(increasingly differentiated as the sector
stratifies?)
44
analytics for building learners who handle…

• uncertainty
• novel dilemmas
• conflicting viewpoints
• VUCA life
in C21
US...
OECD DeSeCo Final Report
Definition & Selection of Key Competencies

“The OECD has collaborated
with a wide range of schol...
analytics for

social capital
47
Social Network Analysis (SNAPP)

What’s going on
in these discussion forums?

48
Social Network Analysis to gain insight into
peer-peer and peer-mentor dynamics
(SNAPP tool)

http://www.slideshare.net/an...
Social Network Analysis (SNAPP)
2 learners connect
otherwise separate
clusters

tutor only engaging
with active students,
...
Social Learning Analytics about to appear in
products…
http://www.desire2learn.com/products/analytics (this is from a beta...
Visualizing activity in OU Facebook sites
What do students say about their courses, and what are the
patterns of engagemen...
discourse analytics for

using language as
a knowledgebuilding tool
53
1st International Workshop on
Discourse-Centric Learning Analytics
solaresearch.org/events/lak/lak13/dcla13

analytics tha...
Discourse analytics on webinar
textchat

Can we spot the

quality learning
conversations in
a 2.5 hr webinar?
Ferguson, R....
Discourse analytics on webinar
textchat
Given a 2.5 hour webinar, where in the live
textchat were the most effective learn...
Discourse analytics on webinar
textchat
Given a 2.5 hour webinar, where in the live
textchat were the most effective learn...
Discourse analytics on webinar
textchat
Given a 2.5 hour webinar, where in the live
textchat were the most effective learn...
Discourse analytics on webinar
textchat
Visualizing by individual user. The gradient of the threshold line is
adjusted to ...
“Rhetorical parsing” to identify constructions
signifying scholarly writing
OPEN QUESTION:
“… little is known …”
“… role …...
“What are the key contributions of this text?

Human analyst

Computational analyst

http://technologies.kmi.open.ac.uk/co...
Social Learning Analytics

•  Explosive growth in social
media
•  The open/free content
paradigm
•  Evidence of a global s...
intrinsic motivation
self-regulation
resilience

63
Why do dispositions matter?
“Knowledge of methods alone
will not suffice: there must

be the desire, the will,
to employ t...
Why do dispositions matter?
“In the growth mindset, people believe
that their talents and abilities can be
developed throu...
Why do dispositions matter?
“We’re looking at the profiles of
what it means to be effective in the
21st century. […] Resil...
Why do dispositions matter?
“It’s more than knowledge and skills.

For the innovation economy,
dispositions come into play...
How can we model and

learning
dispositions in order
quantify

to develop analytics?
68
Dispositional
Learning
Analytics
Workshop

http://learningemergence.net/events/lasi-dla-wkshp

69
Validated as loading onto
7 dimensions of “Learning Power”
Ruth Deakin Crick
Grad. School of Education

Being Stuck & Stat...
Analytics for lifelong/lifewide
learning dispositions: ELLI

Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dis...
Learning to Learn: 7 Dimensions of Learning Power
Factor analysis of the literature plus expert interviews: identified sev...
Learning to Learn: 7 Dimensions of Learning Power

73
next step: platforms for

Dispositional Learning
Analytics

DLA Workshop: http://learningemergence.net/events/lasi-dla-wks...
Primary School EnquiryBloggers
Bushfield School, Wolverton, UK

EnquiryBlogger: blogging for Learning Power & Authentic En...
Masters level EnquiryBloggers
Graduate School of Education, University of Bristol

EnquiryBlogger: blogging for Learning P...
EnquiryBlogger
dashboard – direct
navigation to learner’s
blogs from the visual
analytic
Could a platform generate an
ELLI profile from user traces?
Questioning and
challenging may
load onto Critical
Curiosity

...
Envisioning a social learning analytics
dashboard

1

5
Your most
recent mood
comment:
“Great, at
last I have
found all th...
towards wholistic
learning analytics
Ruth Deakin Crick, Howard Green, Steven Barr
Towards wholistic analytics on the
health of a learning community

Academic results are important but less easily
quantifi...
Hierarchical Process Modelling
(Univ. Bristol PeriMeta tool)

Seeing a learning community as a complex adaptive
system req...
Hierarchical Process Modelling
(Univ. Bristol PeriMeta tool)

The core mission and values of
a network of school academies...
Hierarchical Process Modelling
(Univ. Bristol PeriMeta tool)
Hierarchical Process
Modelling transforms
qualitative and qua...
‘Italian Flag’ visual analytic

Degree of green / white / red reflects current
certainty over availability of evidence
sup...
‘Italian Flag’ model

Converting learner/leader/teachers’ linguistic survey
ratings and confidence into quantitative value...
‘Italian Flag’ visual analytic

Degree of green / white / red reflects current
certainty over availability of evidence
sup...
thorny issues
88
learning analytics
are

not neutral
89
Accounting tools are not neutral

“accounting tools...do not simply
aid the measurement of
economic activity, they shape t...
cf. Bowker and Starr’s “Sorting Things Out”
on classification schemes

“A marker of the health of the
learning analytics f...
Analytics are infused with human values
Data does not ‘speak for itself’
ethics

What kinds of learners?
What kinds of lea...
The Wal-Martification of education?

“data narrowness”
“instrumental learning”
“students with no curiosity”

“The basic qu...
Will staff know how to

read and write analytics?
This will become a key literacy.

94
Learning technology is not neutral
Any technology embodies
epistemological,
pedagogical and
assessment assumptions
epistem...
Algorithms are not neutral

http://governingalgorithms.org

Algorithms increasingly
shape our lives: we need to
go in eyes...
join the learning
analytics global
community
97
Join the community…

http://SoLAResearch.org
replays of all
previous
conference
presentations

http://LAKconference.org

9...
Join the community…

replays of all
sessions

http://www.solaresearch.org/events/lasi

99
JISC Briefings on Learning Analytics

http://publications.cetis.ac.uk/c/analytics

100
EDUCAUSE Briefings on Learning Analytics

http://www.educause.edu/library/learning-analytics

101
Learning Analytics Policy Brief
(UNESCO • IITE)

http://bit.ly/LearningAnalytics

102
Systems leadership and learning:
LearningEmergence.net
what this
all means
104
The big shifts that analytics could bring…
Organisational
Culture

Academic
Culture

evidence-based
decisions and
org lear...
The new research+practice vista…
data-culture
dynamics
how do HEIs manage
the embedding of
real time analytics
services?

...
Upcoming SlideShare
Loading in …5
×

Learning Analytics - UTS 2013

2,998 views

Published on

Learning Analytics: Should we be concerned about the digital “quantification” of learning?

University of Technology, Sydney, 10 Dec 2013

Published in: Education, Technology

Learning Analytics - UTS 2013

  1. 1. University of Technology, Sydney, Dec 2013 Learning Analytics Should we be concerned about the digital “quantification” of learning? Simon Buckingham Shum Knowledge Media Institute The Open University UK simon.buckinghamshum.net http://linkedin.com/in/simon @sbskmi #LearningAnalytics http://
  2. 2. aim leave with a better sense of the analytics design space + better questions than you can ask right now 2
  3. 3. AI & Argumentation KMi, OU Learning Technology Semantic Scientific Publishing Dialogue / Issue / Argument Mapping Learning Dispositions Learning Analytics Human-Centred Informatics Learning Analytics + (Contested) Collective Intelligence
  4. 4. 70-strong lab prototyping next generation learning / collaboration / social media analytics / future internet 4
  5. 5. The Hypermedia Discourse Group Knowledge Media Institute, Open University: http://kmi.open.ac.uk/projects/member/simon-buckingham-shum Funders span disciplines, from basic research to applications: 5
  6. 6. make the invisible visible Scholarship and Research make the opaque permeable make the ephemeral persistent by using digital tools to craft narrative around ideas + documents + multimedia 6
  7. 7. make the invisible visible Learning Analytics make the opaque permeable make the ephemeral persistent by using data about, and generated by learners, to visualize their progress 7
  8. 8. From an analytics product review… 8
  9. 9. From an analytics product review… “Some have tried to argue that this technology doesn't work out cost effectively when compared to conventional tests... but this misses a huge point. More often than not, we test after the event and discover the problem — but this is too late..” 9
  10. 10. Aquarium Analytics! 10
  11. 11. 11
  12. 12. How is your aquatic ecosystem? “This means that the keeper can be notified before water conditions directly harm the fish—an assured outcome of predictive software that lets you know if it looks like the pH is due to drop, or the temperature is on its way up. This way, it’s a real fish saver, as opposed to a forensic examiner, post-wipeout.” (From a review of Seneye, in a hobbyist magazine) 12
  13. 13. How is your learning ecosystem? This means that the teacher can be notified before learning conditions directly harm the students — an assured outcome of predictive software that lets you know if it looks like engagement is due to drop, or distraction is on its way up. This way, it’s a real student saver, as opposed to a forensic examiner, post-wipeout. 13
  14. 14. The rise of analytics… NMC Horizon 2011 Report: Learning Analytics (4-5yrs adoption) L. Johnson, R. Smith, H. Willis, A. Levine, and K. Haywood, The 2011 Horizon Report (Austin, TX: The New Media Consortium, 2011), http://www.nmc.org/pdf/2011-Horizon-Report.pdf 14
  15. 15. The rise of analytics… Ed-Tech startups explosive growth Audrey Waters: http://hackeducation.com/2012/11/19/top-ed-tech-trends-of-2012-the-business-of-ed-tech 15
  16. 16. The rise of analytics… LMS/VLEs + Analytics Publishers + Analytics 16
  17. 17. Back to Aquarium Analytics… 17
  18. 18. Back to Aquarium Analytics… fish aquarium science learners? learning science instructional design 18
  19. 19. Purdue University Signals: real time trafficlights for students based on predictive model 19
  20. 20. Purdue University Signals: real time trafficlights for students based on predictive model MODEL: •  ACT or SAT score •  Overall grade-point average •  CMS usage composite •  CMS assessment composite •  CMS assignment composite •  CMS calendar composite Predicted 66%-80% of struggling students who needed help Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40– 57. http://bit.ly/lmxG2x 20
  21. 21. Purdue University Signals: real time trafficlights for students based on predictive model “Results thus far show that students who have engaged with Course Signals have higher average grades and seek out help resources at a higher rate than other students.” Pistilli, M. D., Arnold, K. and Bethune, M., Signals: Using Academic Analytics to Promote Student Success. EDUCAUSE Review Online, July/Aug., (2012). http://www.educause.edu/ero/article/signals-using-academicanalytics-promote-student-success 21
  22. 22. Predictive analytics @open.edu Demo-­‐ graphics   VLE   interac)on   Registra)on   Pa.ern   Library   interac)on   CRM   contact   OpenLearn   interac)on   Grades   FutureLearn   interac)on   OU  history   Social  App  X   interac)on   ? How early can we predict likelihood of dropout, formal withdrawal, failure? Now exploring conventional statistics, machine learning and growing datasets
  23. 23. Predictive analytics @open.edu Adding in user interaction data from the VLE Test a range of predictive models: final result (pass/fail) final numerical score drop in the next TMA score of the next TMA Demographics Previous results VLE activity A.L. Wolff and Z. Zdrahal (2012). Improving Retention by Identifying and Supporting “At-risk” Students. EDUCAUSE Review Online, JulyAugust 2012. http://www.educause.edu/ero/article/improving-retention-identifying-and-supporting-risk-students
  24. 24. Hmmm… no learning sciences/design underpinning these predictive models of student success models based on a mix of institutional know-how about student success, and mining behavioural data 24
  25. 25. the opportunity for the learning sciences to combine with your university’s collective intelligence 25
  26. 26. predictive models are exciting but there are many other kinds of analytics 26
  27. 27. Analytics 101 Elaborated version of figure from Doug Clow: h.p://www.slideshare.net/dougclow/the-­‐learning-­‐analy)cs-­‐cycle-­‐closing-­‐the-­‐loop-­‐effec)vely  (slide  5) ethics What kinds of learners? What kinds of learning? What data could be generated digitally from the use context? What human +/or software interventions / recommendations? Does your theory predict patterns signifying learning? (you can invent future technologies if need) What analytical tools could be used to find such patterns? How to render the analytics, for whom, and will they understand them? 27
  28. 28. Analytics coming to a VLE near you: Blackboard basic summary stats http://www.blackboard.com/Platforms/Analytics/Products/Blackboard-Analytics-for-Learn.aspx 28
  29. 29. Student Activity Dashboard (Erik Duval) Duval E. (2011) Attention please!: learning analytics for visualization and recommendation. Proceedings of the 1st International Conference on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 9-17. 29
  30. 30. Khan Academy has extended great instructional movies with a tutoring platform with detailed analytics http://www.youtube.com/watch?v=DLt6mMQH1OY 30
  31. 31. Adaptive platforms generate fine-grained analytics on individuals’ curriculum mastery https://grockit.com/research 31
  32. 32. Intelligent tutoring for skills mastery (CMU) http://oli.cmu.edu “In this study, results showed that OLI-Statistics students [blended learning] learned a full semester’s worth of material in half as much time and performed as well or better than students learning from traditional instruction over a full semester.” Lovett M, Meyer O and Thille C. (2008) The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education 14. http://jime.open.ac.uk/article/2008-14/352
  33. 33. Track learner activity with a virtual machine (Abelardo Pardo, LAK13 Conference Keynote) 33 http://www.slideshare.net/abelardo_pardo/bridging-the-middle-space-with-learning-analytics
  34. 34. Track learner activity with a virtual machine (Abelardo Pardo, LAK13 Conference Keynote) http://www.slideshare.net/abelardo_pardo/bridging-the-middle-space-with-learning-analytics Calvo, R., O’Rourke, S.T., Jones, J., Yacef, K., Reimann, P., 2011. Collaborative Writing Support Tools on the Cloud. IEEE Transactions on Learning Technologies, 4(1):88–97 34
  35. 35. macro meso micro analytics 35
  36. 36. Macro/Meso/Micro Learning Analytics League Tables Macro: region/state/national/international Data Interoperability Initiatives
  37. 37. Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide Univ. Student info Systems Business Intelligence products to improve org processes
  38. 38. Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Learning Analytics
  39. 39. Hard distinctions between Learning + Academic analytics may dissolve …as they get joined up, each level enriches the others Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user traces enriches meso + macro analytics with finer-grained process data
  40. 40. Hard distinctions between Learning + Academic analytics may dissolve …as they get joined up, each level enriches the others Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user traces enriches meso + macro analytics with finer-grained process data Breadth + depth from macro + meso levels add power to micro analytics
  41. 41. Analytics are not the end, but a means The goal is to optimize the whole system learners Intent curriculum theories pedagogies assessments tools feedback design outcome Data intent researchers / educators / instructional designers 41
  42. 42. Webinar replay on OU analytics strategy Belinda Tynan & Simon Buckingham Shum (2013). Designing Systemic Learning Analytics at the Open University. SoLAR Open Course, Strategy & Policy for Systemic Learning Analytics, 11th Oct 2013. http://people.kmi.open.ac.uk/sbs/2013/10/designing-systemic-analytics-at-the-open-university
  43. 43. Optimize the system for what? 43
  44. 44. design analytics to achieve your university’s strategic goals (increasingly differentiated as the sector stratifies?) 44
  45. 45. analytics for building learners who handle… • uncertainty • novel dilemmas • conflicting viewpoints • VUCA life in C21 US military: Volatile/Uncertain/Complex/Ambiguous ? 45
  46. 46. OECD DeSeCo Final Report Definition & Selection of Key Competencies “The OECD has collaborated with a wide range of scholars, experts and institutions to identify a small set of key competencies that help individuals and whole societies to meet their goals.” http://www.deseco.admin.ch 46
  47. 47. analytics for social capital 47
  48. 48. Social Network Analysis (SNAPP) What’s going on in these discussion forums? 48
  49. 49. Social Network Analysis to gain insight into peer-peer and peer-mentor dynamics (SNAPP tool) http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 168-173. 49
  50. 50. Social Network Analysis (SNAPP) 2 learners connect otherwise separate clusters tutor only engaging with active students, ignoring disengaged ones on the edge 50 http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
  51. 51. Social Learning Analytics about to appear in products… http://www.desire2learn.com/products/analytics (this is from a beta demo) 51
  52. 52. Visualizing activity in OU Facebook sites What do students say about their courses, and what are the patterns of engagement in online communities?
  53. 53. discourse analytics for using language as a knowledgebuilding tool 53
  54. 54. 1st International Workshop on Discourse-Centric Learning Analytics solaresearch.org/events/lak/lak13/dcla13 analytics that look beneath the surface, and quantify linguistic proxies for ‘deeper learning’ http://www.glennsasscer.com/wordpress/wp-content/uploads/2011/10/iceberg.jpg Beyond number / size / frequency of posts; ‘hottest thread’
  55. 55. Discourse analytics on webinar textchat Can we spot the quality learning conversations in a 2.5 hr webinar? Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM
  56. 56. Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar… Sheffield, UK not as sunny as yesterday - still warm See you! bye for now! Greetings from Hong Kong bye, and thank you Morning from Wiltshire, 80 sunny here! Bye all for now 60 40 0 -20 9:28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 20 -40 Average Exploratory -60 Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
  57. 57. Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar but if we zoom in on a peak… 80 60 40 0 -20 9:28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 20 -40 Average Exploratory -60 Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
  58. 58. Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar but if we zoom in on a peak… Classified as “exploratory talk” (more substantive for learning) 100 0 -50 -100 9:28 9:40 9:50 10:00 10:07 10:17 10:31 10:45 11:04 11:17 11:26 11:32 11:38 11:44 11:52 12:03 50 “nonexploratory” Averag Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
  59. 59. Discourse analytics on webinar textchat Visualizing by individual user. The gradient of the threshold line is adjusted to every 5 posts in 6 classified as “Exploratory Talk” Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
  60. 60. “Rhetorical parsing” to identify constructions signifying scholarly writing OPEN QUESTION: “… little is known …” “… role … has been elusive” “Current data is insufficient …” SURPRISE: “We have recently observed ... surprisingly” “We have identified ... unusual” “The recent discovery ... suggests intriguing roles” CONTRASTING IDEAS: “… unorthodox view resolves …” “In contrast with previous hypotheses ...” “... inconsistent with past findings ...” http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052 Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391
  61. 61. “What are the key contributions of this text? Human analyst Computational analyst http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052 Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391
  62. 62. Social Learning Analytics •  Explosive growth in social media •  The open/free content paradigm •  Evidence of a global shift in societal attitudes which increasingly values participation •  Innovation depends on reciprocal social relationships, tacit knowing Buckingham Shum, Simon and Ferguson, Rebecca (2012). Social Learning Analytics. Journal of Educational Technology and Society, 15(3) pp. 3–26. http://oro.open.ac.uk/34092
  63. 63. intrinsic motivation self-regulation resilience 63
  64. 64. Why do dispositions matter? “Knowledge of methods alone will not suffice: there must be the desire, the will, to employ them. This desire is an affair of personal disposition.” John Dewey Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. Heath and Co, Boston, 1933 64
  65. 65. Why do dispositions matter? “In the growth mindset, people believe that their talents and abilities can be developed through passion, education, and persistence … It’s about a commitment to … taking informed risks … surrounding yourself with people who will challenge you to grow” Carol Dweck Interview with Carol Dweck: http://interviewscoertvisser.blogspot.co.uk/2007/11/interview-with-carol-dweck_4897.html 65
  66. 66. Why do dispositions matter? “We’re looking at the profiles of what it means to be effective in the 21st century. […] Resilience will be the defining concept. When challenged and bent, you learn and bounce back stronger.” “Dispositions are now at least as important as Knowledge and Skills. …They cannot be taught. They can only be cultivated.” John Seely Brown US Dept. of Educ. http://reimaginingeducation.org conference (May 28, 2013) Dispositions clip: http://www.c-spanvideo.org/clip/4457327 Whole talk: http://www.c-spanvideo.org/program/SecD 66
  67. 67. Why do dispositions matter? “It’s more than knowledge and skills. For the innovation economy, dispositions come into play: readiness to collaborate; attention to multiple perspectives; initiative; persistence; curiosity. The purpose of learning in the 21st century is not to recite inert knowledge but to transform it. It’s time to change the subject.” Larry Rosenstock LearningREimagined project: http://learning-reimagined.com Larry Rosenstock: http://audioboo.fm/boos/1669375-50-seconds-of-larry-rosenstock-ceo-of-hightechhigh-on-how-he-would-re-imagine-learning 67
  68. 68. How can we model and learning dispositions in order quantify to develop analytics? 68
  69. 69. Dispositional Learning Analytics Workshop http://learningemergence.net/events/lasi-dla-wkshp 69
  70. 70. Validated as loading onto 7 dimensions of “Learning Power” Ruth Deakin Crick Grad. School of Education Being Stuck & Static Changing & Learning Data Accumulation Meaning Making Passivity Critical Curiosity Being Rule Bound Isolation & Dependence Being Robotic Fragility & Dependence Creativity Learning Relationships Strategic Awareness Resilience
  71. 71. Analytics for lifelong/lifewide learning dispositions: ELLI Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823
  72. 72. Learning to Learn: 7 Dimensions of Learning Power Factor analysis of the literature plus expert interviews: identified seven dimensions of effective “learning power”, since validated empirically with learners at many levels. (Deakin Crick, Broadfoot and Claxton, 2004)
  73. 73. Learning to Learn: 7 Dimensions of Learning Power 73
  74. 74. next step: platforms for Dispositional Learning Analytics DLA Workshop: http://learningemergence.net/events/lasi-dla-wkshp 74
  75. 75. Primary School EnquiryBloggers Bushfield School, Wolverton, UK EnquiryBlogger: blogging for Learning Power & Authentic Enquiry http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry
  76. 76. Masters level EnquiryBloggers Graduate School of Education, University of Bristol EnquiryBlogger: blogging for Learning Power & Authentic Enquiry http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry
  77. 77. EnquiryBlogger dashboard – direct navigation to learner’s blogs from the visual analytic
  78. 78. Could a platform generate an ELLI profile from user traces? Questioning and challenging may load onto Critical Curiosity Sharing relevant resources from other contexts may load onto Meaning Making Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium Different social network patterns in different contexts may load onto Learning Relationships Repeated attempts to pass an online test may load onto Resilience
  79. 79. Envisioning a social learning analytics dashboard 1 5 Your most recent mood comment: “Great, at last I have found all the resources that I have been looking for, thanks to! Steve and Ellen.! 2 In your last discussion with your mentor, you decided to work on your resilience by taking on more learning challenges Your ELLI Spider shows that you have made a start on working on your resilience, and that you are also beginning to work on your creativity, which you identified as another area to work on. Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910 3 4
  80. 80. towards wholistic learning analytics Ruth Deakin Crick, Howard Green, Steven Barr
  81. 81. Towards wholistic analytics on the health of a learning community Academic results are important but less easily quantifiable measures of success are vital to many institutions’ vision to nurture • life-long/life-wide learners • employability skills • citizenship • self-confidence • teamwork • emotional wellbeing… 81
  82. 82. Hierarchical Process Modelling (Univ. Bristol PeriMeta tool) Seeing a learning community as a complex adaptive system requires the voices of learners, teachers, leaders, parents and ‘external’ stakeholders 82
  83. 83. Hierarchical Process Modelling (Univ. Bristol PeriMeta tool) The core mission and values of a network of school academies 83
  84. 84. Hierarchical Process Modelling (Univ. Bristol PeriMeta tool) Hierarchical Process Modelling transforms qualitative and quantitative inputs into a multi-level visual analytic 84
  85. 85. ‘Italian Flag’ visual analytic Degree of green / white / red reflects current certainty over availability of evidence supporting / unknown / challenging 85
  86. 86. ‘Italian Flag’ model Converting learner/leader/teachers’ linguistic survey ratings and confidence into quantitative values 86
  87. 87. ‘Italian Flag’ visual analytic Degree of green / white / red reflects current certainty over availability of evidence supporting / unknown / challenging 87
  88. 88. thorny issues 88
  89. 89. learning analytics are not neutral 89
  90. 90. Accounting tools are not neutral “accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure” Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life. Sage, London. pp. 12-13
  91. 91. cf. Bowker and Starr’s “Sorting Things Out” on classification schemes “A marker of the health of the learning analytics field will be the quality of debate around what the technology renders visible and leaves invisible.” Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, 2012, Vancouver, BC). ACM. Eprint: http://oro.open.ac.uk/32823
  92. 92. Analytics are infused with human values Data does not ‘speak for itself’ ethics What kinds of learners? What kinds of learning? What data could be generated digitally from the use context? What human +/or software interventions / recommendations? Does your theory predict patterns signifying learning? (you can invent future technologies if need) What analytical tools could be used to find such patterns? How to render the analytics, for whom, and will they understand them? 92
  93. 93. The Wal-Martification of education? “data narrowness” “instrumental learning” “students with no curiosity” “The basic question is not what can we measure? The basic question is what does a good education look like? Big questions. http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college http://lak12.wikispaces.com/Recordings 93
  94. 94. Will staff know how to read and write analytics? This will become a key literacy. 94
  95. 95. Learning technology is not neutral Any technology embodies epistemological, pedagogical and assessment assumptions epistemology assessment pedagogy Knight S, Buckingham Shum S and Littleton K. (2013) Epistemology, Pedagogy, Assessment and Learning Analytics. Proc. 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE: ACM, 75-84 Open Access Eprint: http://oro.open.ac.uk/36635 95
  96. 96. Algorithms are not neutral http://governingalgorithms.org Algorithms increasingly shape our lives: we need to go in eyes wide open, and encourage critical debate 96
  97. 97. join the learning analytics global community 97
  98. 98. Join the community… http://SoLAResearch.org replays of all previous conference presentations http://LAKconference.org 98
  99. 99. Join the community… replays of all sessions http://www.solaresearch.org/events/lasi 99
  100. 100. JISC Briefings on Learning Analytics http://publications.cetis.ac.uk/c/analytics 100
  101. 101. EDUCAUSE Briefings on Learning Analytics http://www.educause.edu/library/learning-analytics 101
  102. 102. Learning Analytics Policy Brief (UNESCO • IITE) http://bit.ly/LearningAnalytics 102
  103. 103. Systems leadership and learning: LearningEmergence.net
  104. 104. what this all means 104
  105. 105. The big shifts that analytics could bring… Organisational Culture Academic Culture evidence-based decisions and org learning data-intensive learning sciences/ educ research Practitioner Culture C21 Qualities evidence impact of learning designs; timely interventions place these on a firm empirical evidence base 105
  106. 106. The new research+practice vista… data-culture dynamics how do HEIs manage the embedding of real time analytics services? educator data literacy how do staff learn to read and write analytics? sensemaking meets computation creative intelligence + computational thinking pedagogical innovation how do learning analytics change student experience? 106

×