SlideShare a Scribd company logo
1 of 68
Download to read offline
Designing Systemic Learning Analytics
at the Open University
Simon Buckingham Shum
Knowledge Media Institute
The Open University, UK
Strategy & Policy for Systemic Learning Analytics
SoLAR Open Course, 11th Oct 2013
https://learn.canvas.net/courses/182/wiki/designing-systemic-analytics-at-the-open-university
Belinda Tynan
Pro-Vice-Chancellor Learning & Teaching
The Open University, UK
overview
Belinda: developing an institutional strategy
— framework and implementation
Simon: research perspectives
— 3 metaphors for systemic analytics
Discussion
Learning and Teaching
Strategy for Systemic Deployment of
Analytics at the Open University
Belinda Tynan, Pro-Vice-Chancellor, Learning and Teaching
Kevin Mayles, Senior Manager, Learning and Teaching
Learning and Teaching
Analytics for student success vision
p.4
A clear vision has been developed to galvanise effort across the institution
on the focused use of analytics to drive student success
Vision	
  
To	
  use	
  and	
  apply	
  informa2on	
  strategically	
  (through	
  specified	
  indicators)	
  to	
  
retain	
  students	
  and	
  progress	
  them	
  to	
  complete	
  their	
  study	
  goals	
  
	
  
Mission	
  
This	
  needs	
  to	
  be	
  achieved	
  at	
  :	
  
•  a	
  macro	
  level	
  to	
  aggregate	
  informa5on	
  about	
  the	
  student	
  learning	
  
experience	
  at	
  an	
  ins5tu5onal	
  level	
  to	
  inform	
  strategic	
  priori5es	
  that	
  will	
  
improve	
  student	
  reten5on	
  and	
  progression	
  
•  a	
  micro	
  level	
  to	
  use	
  analy5cs	
  to	
  drive	
  short,	
  medium	
  and	
  long-­‐term	
  
interven5ons	
  
Learning and Teaching
What drives student success?
p.5
We have an emerging picture of the factors affecting student success
based on existing statistical analyses, literature and “institutional
knowledge” and our current use of associated indicators
Framework adapted from Woodley et. al. (2001) Student Progress in Distance Education:
Kember’s model re-visited
• Early	
  contact	
  
• Early	
  engagement	
  
• Study	
  calendar	
  /	
  scheduling	
  
• Tutor	
  support	
  
• Peer	
  support	
  &	
  belonging	
  
• Study	
  habits	
  
• Employer	
  support	
  
• Family	
  support	
  
• Personal	
  	
  life	
  events	
  
• Financial	
  issues	
  
• Advice	
  on	
  course	
  choice	
  
• Subject	
  studied	
  
• Prepara5on	
  for	
  study	
  
• Learning	
  design	
  
• Quality	
  of	
  study	
  materials	
  
• Workload	
  
• Module	
  assessment	
  
strategy	
  
• Language	
  ability	
  
• Previous	
  educa5on	
  /	
  OU	
  study	
  
• Ethnic	
  group	
  
• Socio-­‐economic	
  background	
  
• Disability	
  
• Age	
  
• Study	
  goal	
  /	
  mo5va5on	
  
• Gender	
   Entry	
  
characteris-cs	
  
Academic	
  
compa-bility	
  
Social	
  and	
  
academic	
  
integra-on	
  
External	
  factors	
  
Student	
  
Success	
  
Indicators	
  used	
  in	
  exis-ng	
  analy-cs	
  
There	
  are	
  a	
  number	
  of	
  indicators	
  with	
  
suppor5ng	
  evidence	
  that	
  we	
  currently	
  
use	
  in	
  our	
  analysis	
  models	
  
Clear	
  evidence	
  of	
  impact	
  but	
  currently	
  
not	
  used	
  in	
  analy-cs	
  
We	
  have	
  a	
  number	
  of	
  factors	
  for	
  which	
  
there	
  is	
  clear	
  evidence	
  of	
  the	
  impact	
  on	
  
success	
  but	
  are	
  not	
  being	
  used	
  in	
  current	
  
analy5cs	
  models	
  due	
  to	
  lack	
  of	
  data	
  or	
  
insufficient	
  inves5ga5on	
  
Unclear	
  evidence	
  base	
  
There	
  are	
  a	
  number	
  of	
  factors	
  that	
  the	
  
OU	
  believes	
  or	
  literature	
  suggests	
  have	
  
an	
  impact	
  on	
  student	
  success	
  but	
  where	
  
we	
  have	
  no	
  clear	
  evidence	
  at	
  this	
  5me	
  
due	
  either	
  to	
  lack	
  of	
  data	
  availability	
  or	
  
insufficient	
  inves5ga5on	
  
Results	
  from	
  a	
  review	
  of	
  exis-ng	
  evidence	
  on	
  
the	
  drivers	
  of	
  student	
  success	
  are	
  giving	
  us	
  a	
  
mixed	
  picture	
  
Indicators	
  with	
  evidence	
  of	
  no	
  impact	
  
There	
  are	
  a	
  number	
  of	
  indicators	
  with	
  
suppor5ng	
  evidence	
  that	
  suggest	
  they	
  
have	
  a	
  minimal	
  impact	
  on	
  success	
  
Learning and Teaching
Developing institutional capabilities and strengths
3 year strategic roadmap
The OU is developing its capabilities in 10 key areas that build the
underpinning strengths required for the effective deployment of analytics
We	
  need	
  to	
  ensure	
  we	
  have	
  the	
  right	
  architecture	
  and	
  processes	
  
for	
  collec5ng	
  the	
  right	
  data	
  and	
  making	
  it	
  accessible	
  for	
  analy5cs	
  
–	
  we	
  need	
  a	
  ‘big	
  data’	
  mind-­‐set	
  
The	
  university	
  
needs	
  world	
  class	
  
capability	
  in	
  data	
  
science	
  to	
  
con5nually	
  mine	
  
the	
  data	
  and	
  build	
  
rapid	
  prototypes	
  of	
  
simple	
  tools,	
  and	
  a	
  
clear	
  pipeline	
  for	
  
the	
  outputs	
  to	
  be	
  
mainstreamed	
  into	
  
opera5ons	
  
Benefits	
  will	
  be	
  realised	
  through	
  
exis5ng	
  business	
  processes	
  
impac5ng	
  on	
  students	
  directly	
  
and	
  through	
  enhancement	
  of	
  
the	
  student	
  learning	
  experience	
  
–	
  we	
  will	
  develop	
  an	
  ‘analy5cs	
  
mind-­‐set’	
  in	
  	
  
these	
  areas	
  
p.6
Learning and Teaching
Framework for harnessing analytics for student
success through driving interventions
p.7
Analytics will be applied throughout the cycle of the student learning
experience
Analy-cs	
  applied…	
   Example	
  business	
  processes	
   Example	
  datasets	
  used	
  
For	
  ac-on	
  
Op5mise	
  student	
  alloca5on	
  to	
  tutor	
  groups	
  
	
  
Development	
  of	
  learning	
  systems	
  
Assessment	
  strategy	
  and	
  scheduling	
  
Student	
  pass/fail	
  
predic5ons	
  
Study	
  behaviour	
  profiles	
  
Pass	
  rates	
  modelling	
  
In	
  ac-on	
  
Early	
  contact	
  with	
  ‘at	
  risk’	
  students	
  
	
  
Module	
  presenta5on	
  issue	
  flagging	
  
Student	
  ‘at	
  risk’	
  predic5ve	
  
indicators	
  
Helpdesk	
  contact	
  records	
  
On	
  ac-on	
  
Annual	
  module	
  and	
  programme	
  review	
  
Learning	
  design	
  
Module	
  performance	
  KPIs	
  
Learning	
  design	
  profiles	
  
“In Action, On Action” from Donald Schön The Reflective Practitioner
Learning and Teaching
Applying ‘in action’ analytics to drive student success?
p.8
We will use analytics to put key information relating to student success in the hands
of those in a position to take action
Tutor	
  Group	
  List	
  
Students’	
  study	
  history	
  
Feedback	
  from	
  previous	
  tutors	
  
Predicted	
  probability	
  of	
  passing	
  
‘At	
  risk’	
  factors	
  
Associate	
  Lecturer	
  
Plan	
  early	
  contact	
  with	
  most	
  at	
  risk	
  students	
  
Monitor	
  engagement	
  prior	
  to	
  first	
  TMA	
  
Refer	
  issues	
  to	
  SST	
  quickly	
  	
  
Weekly	
  Alert	
  Dashboard	
  
Weekly	
  update	
  of	
  students’	
  predicted	
  
probability	
  of	
  passing	
  /	
  progressing	
  
List	
  of	
  most	
  ‘at	
  risk’	
  students	
  this	
  week	
  
Target	
  resources	
  at	
  most	
  at	
  risk	
  students	
  
Call	
  or	
  email	
  students	
  on	
  at	
  risk	
  list	
  to	
  offer	
  
support	
  
No5fy	
  ALs	
  of	
  any	
  issues	
  arising	
  in	
  their	
  groups	
  
Student	
  Support	
  Team	
  
Module	
  performance	
  report	
  
Predicted	
  pass	
  rate	
  vs	
  target	
  pass	
  rate	
  
updated	
  during	
  presenta5on	
  
Analysis	
  of	
  online	
  learning	
  ac5vity	
  
usage	
  /	
  engagement	
  pa^erns	
  
Iden5fy	
  any	
  issues	
  with	
  the	
  module	
  whilst	
  in	
  
presenta5on	
  and	
  take	
  ac5on	
  to	
  rec5fy	
  
Evaluate	
  the	
  use	
  of	
  learning	
  assets	
  to	
  inform	
  
future	
  produc5on	
  
Faculty	
  Academics	
  
Senior	
  Management	
  
Student	
  success	
  measures	
  
Indicators	
  derived	
  from	
  
sta5s5cal	
  modelling	
  that	
  
underpin	
  student	
  
progression	
  measures	
  
Monitor	
  student	
  progression	
  forecasts	
  against	
  
target	
  –	
  iden5fy	
  correc5ve	
  ac5on	
  
Target	
  resources	
  at	
  specific	
  ‘pinch	
  points’	
  in	
  the	
  
student	
  journey	
  
Learning and Teaching
Evaluating impact and driving action
p.9
The basis for evaluation needs to link interventions to measurable
outcomes of student success
Learning and Teaching
Evaluating impact and driving action
p.10
The basis for evaluation needs to link interventions to measurable
outcomes of student success
STUDENT	
  
SUCCESS	
  
Learning and Teaching
Evaluating impact and driving action
p.11
The basis for evaluation needs to link interventions to measurable
outcomes of student success
STUDENT	
  
SUCCESS	
  
Interven-ons	
  
For	
  ac5on	
  
In	
  ac5on	
  
On	
  ac5on	
  
Learning and Teaching
Evaluating impact and driving action
p.12
The basis for evaluation needs to link interventions to measurable
outcomes of student success
Governance	
  
and	
  
Management	
  
STUDENT	
  
SUCCESS	
  
Interven-ons	
  
For	
  ac5on	
  
In	
  ac5on	
  
On	
  ac5on	
  
Learning and Teaching
Evaluating impact and driving action
p.13
The basis for evaluation needs to link interventions to measurable
outcomes of student success
Drivers	
  of	
  
student	
  success	
  Governance	
  
and	
  
Management	
  
STUDENT	
  
SUCCESS	
  
Interven-ons	
  
For	
  ac5on	
  
In	
  ac5on	
  
On	
  ac5on	
  
Learning and Teaching
Improve	
  ins-tu-onal	
  capabili-es	
  and	
  
processes	
  
Evaluating impact and driving action
p.14
The basis for evaluation needs to link interventions to measurable
outcomes of student success
Drivers	
  of	
  
student	
  success	
  Governance	
  
and	
  
Management	
  
STUDENT	
  
SUCCESS	
  
Interven-ons	
  
For	
  ac5on	
  
In	
  ac5on	
  
On	
  ac5on	
  
Learning and Teaching
Improve	
  ins-tu-onal	
  capabili-es	
  and	
  
processes	
  
Evaluating impact and driving action
p.15
The basis for evaluation needs to link interventions to measurable
outcomes of student success
Drivers	
  of	
  
student	
  success	
  Governance	
  
and	
  
Management	
  
STUDENT	
  
SUCCESS	
  
Interven-ons	
  
For	
  ac5on	
  
In	
  ac5on	
  
On	
  ac5on	
  
Are	
  we	
  seeing	
  
expected	
  
outcomes	
  of	
  
our	
  
interven5ons?	
  
Are	
  we	
  doing	
  
the	
  right	
  things	
  
as	
  guided	
  by	
  the	
  
evidence?	
  
Analytics will drive action to increase student success
Vision: To use and apply information strategically (through specified indicators) to
retain students and progress them to complete their study goals
Analytics will drive action to increase student success
Vision: To use and apply information strategically (through specified indicators) to
retain students and progress them to complete their study goals
Recruit Retain Progress Complete
Success	
  outcomes	
  
and	
  leading	
  
indicators	
  
Analytics will drive action to increase student success
Vision: To use and apply information strategically (through specified indicators) to
retain students and progress them to complete their study goals
Recruit Retain Progress Complete
Success	
  outcomes	
  
and	
  leading	
  
indicators	
  
Student	
  support	
  
ac5vi5es	
  
Learning	
  &	
  teaching	
  
ac5vi5es	
  
Measures	
  of	
  our	
  
opera5onal	
  
performance	
  and	
  
interven5ons	
  
Drivers	
  of	
  student	
  success	
  
Evidence	
  of	
  the	
  
drivers	
  of	
  student	
  
success	
  guides	
  
what	
  we	
  do	
  and	
  
what	
  we	
  measure	
  
Analytics will drive action to increase student success
Vision: To use and apply information strategically (through specified indicators) to
retain students and progress them to complete their study goals
Recruit Retain Progress Complete
Success	
  outcomes	
  
and	
  leading	
  
indicators	
  
Student	
  support	
  
ac5vi5es	
  
Learning	
  &	
  teaching	
  
ac5vi5es	
  
Measures	
  of	
  our	
  
opera5onal	
  
performance	
  and	
  
interven5ons	
  
Dashboards	
  /	
  
Reports	
  /	
  Tools	
  
Ins5tu5onal	
  
Dashboard	
  
PVCs	
  
Deans	
  
Programme	
  
Directors	
  
Module	
  Teams	
  
Student	
  Support	
  
Teams	
  
Indicators
and
measures
fed into
dashboards
and reports
at relevant
levels
Drivers	
  of	
  student	
  success	
  
Evidence	
  of	
  the	
  
drivers	
  of	
  student	
  
success	
  guides	
  
what	
  we	
  do	
  and	
  
what	
  we	
  measure	
  
Analytics will drive action to increase student success
Vision: To use and apply information strategically (through specified indicators) to
retain students and progress them to complete their study goals
Recruit Retain Progress Complete
Success	
  outcomes	
  
and	
  leading	
  
indicators	
  
Student	
  support	
  
ac5vi5es	
  
Learning	
  &	
  teaching	
  
ac5vi5es	
  
Measures	
  of	
  our	
  
opera5onal	
  
performance	
  and	
  
interven5ons	
  
Dashboards	
  /	
  
Reports	
  /	
  Tools	
  
Ins5tu5onal	
  
Dashboard	
  
PVCs	
  
Deans	
  
Programme	
  
Directors	
  
Module	
  Teams	
  
Student	
  Support	
  
Teams	
  
Indicators
and
measures
fed into
dashboards
and reports
at relevant
levels
Drivers	
  of	
  student	
  success	
  
Evidence	
  of	
  the	
  
drivers	
  of	
  student	
  
success	
  guides	
  
what	
  we	
  do	
  and	
  
what	
  we	
  measure	
  
ACTION	
  
Interven-on	
  
Analytics will drive action to increase student success
Vision: To use and apply information strategically (through specified indicators) to
retain students and progress them to complete their study goals
Recruit Retain Progress Complete
Success	
  outcomes	
  
and	
  leading	
  
indicators	
  
Student	
  support	
  
ac5vi5es	
  
Learning	
  &	
  teaching	
  
ac5vi5es	
  
Measures	
  of	
  our	
  
opera5onal	
  
performance	
  and	
  
interven5ons	
  
Dashboards	
  /	
  
Reports	
  /	
  Tools	
  
Ins5tu5onal	
  
Dashboard	
  
PVCs	
  
Deans	
  
Programme	
  
Directors	
  
Module	
  Teams	
  
Student	
  Support	
  
Teams	
  
Indicators
and
measures
fed into
dashboards
and reports
at relevant
levels
Drivers	
  of	
  student	
  success	
  
Evidence	
  of	
  the	
  
drivers	
  of	
  student	
  
success	
  guides	
  
what	
  we	
  do	
  and	
  
what	
  we	
  measure	
  
ACTION	
  
Interven-on	
  
Evalua-on	
  of	
  the	
  outcomes	
  from	
  interven5ons	
  increases	
  our	
  
evidence	
  base	
  of	
  what	
  drives	
  student	
  success	
  
questions/comments?
3 metaphors for
systemic analytics
1. the aquarium
2. from exoskeleton to nervous system
3. resilience through biodiversity
metaphor 1
the aquarium
systems strategy: research the key variables for
a healthy ecosystem and evolve predictive
models as rapidly as possible to detect variance
Aquarium science enables aquarium analytics
which monitor the health of the ecosystem
fish
aquarium science
learners?
learning science
instructional design
It’s all about knowing what to watch
Purdue University Signals: exemplar
‘healthy ecosystem’ variables
27
Purdue University Signals: exemplar
‘healthy ecosystem’ variables
28
Key variables identified:
•  ACT or SAT score
•  Overall grade-point average
•  CMS usage composite
•  CMS assessment composite
•  CMS assignment composite
•  CMS calendar composite
Hmmm…
no learning sciences
no learning design
underpinning these predictive models of student success
models based on a mix of
institutional know-how
about student success, and analysing
behavioural data
29
the opportunity for the
learning sciences
to combine with your university’s
collective intelligence
30
Predictive modelling of student outcomes
Registra-on	
  
PaMern	
  
CRM	
  
contact	
  
VLE	
  
interac-on	
  
Assignment	
  
grades	
  
Demo-­‐
graphics	
  
?
Can we combine datasets,
and use machine learning to
build models to identify
‘signature’ patterns associated
with different kinds of
students?
Library	
  
interac-on	
  
OpenLearn	
  
interac-on	
  
FutureLearn	
  
interac-on	
  
App	
  X	
  
interac-on	
  
OU	
  track	
  
record	
  
Predictive modelling of student outcomes
Registra-on	
  
PaMern	
  
CRM	
  
contact	
  
VLE	
  
interac-on	
  
Assignment	
  
grades	
  
Demo-­‐
graphics	
  
?
Does VLE data carry
information that provides more
precise early identification of
failing students than is
currently possible?
Simple example (just 3 demographic attributes and VLE):
Input:
Demographic data: New/Continuing student,
Sex, Previous education
VLE interactions without qualifying the type
(any click counts)
Goal:
Evaluate the probability that the student
does not submit TMA1 or submits and
scores lower than 40.
Method: Naïve Bayes network (e.g. see Bishop,
2009)
Library	
  
interac-on	
  
OpenLearn	
  
interac-on	
  
FutureLearn	
  
interac-on	
  
App	
  X	
  
interac-on	
  
OU	
  track	
  
record	
  
Learning and Teaching
Back to the OU’s analytics framework
Learning and Teaching
Predictive modelling within the framework
VLE	
  user	
  trace	
  data	
  /	
  student	
  demographics	
  /	
  
academic	
  achievement	
  
Strategic internal funding to advance a promising technique from an externally
funded (JISC) project, and embed within OU student support processes:
A.L. Wolff and Z. Zdrahal (2012). Improving Retention by Identifying and Supporting “At-risk” Students. EDUCAUSE Review
Online, July-August 2012. http://www.educause.edu/ero/article/improving-retention-identifying-and-supporting-risk-students
Zdenek Zdrahal
Lead, KMi Predictive Modelling Team
http://kmi.open.ac.uk/people/member/zdenek-zdrahal
Learning and Teaching
Predictive modelling within the framework
Develop	
  and	
  Validate	
  
Predic-ve	
  Models	
  of	
  
student	
  success	
  
(module	
  comple-on)	
  
in	
  order	
  to	
  trigger	
  
more	
  -mely	
  alerts	
  
VLE	
  user	
  trace	
  data	
  /	
  student	
  demographics	
  /	
  
academic	
  achievement	
  
Learning and Teaching
Predictive modelling within the framework
Develop	
  and	
  Validate	
  
Predic-ve	
  Models	
  of	
  
student	
  success	
  
(module	
  comple-on)	
  
in	
  order	
  to	
  trigger	
  
more	
  -mely	
  alerts	
  
Requirements	
  
to	
  mainstream	
  
the	
  models	
  in	
  
the	
  VLE	
  
VLE	
  user	
  trace	
  data	
  /	
  student	
  demographics	
  /	
  
academic	
  achievement	
  
Learning and Teaching
Predictive modelling within the framework
Develop	
  and	
  Validate	
  
Predic-ve	
  Models	
  of	
  
student	
  success	
  
(module	
  comple-on)	
  
in	
  order	
  to	
  trigger	
  
more	
  -mely	
  alerts	
  
Prototype	
  Student	
  
Support	
  Team	
  
dashboards	
  
Requirements	
  
to	
  mainstream	
  
the	
  models	
  in	
  
the	
  VLE	
  
VLE	
  user	
  trace	
  data	
  /	
  student	
  demographics	
  /	
  
academic	
  achievement	
  
Learning and Teaching
Predictive modelling within the framework
Develop	
  and	
  Validate	
  
Predic-ve	
  Models	
  of	
  
student	
  success	
  
(module	
  comple-on)	
  
in	
  order	
  to	
  trigger	
  
more	
  -mely	
  alerts	
  
New	
  tool	
  for	
  Student	
  Support	
  Teams,	
  to	
  enable	
  -mely	
  
interven-on.	
  E.g.	
  real-­‐-me	
  traffic	
  lights	
  on	
  at	
  risk	
  students	
  
Requirements	
  
to	
  mainstream	
  
the	
  models	
  in	
  
the	
  VLE	
  
VLE	
  user	
  trace	
  data	
  /	
  student	
  demographics	
  /	
  
academic	
  achievement	
  
Prototype	
  Student	
  
Support	
  Team	
  
dashboards	
  
Learning and Teaching
Predictive modelling within the framework
Develop	
  and	
  Validate	
  
Predic-ve	
  Models	
  of	
  
student	
  success	
  
(module	
  comple-on)	
  
in	
  order	
  to	
  trigger	
  
more	
  -mely	
  alerts	
  
New	
  tool	
  for	
  Student	
  Support	
  Teams,	
  to	
  enable	
  -mely	
  
interven-on.	
  E.g.	
  real-­‐-me	
  traffic	
  lights	
  on	
  at	
  risk	
  students	
  
Modules	
  are	
  accompanied	
  by	
  
machine-­‐readable	
  metadata	
  that	
  
increases	
  the	
  power	
  of	
  machine	
  
learning	
  when	
  it	
  comes	
  to	
  data	
  
analysis	
  
Requirements	
  
to	
  mainstream	
  
the	
  models	
  in	
  
the	
  VLE	
  
VLE	
  user	
  trace	
  data	
  /	
  student	
  demographics	
  /	
  
academic	
  achievement	
  
Prototype	
  Student	
  
Support	
  Team	
  
dashboards	
  
Problem specification
Learning to dynamically id an at-risk student
We are hereWe know
Problem specification
Learning to dynamically id an at-risk student
We are hereWe know We predict
e.g. The Retain project
Does VLE activity add value to predictive models?
We know We predict
Sex
Educ
New/
Cont
VLE
Model the probability of failing at
TMA1 which is known to be a key
predictor of final outcome
either by not submitting TMA1,
or by submitting with score < 40.
TMA1
•  Demographic	
  profile	
  1	
  
–  X	
  
–  Y	
  
–  Z	
  	
  
Sex	
  
Educa5on	
  
N/C	
  
TMA1	
  
Without	
  VLE	
  data:	
  
Probability	
  of	
  failing	
  at	
  TMA1	
  	
  =	
  18.5%	
  
Student profile 1
Student profile 1	
  
•  Demographic	
  profile	
  1	
  
–  X	
  
–  Y	
  
–  Z	
  	
  
Sex	
  
Educa5on	
  
N/C	
  
TMA1	
  
Without	
  VLE	
  data:	
  
Probability	
  of	
  failing	
  at	
  TMA1	
  	
  =	
  18.5%	
  
Sex	
  
Educa5on	
  
N/C	
  
VLE	
  
TMA1	
  
Clicks	
   Probability	
   Nr	
  of	
  students	
  
0	
   64%	
   4	
  
1-­‐20	
   44%	
   3	
  
21-­‐100	
   26%	
   5	
  
101-­‐800	
   6.3%	
   14	
  
With	
  VLE	
  data,	
  a	
  higher	
  fidelity	
  story:	
  
Student profile 2	
  
Sex	
  
Educa5on	
  
N/C	
  
TMA1	
  
Without	
  VLE	
  data:	
  
Probability	
  of	
  failing	
  at	
  TMA1	
  	
  =	
  7.7%	
  
•  Demographic	
  profile	
  2	
  
–  X	
  
–  Y	
  
–  Z	
  	
  
Student profile 2	
  
Sex	
  
Educa5on	
  
N/C	
  
TMA1	
  
Without	
  VLE	
  data:	
  
Probability	
  of	
  failing	
  at	
  TMA1	
  	
  =	
  7.7%	
  
Sex	
  
Educa5on	
  
N/C	
  
VLE	
  
TMA1	
  
Clicks	
   Probability	
   Nr	
  of	
  students	
  
0	
   39%	
   35	
  
1-­‐20	
   22%	
   74	
  
21-­‐100	
   11.2%	
   178	
  
101-­‐800	
   2.4%	
   461	
  
With	
  VLE	
  data,	
  a	
  higher	
  fidelity	
  story:	
  
•  Demographic	
  profile	
  2	
  
–  X	
  
–  Y	
  
–  Z	
  	
  
Potential to augment student support teams
with predictive modelling once validated
Query
VLE	
  
interac-on	
  
Assignment	
  
grades	
  
Demo-­‐
graphics	
  
OU	
  track	
  
record	
  
Potential to augment student support teams
with predictive modelling once validated
7 of your students have fail trajectory
BUT prioritize Nigel, then Sue, then Ian
because
- has not engaged with VLE
- at least one TMA below 40
- has not submitted 5 TMAs
Query
VLE	
  
interac-on	
  
Assignment	
  
grades	
  
Demo-­‐
graphics	
  
OU	
  track	
  
record	
  
Why do I need a variable ML approach?
Can’t I just use one method (off the shelf)?
Registra5on	
  
Pa^ern	
  
CRM	
  
interac5ons	
  
Library	
  
interac5on	
  
FutureLearn	
  
interac5on	
  
Train	
  and	
  Learn	
  	
  
as	
  new	
  data	
  is	
  added	
  
using	
  variable	
  methods	
  
Methods successfully tested, to be further developed:
•  Induction of decision tree (ID3, C4.5 from the Weka toolkit)
•  Support Vector Machine (from Weka)
•  Bayes network (Microsoft Infer.NET; SamIam - Stanford Univ.)
•  Naïve Bayes (see the example and Demo Cases)
•  Linear regression
•  Logistic regression
•  GUHA (General Unary Hypotheses Automaton)
metaphor 2
from exoskeleton
to nervous system
systems strategy: embed faster feedback loops,
and build sensemaking capacity at all levels
Evolving the OU from a digital
exoskeleton to a nervous system?
Ed Dumbill: http://strata.oreilly.com/2012/08/digital-nervous-system-big-data.html
Learning and Teaching
The OU’s collective intelligence
Macro Level Analytics
Towards multilevel systemic analytics for student success
Designing better feedback loops at all levels of learning (students + staff)
Micro Level
Analytics
Student Interaction Traces
Learning and Teaching
The OU’s collective intelligence
Macro Level Analytics
Towards multilevel systemic analytics for student success
Designing better feedback loops at all levels of learning (students + staff)
Micro Level
Analytics
Student Interaction Traces
Student Support Teams
Associate Lecturers
Researchers
interpretation/intervention
Learning and Teaching
The OU’s collective intelligence
Macro Level Analytics
Towards multilevel systemic analytics for student success
Designing better feedback loops at all levels of learning (students + staff)
Micro Level
Analytics
Student Interaction Traces
Student Support Teams
Associate Lecturers
Researchers
interpretation/intervention
Data Wranglers
Researchers
Learning and Teaching
The OU’s collective intelligence
Macro Level Analytics
Towards multilevel systemic analytics for student success
Designing better feedback loops at all levels of learning (students + staff)
Micro Level
Analytics
Student Interaction Traces
VC Executive
Faculties
Module Teams
Student Support Teams
Associate Lecturers
Researchers
interpretation/intervention
Data Wranglers
Researchers
Learning and Teaching
The OU’s collective intelligence
Macro Level Analytics
Towards multilevel systemic analytics for student success
Designing better feedback loops at all levels of learning (students + staff)
Micro Level
Analytics
Student Interaction Traces
VC Executive
Faculties
Module Teams
Student Support Teams
Associate Lecturers
Researchers
interpretation/intervention
Data Wranglers
Researchers
Visual Analytics Design
Quality Data, Integrated
Analytics Competency Team
Organisational Capacity building
Analytics Research
Computational Platforms
metaphor 3
build resilience
systems strategy: MOOCs can be viewed as a system-
level ‘shock’ to the HigherEd ecology (‘regime shift’?)
build resilience by expanding our diversity and capacity
to sense the dynamic environment
MOOCs are an innovation and research
platform — analytics will be critical
http://www.ted.com/talks/daphne_koller_what_we_re_learning_from_online_education.html
http://people.kmi.open.ac.uk/sbs/2013/01/emerging-mooc-data-analytics-ecosystem
http://www.slideshare.net/abelardo_pardo/pushing-the-mooc-envelope-with-learning-analytics
http://www.moocresearch.com/research-initiative/about#Cost,%20Performance%20Metrics%20and%20Learner%20Analytics
FLx: experimental analytics ecosystem
Partners share ideas, workflows, analytics, and visualizations, collaborating
around common interests.
FLx: experimental analytics ecosystem
Partners share ideas, workflows, analytics, and visualizations, collaborating
around common interests.
FLx: experimental analytics ecosystem
Partners share ideas, workflows, analytics, and visualizations, collaborating
around common interests.
FLx: experimental analytics ecosystem
Partners share ideas, workflows, analytics, and visualizations, collaborating
around common interests.
FLx: experimental analytics ecosystem
Partners share ideas, workflows, analytics, and visualizations, collaborating
around common interests.
FLx: experimental analytics ecosystem
Partners share ideas, workflows, analytics, and visualizations, collaborating
around common interests.
Workflow for social network analytics in NodeXL:
are learners forming effective peer-relationships?
Import data into OpenRefine
Reshape using template
Export data to CSV
Process in NodeXL
and generate network
Martin Hawksey
http://mashe.hawksey.info/2013/02/lak13-recipes-in-capturing-and-analyzing-data-using-sna-on-canvas-discussions-with-nodexl-for-when-its-not-a-snapp
Workflow for quantifying reflections in forum
posts: what elements of reflection are evident?
Convert discussion threads in comma-separated file format
Annotate text segments using
custom components for UIMA
Convert results in CSV
Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing Automatically Detected Reflective Texts with Human Judgements. In 2nd Workshop on Awareness and
Reflection in Technology-Enhanced Learning. Presented at the 7th European Conference on Technology-Enhanced Learning, Saarbruecken, Germany.
Retrieved from http://ceur-ws.org/Vol-931/paper8.pdf
Inspect and analyse data with R
Reason over annotations with Drools
Workflow for academic writing analytics: to
what extent does student writing display the
hallmarks of scholarly argument?
Extract submitted essay drafts from Course XYZ
Convert to text files for XIP
Analyse using rhetorical parser
Render in custom dashboard
Annotate onto source text
Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of
Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics &
Knowledge. Leuven, BE (Apr. 8-12, 2013). Open Access Eprint: http://oro.open.ac.uk/37391
Thank you… Q&A
Belinda Tynan
http://www.open.ac.uk/about/main/admin-and-governance/executive-team/pro-vice-chancellor-learning-and-teaching
Simon Buckingham Shum
http://simon.buckinghamshum.net / @sbskmi

More Related Content

What's hot

Bank: Trends, Tech and Future
Bank: Trends, Tech and FutureBank: Trends, Tech and Future
Bank: Trends, Tech and FutureIvano Digital
 
The Inevitable Future of Banking
The Inevitable Future of BankingThe Inevitable Future of Banking
The Inevitable Future of Bankingaccenture
 
Swarovski case study presentation EEA Specific Markets
Swarovski case study presentation EEA Specific MarketsSwarovski case study presentation EEA Specific Markets
Swarovski case study presentation EEA Specific MarketsIstvan Brem
 
Bank of ceylon Strategy Analysis
Bank of ceylon    Strategy Analysis Bank of ceylon    Strategy Analysis
Bank of ceylon Strategy Analysis Zeinul Haleem
 
Accenture and SAP Business Solutions Group
Accenture and SAP Business Solutions GroupAccenture and SAP Business Solutions Group
Accenture and SAP Business Solutions GroupAccenture Technology
 
Application or Information System at Starbucks
Application or Information System at StarbucksApplication or Information System at Starbucks
Application or Information System at StarbucksTony Kennedy Antronisamy
 
Ali Baba: A Case Study
Ali Baba: A Case StudyAli Baba: A Case Study
Ali Baba: A Case StudyKing Abidi
 
Retail Business Management PowerPoint Presentation Slides
Retail Business Management PowerPoint Presentation Slides Retail Business Management PowerPoint Presentation Slides
Retail Business Management PowerPoint Presentation Slides SlideTeam
 
Online Shopping Behaviour in India
Online Shopping Behaviour in IndiaOnline Shopping Behaviour in India
Online Shopping Behaviour in IndiaIJERD Editor
 
Farfetch - NOAH17 Berlin
Farfetch - NOAH17 BerlinFarfetch - NOAH17 Berlin
Farfetch - NOAH17 BerlinNOAH Advisors
 

What's hot (17)

ZARA.pdf
ZARA.pdfZARA.pdf
ZARA.pdf
 
Bank: Trends, Tech and Future
Bank: Trends, Tech and FutureBank: Trends, Tech and Future
Bank: Trends, Tech and Future
 
The Inevitable Future of Banking
The Inevitable Future of BankingThe Inevitable Future of Banking
The Inevitable Future of Banking
 
Swarovski case study presentation EEA Specific Markets
Swarovski case study presentation EEA Specific MarketsSwarovski case study presentation EEA Specific Markets
Swarovski case study presentation EEA Specific Markets
 
Bank of ceylon Strategy Analysis
Bank of ceylon    Strategy Analysis Bank of ceylon    Strategy Analysis
Bank of ceylon Strategy Analysis
 
Accenture and SAP Business Solutions Group
Accenture and SAP Business Solutions GroupAccenture and SAP Business Solutions Group
Accenture and SAP Business Solutions Group
 
Application or Information System at Starbucks
Application or Information System at StarbucksApplication or Information System at Starbucks
Application or Information System at Starbucks
 
Brand audit
Brand auditBrand audit
Brand audit
 
Ali Baba: A Case Study
Ali Baba: A Case StudyAli Baba: A Case Study
Ali Baba: A Case Study
 
Alibaba Group
Alibaba GroupAlibaba Group
Alibaba Group
 
CRM Final Report
CRM Final ReportCRM Final Report
CRM Final Report
 
Artificial Intelligence in Banking
Artificial Intelligence in BankingArtificial Intelligence in Banking
Artificial Intelligence in Banking
 
H+M Brand Audit
H+M Brand AuditH+M Brand Audit
H+M Brand Audit
 
GEA
GEAGEA
GEA
 
Retail Business Management PowerPoint Presentation Slides
Retail Business Management PowerPoint Presentation Slides Retail Business Management PowerPoint Presentation Slides
Retail Business Management PowerPoint Presentation Slides
 
Online Shopping Behaviour in India
Online Shopping Behaviour in IndiaOnline Shopping Behaviour in India
Online Shopping Behaviour in India
 
Farfetch - NOAH17 Berlin
Farfetch - NOAH17 BerlinFarfetch - NOAH17 Berlin
Farfetch - NOAH17 Berlin
 

Viewers also liked

insight-centre-galway-learning-analytics
insight-centre-galway-learning-analyticsinsight-centre-galway-learning-analytics
insight-centre-galway-learning-analyticsSimon Buckingham Shum
 
Learning Analytics - A New Discipline and Linked Data
Learning Analytics - A New Discipline and Linked DataLearning Analytics - A New Discipline and Linked Data
Learning Analytics - A New Discipline and Linked DataDragan Gasevic
 
Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...
Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...
Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...BCcampus
 
Learning Analytics: Notes on the Future
Learning Analytics: Notes on the FutureLearning Analytics: Notes on the Future
Learning Analytics: Notes on the FutureSimon Buckingham Shum
 
Our Learning Analytics are Our Pedagogy
Our Learning Analytics are Our PedagogyOur Learning Analytics are Our Pedagogy
Our Learning Analytics are Our PedagogySimon Buckingham Shum
 
Learning Analytics -Towards a New Discipline-
Learning Analytics -Towards a New Discipline-Learning Analytics -Towards a New Discipline-
Learning Analytics -Towards a New Discipline-Dragan Gasevic
 
Big Data in Learning Analytics - Analytics for Everyday Learning
Big Data in Learning Analytics - Analytics for Everyday LearningBig Data in Learning Analytics - Analytics for Everyday Learning
Big Data in Learning Analytics - Analytics for Everyday LearningStefan Dietze
 
Learning analytics, action science and critical realism
Learning analytics, action science and critical realismLearning analytics, action science and critical realism
Learning analytics, action science and critical realismTim Rogers
 
Will Learning Analytics Transform Higher Education?
Will Learning Analytics Transform Higher Education?Will Learning Analytics Transform Higher Education?
Will Learning Analytics Transform Higher Education?Abelardo Pardo
 
Personal Learning Graph (PLeG)
Personal Learning Graph (PLeG)Personal Learning Graph (PLeG)
Personal Learning Graph (PLeG)Dragan Gasevic
 
Learning Analytics Dashboards
Learning Analytics DashboardsLearning Analytics Dashboards
Learning Analytics DashboardsSten Govaerts
 
Learning analytics: study goal and data explorer
Learning analytics: study goal and data explorerLearning analytics: study goal and data explorer
Learning analytics: study goal and data explorerJisc
 
uts-learning-futures-learning-analytics
uts-learning-futures-learning-analyticsuts-learning-futures-learning-analytics
uts-learning-futures-learning-analyticsSimon Buckingham Shum
 

Viewers also liked (15)

SysLL Evidence Hub
SysLL Evidence HubSysLL Evidence Hub
SysLL Evidence Hub
 
insight-centre-galway-learning-analytics
insight-centre-galway-learning-analyticsinsight-centre-galway-learning-analytics
insight-centre-galway-learning-analytics
 
Learning Analytics - A New Discipline and Linked Data
Learning Analytics - A New Discipline and Linked DataLearning Analytics - A New Discipline and Linked Data
Learning Analytics - A New Discipline and Linked Data
 
Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...
Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...
Perspectives on Sustainability in Higher Education: Inviting and Leveraging C...
 
Learning Analytics: Notes on the Future
Learning Analytics: Notes on the FutureLearning Analytics: Notes on the Future
Learning Analytics: Notes on the Future
 
Our Learning Analytics are Our Pedagogy
Our Learning Analytics are Our PedagogyOur Learning Analytics are Our Pedagogy
Our Learning Analytics are Our Pedagogy
 
Social Learning Analytics
Social Learning AnalyticsSocial Learning Analytics
Social Learning Analytics
 
Learning Analytics -Towards a New Discipline-
Learning Analytics -Towards a New Discipline-Learning Analytics -Towards a New Discipline-
Learning Analytics -Towards a New Discipline-
 
Big Data in Learning Analytics - Analytics for Everyday Learning
Big Data in Learning Analytics - Analytics for Everyday LearningBig Data in Learning Analytics - Analytics for Everyday Learning
Big Data in Learning Analytics - Analytics for Everyday Learning
 
Learning analytics, action science and critical realism
Learning analytics, action science and critical realismLearning analytics, action science and critical realism
Learning analytics, action science and critical realism
 
Will Learning Analytics Transform Higher Education?
Will Learning Analytics Transform Higher Education?Will Learning Analytics Transform Higher Education?
Will Learning Analytics Transform Higher Education?
 
Personal Learning Graph (PLeG)
Personal Learning Graph (PLeG)Personal Learning Graph (PLeG)
Personal Learning Graph (PLeG)
 
Learning Analytics Dashboards
Learning Analytics DashboardsLearning Analytics Dashboards
Learning Analytics Dashboards
 
Learning analytics: study goal and data explorer
Learning analytics: study goal and data explorerLearning analytics: study goal and data explorer
Learning analytics: study goal and data explorer
 
uts-learning-futures-learning-analytics
uts-learning-futures-learning-analyticsuts-learning-futures-learning-analytics
uts-learning-futures-learning-analytics
 

Similar to Designing Systemic Learning Analytics at the Open University

Wsu District Capacity Of Well Crafted District Wide System Of Support
Wsu District Capacity Of Well Crafted District Wide System Of SupportWsu District Capacity Of Well Crafted District Wide System Of Support
Wsu District Capacity Of Well Crafted District Wide System Of SupportWSU Cougars
 
Leadership in eLearning: Developing the Next Generation of Leaders
Leadership in eLearning: Developing the Next Generation of LeadersLeadership in eLearning: Developing the Next Generation of Leaders
Leadership in eLearning: Developing the Next Generation of LeadersMark Brown
 
Sacred heart 2014
Sacred heart 2014Sacred heart 2014
Sacred heart 2014EdAdvance
 
FACT2 Learning Analytics Task Group Phase 2 report - CIT2014
FACT2 Learning Analytics Task Group Phase 2 report - CIT2014FACT2 Learning Analytics Task Group Phase 2 report - CIT2014
FACT2 Learning Analytics Task Group Phase 2 report - CIT2014Greg Ketcham
 
ASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptxASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptxGIRLYCAMACHO
 
Exploring learning analytics
Exploring learning analyticsExploring learning analytics
Exploring learning analyticsJisc
 
EDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEllen Wagner
 
NCDPI NCTED Meeting April 2014
NCDPI NCTED Meeting April 2014NCDPI NCTED Meeting April 2014
NCDPI NCTED Meeting April 2014rachelmcbroom
 
Key elements in successful online faculty development
Key elements in successful online faculty developmentKey elements in successful online faculty development
Key elements in successful online faculty developmentAlexandra M. Pickett
 
Reimagining and Reinforcing Student Success Into Career Success Across the Cu...
Reimagining and Reinforcing Student Success Into Career Success Across the Cu...Reimagining and Reinforcing Student Success Into Career Success Across the Cu...
Reimagining and Reinforcing Student Success Into Career Success Across the Cu...credomarketing
 
Life Without Levels INSET
Life Without Levels INSETLife Without Levels INSET
Life Without Levels INSETAlex Gingell
 
SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018LACE Project
 
Wsu Ppt Building District Data Capacity
Wsu Ppt Building District Data CapacityWsu Ppt Building District Data Capacity
Wsu Ppt Building District Data CapacityGlenn E. Malone, EdD
 
APP and Controlled Assessment in History - June 2009
APP and Controlled Assessment in History - June 2009APP and Controlled Assessment in History - June 2009
APP and Controlled Assessment in History - June 2009David Drake
 
Scaling up learning analytics
Scaling up learning analyticsScaling up learning analytics
Scaling up learning analyticsRebecca Ferguson
 
SAAIR 2014 keynote Sharon Slade
SAAIR 2014 keynote Sharon SladeSAAIR 2014 keynote Sharon Slade
SAAIR 2014 keynote Sharon SladeSharon Slade
 
Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...
Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...
Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...Blackboard APAC
 

Similar to Designing Systemic Learning Analytics at the Open University (20)

Wsu District Capacity Of Well Crafted District Wide System Of Support
Wsu District Capacity Of Well Crafted District Wide System Of SupportWsu District Capacity Of Well Crafted District Wide System Of Support
Wsu District Capacity Of Well Crafted District Wide System Of Support
 
Leadership in eLearning: Developing the Next Generation of Leaders
Leadership in eLearning: Developing the Next Generation of LeadersLeadership in eLearning: Developing the Next Generation of Leaders
Leadership in eLearning: Developing the Next Generation of Leaders
 
Sacred heart 2014
Sacred heart 2014Sacred heart 2014
Sacred heart 2014
 
FACT2 Learning Analytics Task Group Phase 2 report - CIT2014
FACT2 Learning Analytics Task Group Phase 2 report - CIT2014FACT2 Learning Analytics Task Group Phase 2 report - CIT2014
FACT2 Learning Analytics Task Group Phase 2 report - CIT2014
 
ASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptxASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptx
 
Exploring learning analytics
Exploring learning analyticsExploring learning analytics
Exploring learning analytics
 
EDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINAL
 
NCDPI NCTED Meeting April 2014
NCDPI NCTED Meeting April 2014NCDPI NCTED Meeting April 2014
NCDPI NCTED Meeting April 2014
 
Key elements in successful online faculty development
Key elements in successful online faculty developmentKey elements in successful online faculty development
Key elements in successful online faculty development
 
Reimagining and Reinforcing Student Success Into Career Success Across the Cu...
Reimagining and Reinforcing Student Success Into Career Success Across the Cu...Reimagining and Reinforcing Student Success Into Career Success Across the Cu...
Reimagining and Reinforcing Student Success Into Career Success Across the Cu...
 
Life Without Levels INSET
Life Without Levels INSETLife Without Levels INSET
Life Without Levels INSET
 
FTCC - Executive Leadership Track
FTCC - Executive Leadership TrackFTCC - Executive Leadership Track
FTCC - Executive Leadership Track
 
SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018
 
Wsu Ppt Building District Data Capacity
Wsu Ppt Building District Data CapacityWsu Ppt Building District Data Capacity
Wsu Ppt Building District Data Capacity
 
Session 4
Session 4Session 4
Session 4
 
APP and Controlled Assessment in History - June 2009
APP and Controlled Assessment in History - June 2009APP and Controlled Assessment in History - June 2009
APP and Controlled Assessment in History - June 2009
 
Scaling up learning analytics
Scaling up learning analyticsScaling up learning analytics
Scaling up learning analytics
 
Leading with Data: Impacting Change on Scale featuring Belinda Tynan
Leading with Data: Impacting Change on Scale featuring Belinda TynanLeading with Data: Impacting Change on Scale featuring Belinda Tynan
Leading with Data: Impacting Change on Scale featuring Belinda Tynan
 
SAAIR 2014 keynote Sharon Slade
SAAIR 2014 keynote Sharon SladeSAAIR 2014 keynote Sharon Slade
SAAIR 2014 keynote Sharon Slade
 
Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...
Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...
Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darw...
 

More from Simon Buckingham Shum

The Generative AI System Shock, and some thoughts on Collective Intelligence ...
The Generative AI System Shock, and some thoughts on Collective Intelligence ...The Generative AI System Shock, and some thoughts on Collective Intelligence ...
The Generative AI System Shock, and some thoughts on Collective Intelligence ...Simon Buckingham Shum
 
Could Generative AI Augment Reflection, Deliberation and Argumentation?
Could Generative AI Augment Reflection, Deliberation and Argumentation?Could Generative AI Augment Reflection, Deliberation and Argumentation?
Could Generative AI Augment Reflection, Deliberation and Argumentation?Simon Buckingham Shum
 
Conversational, generative AI as a cognitive tool for critical thinking
Conversational, generative AI as a cognitive tool for critical thinkingConversational, generative AI as a cognitive tool for critical thinking
Conversational, generative AI as a cognitive tool for critical thinkingSimon Buckingham Shum
 
On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...
On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...
On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...Simon Buckingham Shum
 
Is “The Matter With Things” also what’s the matter with Learning Analytics?
Is “The Matter With Things” also what’s the matter with Learning Analytics?Is “The Matter With Things” also what’s the matter with Learning Analytics?
Is “The Matter With Things” also what’s the matter with Learning Analytics?Simon Buckingham Shum
 
Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...Simon Buckingham Shum
 
Knowledge Art or… “Participatory Improvisational DVN”
Knowledge Art or… “Participatory Improvisational DVN”Knowledge Art or… “Participatory Improvisational DVN”
Knowledge Art or… “Participatory Improvisational DVN”Simon Buckingham Shum
 
March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...
March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...
March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...Simon Buckingham Shum
 
ICQE20: Quantitative Ethnography Visualizations as Tools for Thinking
ICQE20: Quantitative Ethnography Visualizations as Tools for ThinkingICQE20: Quantitative Ethnography Visualizations as Tools for Thinking
ICQE20: Quantitative Ethnography Visualizations as Tools for ThinkingSimon Buckingham Shum
 
24/7 Instant Feedback on Writing: Integrating AcaWriter into your Teaching
24/7 Instant Feedback on Writing: Integrating AcaWriter into your Teaching24/7 Instant Feedback on Writing: Integrating AcaWriter into your Teaching
24/7 Instant Feedback on Writing: Integrating AcaWriter into your TeachingSimon Buckingham Shum
 
Argumentation 101 for Learning Analytics PhDs!
Argumentation 101 for Learning Analytics PhDs!Argumentation 101 for Learning Analytics PhDs!
Argumentation 101 for Learning Analytics PhDs!Simon Buckingham Shum
 
Learning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • AgencyLearning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • AgencySimon Buckingham Shum
 
AI/Data Analytics (AIDA): Key concepts, examples & risks
AI/Data Analytics (AIDA): Key concepts, examples & risksAI/Data Analytics (AIDA): Key concepts, examples & risks
AI/Data Analytics (AIDA): Key concepts, examples & risksSimon Buckingham Shum
 
Learning Analytics as Educational Knowledge Infrastructure
Learning Analytics as Educational Knowledge InfrastructureLearning Analytics as Educational Knowledge Infrastructure
Learning Analytics as Educational Knowledge InfrastructureSimon Buckingham Shum
 
Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data
Towards Collaboration Translucence: Giving Meaning to Multimodal Group DataTowards Collaboration Translucence: Giving Meaning to Multimodal Group Data
Towards Collaboration Translucence: Giving Meaning to Multimodal Group DataSimon Buckingham Shum
 
Knowledge Art - MDSI Guest Lecture - 1st May 2019
Knowledge Art - MDSI Guest Lecture - 1st May 2019Knowledge Art - MDSI Guest Lecture - 1st May 2019
Knowledge Art - MDSI Guest Lecture - 1st May 2019Simon Buckingham Shum
 
Educational Data Scientists: A Scarce Breed
Educational Data Scientists: A Scarce BreedEducational Data Scientists: A Scarce Breed
Educational Data Scientists: A Scarce BreedSimon Buckingham Shum
 
Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018Simon Buckingham Shum
 

More from Simon Buckingham Shum (20)

The Generative AI System Shock, and some thoughts on Collective Intelligence ...
The Generative AI System Shock, and some thoughts on Collective Intelligence ...The Generative AI System Shock, and some thoughts on Collective Intelligence ...
The Generative AI System Shock, and some thoughts on Collective Intelligence ...
 
Could Generative AI Augment Reflection, Deliberation and Argumentation?
Could Generative AI Augment Reflection, Deliberation and Argumentation?Could Generative AI Augment Reflection, Deliberation and Argumentation?
Could Generative AI Augment Reflection, Deliberation and Argumentation?
 
Conversational, generative AI as a cognitive tool for critical thinking
Conversational, generative AI as a cognitive tool for critical thinkingConversational, generative AI as a cognitive tool for critical thinking
Conversational, generative AI as a cognitive tool for critical thinking
 
On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...
On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...
On the Design of a Writing App offering 24/7 Formative Feedback on Reflective...
 
SBS_ISLS2022.pdf
SBS_ISLS2022.pdfSBS_ISLS2022.pdf
SBS_ISLS2022.pdf
 
Is “The Matter With Things” also what’s the matter with Learning Analytics?
Is “The Matter With Things” also what’s the matter with Learning Analytics?Is “The Matter With Things” also what’s the matter with Learning Analytics?
Is “The Matter With Things” also what’s the matter with Learning Analytics?
 
Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...
 
Knowledge Art or… “Participatory Improvisational DVN”
Knowledge Art or… “Participatory Improvisational DVN”Knowledge Art or… “Participatory Improvisational DVN”
Knowledge Art or… “Participatory Improvisational DVN”
 
March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...
March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...
March 2021 • 24/7 Instant Feedback on Writing: Integrating AcaWriter into yo...
 
ICQE20: Quantitative Ethnography Visualizations as Tools for Thinking
ICQE20: Quantitative Ethnography Visualizations as Tools for ThinkingICQE20: Quantitative Ethnography Visualizations as Tools for Thinking
ICQE20: Quantitative Ethnography Visualizations as Tools for Thinking
 
24/7 Instant Feedback on Writing: Integrating AcaWriter into your Teaching
24/7 Instant Feedback on Writing: Integrating AcaWriter into your Teaching24/7 Instant Feedback on Writing: Integrating AcaWriter into your Teaching
24/7 Instant Feedback on Writing: Integrating AcaWriter into your Teaching
 
Argumentation 101 for Learning Analytics PhDs!
Argumentation 101 for Learning Analytics PhDs!Argumentation 101 for Learning Analytics PhDs!
Argumentation 101 for Learning Analytics PhDs!
 
Learning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • AgencyLearning Informatics: AI • Analytics • Accountability • Agency
Learning Informatics: AI • Analytics • Accountability • Agency
 
AI/Data Analytics (AIDA): Key concepts, examples & risks
AI/Data Analytics (AIDA): Key concepts, examples & risksAI/Data Analytics (AIDA): Key concepts, examples & risks
AI/Data Analytics (AIDA): Key concepts, examples & risks
 
Learning Analytics as Educational Knowledge Infrastructure
Learning Analytics as Educational Knowledge InfrastructureLearning Analytics as Educational Knowledge Infrastructure
Learning Analytics as Educational Knowledge Infrastructure
 
Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data
Towards Collaboration Translucence: Giving Meaning to Multimodal Group DataTowards Collaboration Translucence: Giving Meaning to Multimodal Group Data
Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data
 
Knowledge Art - MDSI Guest Lecture - 1st May 2019
Knowledge Art - MDSI Guest Lecture - 1st May 2019Knowledge Art - MDSI Guest Lecture - 1st May 2019
Knowledge Art - MDSI Guest Lecture - 1st May 2019
 
UX/LX for PLSA: Workshop Welcome
UX/LX for PLSA: Workshop WelcomeUX/LX for PLSA: Workshop Welcome
UX/LX for PLSA: Workshop Welcome
 
Educational Data Scientists: A Scarce Breed
Educational Data Scientists: A Scarce BreedEducational Data Scientists: A Scarce Breed
Educational Data Scientists: A Scarce Breed
 
Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018
 

Recently uploaded

Introduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher EducationIntroduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher Educationpboyjonauth
 
Difference Between Search & Browse Methods in Odoo 17
Difference Between Search & Browse Methods in Odoo 17Difference Between Search & Browse Methods in Odoo 17
Difference Between Search & Browse Methods in Odoo 17Celine George
 
EPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptxEPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptxRaymartEstabillo3
 
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...Nguyen Thanh Tu Collection
 
Employee wellbeing at the workplace.pptx
Employee wellbeing at the workplace.pptxEmployee wellbeing at the workplace.pptx
Employee wellbeing at the workplace.pptxNirmalaLoungPoorunde1
 
Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designMIPLM
 
Judging the Relevance and worth of ideas part 2.pptx
Judging the Relevance  and worth of ideas part 2.pptxJudging the Relevance  and worth of ideas part 2.pptx
Judging the Relevance and worth of ideas part 2.pptxSherlyMaeNeri
 
Quarter 4 Peace-education.pptx Catch Up Friday
Quarter 4 Peace-education.pptx Catch Up FridayQuarter 4 Peace-education.pptx Catch Up Friday
Quarter 4 Peace-education.pptx Catch Up FridayMakMakNepo
 
Crayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon ACrayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon AUnboundStockton
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Celine George
 
Influencing policy (training slides from Fast Track Impact)
Influencing policy (training slides from Fast Track Impact)Influencing policy (training slides from Fast Track Impact)
Influencing policy (training slides from Fast Track Impact)Mark Reed
 
Grade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptxGrade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptxChelloAnnAsuncion2
 
Romantic Opera MUSIC FOR GRADE NINE pptx
Romantic Opera MUSIC FOR GRADE NINE pptxRomantic Opera MUSIC FOR GRADE NINE pptx
Romantic Opera MUSIC FOR GRADE NINE pptxsqpmdrvczh
 
Roles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in PharmacovigilanceRoles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in PharmacovigilanceSamikshaHamane
 
ECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptx
ECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptxECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptx
ECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptxiammrhaywood
 
Full Stack Web Development Course for Beginners
Full Stack Web Development Course  for BeginnersFull Stack Web Development Course  for Beginners
Full Stack Web Development Course for BeginnersSabitha Banu
 
Introduction to AI in Higher Education_draft.pptx
Introduction to AI in Higher Education_draft.pptxIntroduction to AI in Higher Education_draft.pptx
Introduction to AI in Higher Education_draft.pptxpboyjonauth
 
Computed Fields and api Depends in the Odoo 17
Computed Fields and api Depends in the Odoo 17Computed Fields and api Depends in the Odoo 17
Computed Fields and api Depends in the Odoo 17Celine George
 
How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17Celine George
 

Recently uploaded (20)

Introduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher EducationIntroduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher Education
 
Difference Between Search & Browse Methods in Odoo 17
Difference Between Search & Browse Methods in Odoo 17Difference Between Search & Browse Methods in Odoo 17
Difference Between Search & Browse Methods in Odoo 17
 
EPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptxEPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptx
 
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
 
Employee wellbeing at the workplace.pptx
Employee wellbeing at the workplace.pptxEmployee wellbeing at the workplace.pptx
Employee wellbeing at the workplace.pptx
 
Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-design
 
Judging the Relevance and worth of ideas part 2.pptx
Judging the Relevance  and worth of ideas part 2.pptxJudging the Relevance  and worth of ideas part 2.pptx
Judging the Relevance and worth of ideas part 2.pptx
 
Quarter 4 Peace-education.pptx Catch Up Friday
Quarter 4 Peace-education.pptx Catch Up FridayQuarter 4 Peace-education.pptx Catch Up Friday
Quarter 4 Peace-education.pptx Catch Up Friday
 
Crayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon ACrayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon A
 
Rapple "Scholarly Communications and the Sustainable Development Goals"
Rapple "Scholarly Communications and the Sustainable Development Goals"Rapple "Scholarly Communications and the Sustainable Development Goals"
Rapple "Scholarly Communications and the Sustainable Development Goals"
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17
 
Influencing policy (training slides from Fast Track Impact)
Influencing policy (training slides from Fast Track Impact)Influencing policy (training slides from Fast Track Impact)
Influencing policy (training slides from Fast Track Impact)
 
Grade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptxGrade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptx
 
Romantic Opera MUSIC FOR GRADE NINE pptx
Romantic Opera MUSIC FOR GRADE NINE pptxRomantic Opera MUSIC FOR GRADE NINE pptx
Romantic Opera MUSIC FOR GRADE NINE pptx
 
Roles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in PharmacovigilanceRoles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in Pharmacovigilance
 
ECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptx
ECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptxECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptx
ECONOMIC CONTEXT - PAPER 1 Q3: NEWSPAPERS.pptx
 
Full Stack Web Development Course for Beginners
Full Stack Web Development Course  for BeginnersFull Stack Web Development Course  for Beginners
Full Stack Web Development Course for Beginners
 
Introduction to AI in Higher Education_draft.pptx
Introduction to AI in Higher Education_draft.pptxIntroduction to AI in Higher Education_draft.pptx
Introduction to AI in Higher Education_draft.pptx
 
Computed Fields and api Depends in the Odoo 17
Computed Fields and api Depends in the Odoo 17Computed Fields and api Depends in the Odoo 17
Computed Fields and api Depends in the Odoo 17
 
How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17
 

Designing Systemic Learning Analytics at the Open University

  • 1. Designing Systemic Learning Analytics at the Open University Simon Buckingham Shum Knowledge Media Institute The Open University, UK Strategy & Policy for Systemic Learning Analytics SoLAR Open Course, 11th Oct 2013 https://learn.canvas.net/courses/182/wiki/designing-systemic-analytics-at-the-open-university Belinda Tynan Pro-Vice-Chancellor Learning & Teaching The Open University, UK
  • 2. overview Belinda: developing an institutional strategy — framework and implementation Simon: research perspectives — 3 metaphors for systemic analytics Discussion
  • 3. Learning and Teaching Strategy for Systemic Deployment of Analytics at the Open University Belinda Tynan, Pro-Vice-Chancellor, Learning and Teaching Kevin Mayles, Senior Manager, Learning and Teaching
  • 4. Learning and Teaching Analytics for student success vision p.4 A clear vision has been developed to galvanise effort across the institution on the focused use of analytics to drive student success Vision   To  use  and  apply  informa2on  strategically  (through  specified  indicators)  to   retain  students  and  progress  them  to  complete  their  study  goals     Mission   This  needs  to  be  achieved  at  :   •  a  macro  level  to  aggregate  informa5on  about  the  student  learning   experience  at  an  ins5tu5onal  level  to  inform  strategic  priori5es  that  will   improve  student  reten5on  and  progression   •  a  micro  level  to  use  analy5cs  to  drive  short,  medium  and  long-­‐term   interven5ons  
  • 5. Learning and Teaching What drives student success? p.5 We have an emerging picture of the factors affecting student success based on existing statistical analyses, literature and “institutional knowledge” and our current use of associated indicators Framework adapted from Woodley et. al. (2001) Student Progress in Distance Education: Kember’s model re-visited • Early  contact   • Early  engagement   • Study  calendar  /  scheduling   • Tutor  support   • Peer  support  &  belonging   • Study  habits   • Employer  support   • Family  support   • Personal    life  events   • Financial  issues   • Advice  on  course  choice   • Subject  studied   • Prepara5on  for  study   • Learning  design   • Quality  of  study  materials   • Workload   • Module  assessment   strategy   • Language  ability   • Previous  educa5on  /  OU  study   • Ethnic  group   • Socio-­‐economic  background   • Disability   • Age   • Study  goal  /  mo5va5on   • Gender   Entry   characteris-cs   Academic   compa-bility   Social  and   academic   integra-on   External  factors   Student   Success   Indicators  used  in  exis-ng  analy-cs   There  are  a  number  of  indicators  with   suppor5ng  evidence  that  we  currently   use  in  our  analysis  models   Clear  evidence  of  impact  but  currently   not  used  in  analy-cs   We  have  a  number  of  factors  for  which   there  is  clear  evidence  of  the  impact  on   success  but  are  not  being  used  in  current   analy5cs  models  due  to  lack  of  data  or   insufficient  inves5ga5on   Unclear  evidence  base   There  are  a  number  of  factors  that  the   OU  believes  or  literature  suggests  have   an  impact  on  student  success  but  where   we  have  no  clear  evidence  at  this  5me   due  either  to  lack  of  data  availability  or   insufficient  inves5ga5on   Results  from  a  review  of  exis-ng  evidence  on   the  drivers  of  student  success  are  giving  us  a   mixed  picture   Indicators  with  evidence  of  no  impact   There  are  a  number  of  indicators  with   suppor5ng  evidence  that  suggest  they   have  a  minimal  impact  on  success  
  • 6. Learning and Teaching Developing institutional capabilities and strengths 3 year strategic roadmap The OU is developing its capabilities in 10 key areas that build the underpinning strengths required for the effective deployment of analytics We  need  to  ensure  we  have  the  right  architecture  and  processes   for  collec5ng  the  right  data  and  making  it  accessible  for  analy5cs   –  we  need  a  ‘big  data’  mind-­‐set   The  university   needs  world  class   capability  in  data   science  to   con5nually  mine   the  data  and  build   rapid  prototypes  of   simple  tools,  and  a   clear  pipeline  for   the  outputs  to  be   mainstreamed  into   opera5ons   Benefits  will  be  realised  through   exis5ng  business  processes   impac5ng  on  students  directly   and  through  enhancement  of   the  student  learning  experience   –  we  will  develop  an  ‘analy5cs   mind-­‐set’  in     these  areas   p.6
  • 7. Learning and Teaching Framework for harnessing analytics for student success through driving interventions p.7 Analytics will be applied throughout the cycle of the student learning experience Analy-cs  applied…   Example  business  processes   Example  datasets  used   For  ac-on   Op5mise  student  alloca5on  to  tutor  groups     Development  of  learning  systems   Assessment  strategy  and  scheduling   Student  pass/fail   predic5ons   Study  behaviour  profiles   Pass  rates  modelling   In  ac-on   Early  contact  with  ‘at  risk’  students     Module  presenta5on  issue  flagging   Student  ‘at  risk’  predic5ve   indicators   Helpdesk  contact  records   On  ac-on   Annual  module  and  programme  review   Learning  design   Module  performance  KPIs   Learning  design  profiles   “In Action, On Action” from Donald Schön The Reflective Practitioner
  • 8. Learning and Teaching Applying ‘in action’ analytics to drive student success? p.8 We will use analytics to put key information relating to student success in the hands of those in a position to take action Tutor  Group  List   Students’  study  history   Feedback  from  previous  tutors   Predicted  probability  of  passing   ‘At  risk’  factors   Associate  Lecturer   Plan  early  contact  with  most  at  risk  students   Monitor  engagement  prior  to  first  TMA   Refer  issues  to  SST  quickly     Weekly  Alert  Dashboard   Weekly  update  of  students’  predicted   probability  of  passing  /  progressing   List  of  most  ‘at  risk’  students  this  week   Target  resources  at  most  at  risk  students   Call  or  email  students  on  at  risk  list  to  offer   support   No5fy  ALs  of  any  issues  arising  in  their  groups   Student  Support  Team   Module  performance  report   Predicted  pass  rate  vs  target  pass  rate   updated  during  presenta5on   Analysis  of  online  learning  ac5vity   usage  /  engagement  pa^erns   Iden5fy  any  issues  with  the  module  whilst  in   presenta5on  and  take  ac5on  to  rec5fy   Evaluate  the  use  of  learning  assets  to  inform   future  produc5on   Faculty  Academics   Senior  Management   Student  success  measures   Indicators  derived  from   sta5s5cal  modelling  that   underpin  student   progression  measures   Monitor  student  progression  forecasts  against   target  –  iden5fy  correc5ve  ac5on   Target  resources  at  specific  ‘pinch  points’  in  the   student  journey  
  • 9. Learning and Teaching Evaluating impact and driving action p.9 The basis for evaluation needs to link interventions to measurable outcomes of student success
  • 10. Learning and Teaching Evaluating impact and driving action p.10 The basis for evaluation needs to link interventions to measurable outcomes of student success STUDENT   SUCCESS  
  • 11. Learning and Teaching Evaluating impact and driving action p.11 The basis for evaluation needs to link interventions to measurable outcomes of student success STUDENT   SUCCESS   Interven-ons   For  ac5on   In  ac5on   On  ac5on  
  • 12. Learning and Teaching Evaluating impact and driving action p.12 The basis for evaluation needs to link interventions to measurable outcomes of student success Governance   and   Management   STUDENT   SUCCESS   Interven-ons   For  ac5on   In  ac5on   On  ac5on  
  • 13. Learning and Teaching Evaluating impact and driving action p.13 The basis for evaluation needs to link interventions to measurable outcomes of student success Drivers  of   student  success  Governance   and   Management   STUDENT   SUCCESS   Interven-ons   For  ac5on   In  ac5on   On  ac5on  
  • 14. Learning and Teaching Improve  ins-tu-onal  capabili-es  and   processes   Evaluating impact and driving action p.14 The basis for evaluation needs to link interventions to measurable outcomes of student success Drivers  of   student  success  Governance   and   Management   STUDENT   SUCCESS   Interven-ons   For  ac5on   In  ac5on   On  ac5on  
  • 15. Learning and Teaching Improve  ins-tu-onal  capabili-es  and   processes   Evaluating impact and driving action p.15 The basis for evaluation needs to link interventions to measurable outcomes of student success Drivers  of   student  success  Governance   and   Management   STUDENT   SUCCESS   Interven-ons   For  ac5on   In  ac5on   On  ac5on   Are  we  seeing   expected   outcomes  of   our   interven5ons?   Are  we  doing   the  right  things   as  guided  by  the   evidence?  
  • 16. Analytics will drive action to increase student success Vision: To use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals
  • 17. Analytics will drive action to increase student success Vision: To use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals Recruit Retain Progress Complete Success  outcomes   and  leading   indicators  
  • 18. Analytics will drive action to increase student success Vision: To use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals Recruit Retain Progress Complete Success  outcomes   and  leading   indicators   Student  support   ac5vi5es   Learning  &  teaching   ac5vi5es   Measures  of  our   opera5onal   performance  and   interven5ons   Drivers  of  student  success   Evidence  of  the   drivers  of  student   success  guides   what  we  do  and   what  we  measure  
  • 19. Analytics will drive action to increase student success Vision: To use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals Recruit Retain Progress Complete Success  outcomes   and  leading   indicators   Student  support   ac5vi5es   Learning  &  teaching   ac5vi5es   Measures  of  our   opera5onal   performance  and   interven5ons   Dashboards  /   Reports  /  Tools   Ins5tu5onal   Dashboard   PVCs   Deans   Programme   Directors   Module  Teams   Student  Support   Teams   Indicators and measures fed into dashboards and reports at relevant levels Drivers  of  student  success   Evidence  of  the   drivers  of  student   success  guides   what  we  do  and   what  we  measure  
  • 20. Analytics will drive action to increase student success Vision: To use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals Recruit Retain Progress Complete Success  outcomes   and  leading   indicators   Student  support   ac5vi5es   Learning  &  teaching   ac5vi5es   Measures  of  our   opera5onal   performance  and   interven5ons   Dashboards  /   Reports  /  Tools   Ins5tu5onal   Dashboard   PVCs   Deans   Programme   Directors   Module  Teams   Student  Support   Teams   Indicators and measures fed into dashboards and reports at relevant levels Drivers  of  student  success   Evidence  of  the   drivers  of  student   success  guides   what  we  do  and   what  we  measure   ACTION   Interven-on  
  • 21. Analytics will drive action to increase student success Vision: To use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals Recruit Retain Progress Complete Success  outcomes   and  leading   indicators   Student  support   ac5vi5es   Learning  &  teaching   ac5vi5es   Measures  of  our   opera5onal   performance  and   interven5ons   Dashboards  /   Reports  /  Tools   Ins5tu5onal   Dashboard   PVCs   Deans   Programme   Directors   Module  Teams   Student  Support   Teams   Indicators and measures fed into dashboards and reports at relevant levels Drivers  of  student  success   Evidence  of  the   drivers  of  student   success  guides   what  we  do  and   what  we  measure   ACTION   Interven-on   Evalua-on  of  the  outcomes  from  interven5ons  increases  our   evidence  base  of  what  drives  student  success  
  • 23. 3 metaphors for systemic analytics 1. the aquarium 2. from exoskeleton to nervous system 3. resilience through biodiversity
  • 24. metaphor 1 the aquarium systems strategy: research the key variables for a healthy ecosystem and evolve predictive models as rapidly as possible to detect variance
  • 25. Aquarium science enables aquarium analytics which monitor the health of the ecosystem
  • 26. fish aquarium science learners? learning science instructional design It’s all about knowing what to watch
  • 27. Purdue University Signals: exemplar ‘healthy ecosystem’ variables 27
  • 28. Purdue University Signals: exemplar ‘healthy ecosystem’ variables 28 Key variables identified: •  ACT or SAT score •  Overall grade-point average •  CMS usage composite •  CMS assessment composite •  CMS assignment composite •  CMS calendar composite
  • 29. Hmmm… no learning sciences no learning design underpinning these predictive models of student success models based on a mix of institutional know-how about student success, and analysing behavioural data 29
  • 30. the opportunity for the learning sciences to combine with your university’s collective intelligence 30
  • 31. Predictive modelling of student outcomes Registra-on   PaMern   CRM   contact   VLE   interac-on   Assignment   grades   Demo-­‐ graphics   ? Can we combine datasets, and use machine learning to build models to identify ‘signature’ patterns associated with different kinds of students? Library   interac-on   OpenLearn   interac-on   FutureLearn   interac-on   App  X   interac-on   OU  track   record  
  • 32. Predictive modelling of student outcomes Registra-on   PaMern   CRM   contact   VLE   interac-on   Assignment   grades   Demo-­‐ graphics   ? Does VLE data carry information that provides more precise early identification of failing students than is currently possible? Simple example (just 3 demographic attributes and VLE): Input: Demographic data: New/Continuing student, Sex, Previous education VLE interactions without qualifying the type (any click counts) Goal: Evaluate the probability that the student does not submit TMA1 or submits and scores lower than 40. Method: Naïve Bayes network (e.g. see Bishop, 2009) Library   interac-on   OpenLearn   interac-on   FutureLearn   interac-on   App  X   interac-on   OU  track   record  
  • 33. Learning and Teaching Back to the OU’s analytics framework
  • 34. Learning and Teaching Predictive modelling within the framework VLE  user  trace  data  /  student  demographics  /   academic  achievement   Strategic internal funding to advance a promising technique from an externally funded (JISC) project, and embed within OU student support processes: A.L. Wolff and Z. Zdrahal (2012). Improving Retention by Identifying and Supporting “At-risk” Students. EDUCAUSE Review Online, July-August 2012. http://www.educause.edu/ero/article/improving-retention-identifying-and-supporting-risk-students Zdenek Zdrahal Lead, KMi Predictive Modelling Team http://kmi.open.ac.uk/people/member/zdenek-zdrahal
  • 35. Learning and Teaching Predictive modelling within the framework Develop  and  Validate   Predic-ve  Models  of   student  success   (module  comple-on)   in  order  to  trigger   more  -mely  alerts   VLE  user  trace  data  /  student  demographics  /   academic  achievement  
  • 36. Learning and Teaching Predictive modelling within the framework Develop  and  Validate   Predic-ve  Models  of   student  success   (module  comple-on)   in  order  to  trigger   more  -mely  alerts   Requirements   to  mainstream   the  models  in   the  VLE   VLE  user  trace  data  /  student  demographics  /   academic  achievement  
  • 37. Learning and Teaching Predictive modelling within the framework Develop  and  Validate   Predic-ve  Models  of   student  success   (module  comple-on)   in  order  to  trigger   more  -mely  alerts   Prototype  Student   Support  Team   dashboards   Requirements   to  mainstream   the  models  in   the  VLE   VLE  user  trace  data  /  student  demographics  /   academic  achievement  
  • 38. Learning and Teaching Predictive modelling within the framework Develop  and  Validate   Predic-ve  Models  of   student  success   (module  comple-on)   in  order  to  trigger   more  -mely  alerts   New  tool  for  Student  Support  Teams,  to  enable  -mely   interven-on.  E.g.  real-­‐-me  traffic  lights  on  at  risk  students   Requirements   to  mainstream   the  models  in   the  VLE   VLE  user  trace  data  /  student  demographics  /   academic  achievement   Prototype  Student   Support  Team   dashboards  
  • 39. Learning and Teaching Predictive modelling within the framework Develop  and  Validate   Predic-ve  Models  of   student  success   (module  comple-on)   in  order  to  trigger   more  -mely  alerts   New  tool  for  Student  Support  Teams,  to  enable  -mely   interven-on.  E.g.  real-­‐-me  traffic  lights  on  at  risk  students   Modules  are  accompanied  by   machine-­‐readable  metadata  that   increases  the  power  of  machine   learning  when  it  comes  to  data   analysis   Requirements   to  mainstream   the  models  in   the  VLE   VLE  user  trace  data  /  student  demographics  /   academic  achievement   Prototype  Student   Support  Team   dashboards  
  • 40. Problem specification Learning to dynamically id an at-risk student We are hereWe know
  • 41. Problem specification Learning to dynamically id an at-risk student We are hereWe know We predict
  • 42. e.g. The Retain project Does VLE activity add value to predictive models? We know We predict Sex Educ New/ Cont VLE Model the probability of failing at TMA1 which is known to be a key predictor of final outcome either by not submitting TMA1, or by submitting with score < 40. TMA1
  • 43. •  Demographic  profile  1   –  X   –  Y   –  Z     Sex   Educa5on   N/C   TMA1   Without  VLE  data:   Probability  of  failing  at  TMA1    =  18.5%   Student profile 1
  • 44. Student profile 1   •  Demographic  profile  1   –  X   –  Y   –  Z     Sex   Educa5on   N/C   TMA1   Without  VLE  data:   Probability  of  failing  at  TMA1    =  18.5%   Sex   Educa5on   N/C   VLE   TMA1   Clicks   Probability   Nr  of  students   0   64%   4   1-­‐20   44%   3   21-­‐100   26%   5   101-­‐800   6.3%   14   With  VLE  data,  a  higher  fidelity  story:  
  • 45. Student profile 2   Sex   Educa5on   N/C   TMA1   Without  VLE  data:   Probability  of  failing  at  TMA1    =  7.7%   •  Demographic  profile  2   –  X   –  Y   –  Z    
  • 46. Student profile 2   Sex   Educa5on   N/C   TMA1   Without  VLE  data:   Probability  of  failing  at  TMA1    =  7.7%   Sex   Educa5on   N/C   VLE   TMA1   Clicks   Probability   Nr  of  students   0   39%   35   1-­‐20   22%   74   21-­‐100   11.2%   178   101-­‐800   2.4%   461   With  VLE  data,  a  higher  fidelity  story:   •  Demographic  profile  2   –  X   –  Y   –  Z    
  • 47. Potential to augment student support teams with predictive modelling once validated Query VLE   interac-on   Assignment   grades   Demo-­‐ graphics   OU  track   record  
  • 48. Potential to augment student support teams with predictive modelling once validated 7 of your students have fail trajectory BUT prioritize Nigel, then Sue, then Ian because - has not engaged with VLE - at least one TMA below 40 - has not submitted 5 TMAs Query VLE   interac-on   Assignment   grades   Demo-­‐ graphics   OU  track   record  
  • 49. Why do I need a variable ML approach? Can’t I just use one method (off the shelf)? Registra5on   Pa^ern   CRM   interac5ons   Library   interac5on   FutureLearn   interac5on   Train  and  Learn     as  new  data  is  added   using  variable  methods   Methods successfully tested, to be further developed: •  Induction of decision tree (ID3, C4.5 from the Weka toolkit) •  Support Vector Machine (from Weka) •  Bayes network (Microsoft Infer.NET; SamIam - Stanford Univ.) •  Naïve Bayes (see the example and Demo Cases) •  Linear regression •  Logistic regression •  GUHA (General Unary Hypotheses Automaton)
  • 50. metaphor 2 from exoskeleton to nervous system systems strategy: embed faster feedback loops, and build sensemaking capacity at all levels
  • 51. Evolving the OU from a digital exoskeleton to a nervous system? Ed Dumbill: http://strata.oreilly.com/2012/08/digital-nervous-system-big-data.html
  • 52. Learning and Teaching The OU’s collective intelligence Macro Level Analytics Towards multilevel systemic analytics for student success Designing better feedback loops at all levels of learning (students + staff) Micro Level Analytics Student Interaction Traces
  • 53. Learning and Teaching The OU’s collective intelligence Macro Level Analytics Towards multilevel systemic analytics for student success Designing better feedback loops at all levels of learning (students + staff) Micro Level Analytics Student Interaction Traces Student Support Teams Associate Lecturers Researchers interpretation/intervention
  • 54. Learning and Teaching The OU’s collective intelligence Macro Level Analytics Towards multilevel systemic analytics for student success Designing better feedback loops at all levels of learning (students + staff) Micro Level Analytics Student Interaction Traces Student Support Teams Associate Lecturers Researchers interpretation/intervention Data Wranglers Researchers
  • 55. Learning and Teaching The OU’s collective intelligence Macro Level Analytics Towards multilevel systemic analytics for student success Designing better feedback loops at all levels of learning (students + staff) Micro Level Analytics Student Interaction Traces VC Executive Faculties Module Teams Student Support Teams Associate Lecturers Researchers interpretation/intervention Data Wranglers Researchers
  • 56. Learning and Teaching The OU’s collective intelligence Macro Level Analytics Towards multilevel systemic analytics for student success Designing better feedback loops at all levels of learning (students + staff) Micro Level Analytics Student Interaction Traces VC Executive Faculties Module Teams Student Support Teams Associate Lecturers Researchers interpretation/intervention Data Wranglers Researchers Visual Analytics Design Quality Data, Integrated Analytics Competency Team Organisational Capacity building Analytics Research Computational Platforms
  • 57. metaphor 3 build resilience systems strategy: MOOCs can be viewed as a system- level ‘shock’ to the HigherEd ecology (‘regime shift’?) build resilience by expanding our diversity and capacity to sense the dynamic environment
  • 58. MOOCs are an innovation and research platform — analytics will be critical http://www.ted.com/talks/daphne_koller_what_we_re_learning_from_online_education.html http://people.kmi.open.ac.uk/sbs/2013/01/emerging-mooc-data-analytics-ecosystem http://www.slideshare.net/abelardo_pardo/pushing-the-mooc-envelope-with-learning-analytics http://www.moocresearch.com/research-initiative/about#Cost,%20Performance%20Metrics%20and%20Learner%20Analytics
  • 59. FLx: experimental analytics ecosystem Partners share ideas, workflows, analytics, and visualizations, collaborating around common interests.
  • 60. FLx: experimental analytics ecosystem Partners share ideas, workflows, analytics, and visualizations, collaborating around common interests.
  • 61. FLx: experimental analytics ecosystem Partners share ideas, workflows, analytics, and visualizations, collaborating around common interests.
  • 62. FLx: experimental analytics ecosystem Partners share ideas, workflows, analytics, and visualizations, collaborating around common interests.
  • 63. FLx: experimental analytics ecosystem Partners share ideas, workflows, analytics, and visualizations, collaborating around common interests.
  • 64. FLx: experimental analytics ecosystem Partners share ideas, workflows, analytics, and visualizations, collaborating around common interests.
  • 65. Workflow for social network analytics in NodeXL: are learners forming effective peer-relationships? Import data into OpenRefine Reshape using template Export data to CSV Process in NodeXL and generate network Martin Hawksey http://mashe.hawksey.info/2013/02/lak13-recipes-in-capturing-and-analyzing-data-using-sna-on-canvas-discussions-with-nodexl-for-when-its-not-a-snapp
  • 66. Workflow for quantifying reflections in forum posts: what elements of reflection are evident? Convert discussion threads in comma-separated file format Annotate text segments using custom components for UIMA Convert results in CSV Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing Automatically Detected Reflective Texts with Human Judgements. In 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning. Presented at the 7th European Conference on Technology-Enhanced Learning, Saarbruecken, Germany. Retrieved from http://ceur-ws.org/Vol-931/paper8.pdf Inspect and analyse data with R Reason over annotations with Drools
  • 67. Workflow for academic writing analytics: to what extent does student writing display the hallmarks of scholarly argument? Extract submitted essay drafts from Course XYZ Convert to text files for XIP Analyse using rhetorical parser Render in custom dashboard Annotate onto source text Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013). Open Access Eprint: http://oro.open.ac.uk/37391
  • 68. Thank you… Q&A Belinda Tynan http://www.open.ac.uk/about/main/admin-and-governance/executive-team/pro-vice-chancellor-learning-and-teaching Simon Buckingham Shum http://simon.buckinghamshum.net / @sbskmi