ascilite-webinar-oct2012

Ascilite Webinar, Oct 2012

Our Learning Analytics
are Our Pedagogy
Simon Buckingham Shum @
http://twitter.com/sbskmi
Knowledge Media Institute, The Open University UK
http://simon.buckinghamshum.net

1

learning objective:
walk out with
better questions
+ lightning overview of learning analytics

+ glimpses of how analytics might nurture
learning for the new terrain we face

2

Musicality ≠ Musical Reproduction

In those early days the children were taught from the start to develop
their own voice, whether literally singing, or through the
instrument they played. They were not taught music,

but musicality. Central to this tuition were the partimenti, many
pages of detailed music notes which pose many questions,

but leave the pupil to find the solutions. The
music is not a literal transcript, which the musician reads and reproduces.
set of rules and then
The partimenti establish, at the start, a

pose a set of conflicts for the musician to
resolve, in their own way.
3
http://bit.ly/onmusicality

is education
poised to become a
data-driven
enterprise
and science ? 4

Possibly 90% of the digital data we have
today was generated in the last 2 years

Volume outstrips old infrastructure

Variety Internet of things, e-business transactions, environmental
sensors, social media, audio, video, mobile…

Velocity The speed of data access and analysis is exploding

A quantitative shift on this scale is in fact a qualitative shift, requiring
new ways of thinking about
societal phenomena
5

edX: “this is big data, giving us the chance
to ask big questions about learning”

Will the tomorrow’s
educational researcher be
as helpless without an
analytics infrastructure, as
a geneticist without
genome databases, or a
physicist without CERN? 6

Lifelogging: explosion of data capture
and sharing about personal activities

http://www.mirror-project.eu

http://quantifiedself.com/guide 7

Educational Data Mining research community

Learning Analytics research community

Learning Analytics research community

http://www.educause.edu/library/learning-analytics

different levels
of analytic

11

‘Learning Analytics’ and
‘Academic Analytics’

Long, P. and Siemens, G. (2011), Penetrating the fog: analytics in learning and education. Educause Review Online,
46, 5, pp.31-40. http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education 12

Macro/Meso/Micro Learning Analytics

Macro:
region/state/national/international


Macro:

Meso:
institution-wide


Macro:

Meso:
institution-wide

Micro:
individual user actions
(and hence cohort)

Will institutions be dazzled by the
dashboards, or know what
questions to ask at each level?

US states are getting the infrastructure
in place
dataqualitycampaign.org

17

Shared Learning Collaborative
http://slcedu.org

18

National league tables for English schools

19


Meso:
institution-wide

Business Intelligence companies see an
education market opening up

These are pedagogically agnostic:
they seek to optimize operational
efficiency whatever the sector

These may make pedagogical
assumptions: how will learning
design and assessment regimes
shape the analytics they offer?

http://www.sas.com/industry/education/highered 21

Business Intelligence companies see an
education market opening up

…but do they know anything about
the roles that language plays in
learning and knowledge
construction? 22

BI+HigherEd communities of practice

23


Micro:
(and hence cohort)

Analytics in your VLE:
Blackboard: feedback to students
http://www.blackboard.com/Platforms/Analytics/Overview.aspx

25

Purdue University Signals: real time traffic-
lights for students based on predictive model

Premise: academic success is defined as a function of
aptitude (as measured by standardized test scores and
similar information) and effort (as measured by participation
within the online learning environment).

Using factor analysis and logistic regression, a model was
tested to predict student success based on:

•  ACT or SAT score
•  Overall grade-point average
Predicted 66%-80% •  CMS usage composite
of struggling •  CMS assessment composite
students who •  CMS assignment composite
needed help •  CMS calendar composite

Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE
Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x 26

Desire2Learn visual analytics & predictive models
which can be interrogated on different dimensions
http://www.desire2learn.com/products/analytics

27

Desire2Learn visual analytics & predictive models
which can be interrogated on different dimensions
http://www.desire2learn.com/products/analytics

28

Socrato: train for SATs

http://www.socrato.com 29

Khan Academy: more data to teachers,
finer-grained feedback to students

http://www.thegatesnotes.com/Topics/Education/Sal-Khan-Analytics-Khan-Academy 30

Adaptive platforms generate fine-
grained analytics
https://grockit.com/research

31

Adaptive platforms generate fine-grained
analytics

http://knewton.com

The VLE—BI—ITS convergence

33

Hard distinctions between Learning +
Academic analytics may dissolve
…as they get joined up, each level enriches the others

Macro:

Meso:
institution-wide

Micro:
(and hence cohort)

Aggregation of user traces
enriches meso + macro analytics
with finer-grained process data

Hard distinctions between Learning +
Academic analytics may dissolve
…as they get joined up, each level enriches the others

Macro:

Meso:
institution-wide

Micro:
(and hence cohort)

Aggregation of user traces Breadth + depth from macro
enriches meso + macro analytics + meso levels add power to
with finer-grained process data micro analytics

questions
/comments?

36

but how do we do
analytics for
this kind of learning?...

37

Learning analytics for this?

“We are preparing students for jobs
that do not exist yet, that will use
technologies that have not been
invented yet, in order to solve
problems that are not even
problems yet.”

“Shift Happens”
http://shifthappens.wikispaces.com

38


“While employers continue to demand high academic
standards, they also now want more. They want
people who can adapt, see connections,
innovate, communicate and work with
others. This is true in many areas of work. The new
knowledge-based economies in particular will
increasingly depend on these abilities. Many
businesses are paying for courses to promote creative
abilities, to teach the skills
and attitudes that
are now essential for economic
success…”
All our Futures: Creativity, culture & education, May 1999 39


Think about the analytics
products and initiatives
reviewed above – where
would you locate them
on these dimensions?

Creativity, Culture and
Education (2009)
Changing Young Lives
2012. Newcastle: CCE.
http://www.creativitycultureeducation.org/
changing-young-lives-2012 40


The Knowledge-Agency Window
co-generation

Expert-led enquiry Student-led enquiry
Knowledge

and use

Teaching as
Authenticity
learning design
Agency
Identity

Repetition,
Pre-scribed
Knowledge

Abstraction
Acquisition

Expert-led teaching Student-led revision

Teacher agency Student agency

Ruth Deakin Crick, Univ. Bristol, Centre for Systems Learning & Leadership
“Pedagogy of Hope”: http://learningemergence.net/2012/09/21/pedagogy-of-hope

analytics grounded in the
principles of good
assessment
for learning?
(not summative assessment for
grading pupils, teachers,
institutions or nations)

42

Assessment for Learning Few learning analytics are
http://assessment-reform-group.org currently able to take o
board the richness of this
original conception of
assessment for learning

43

Assessment for Learning
http://assessment-reform-group.org

44


To what extent
could automated
feedback be
designed and
evaluated with
emotional impact
in mind?

45


Can analytics
identify proxies
for such
advanced
qualities? 46


Do analytics provide
constructive next
steps?

47


How do we provide
analytics feedback
that does not
disempower and de-
motivate struggling
learners?

48

Dispositional
Learning Analytics

49

Musicality ≠ Musical Reproduction

In those early days the children were taught from the start to develop
their own voice, whether literally singing, or through the
instrument they played. They were not taught music,

but musicality. Central to this tuition were the partimenti, many
pages of detailed music notes which pose many questions,

but leave the pupil to find the solutions. The
music is not a literal transcript, which the musician reads and reproduces.
set of rules and then
The partimenti establish, at the start, a

pose a set of conflicts for the musician to
resolve, in their own way.
50
http://bit.ly/onmusicality

Dispositions are important

“Knowledge of methods alone
will not suffice: there must be
the desire, the will, to employ
them. This desire is an affair
of personal disposition.”

John Dewey, 1933

Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to the
Educative Process. Heath and Co, Boston, 1933
51


“The test of successful education
is not the amount of knowledge
that pupils take away from school,
but their appetite to know and
their capacity to learn.”

Sir Richard Livingstone, 1941

52


Slide from Guy Claxton: http://www.scribd.com/doc/26685380/Guy-Claxton-Learning-to-Learn
Perkins, D.N., Jay, E., & Tishman, S. (1993). Beyond abilities: A dispositional theory of thinking. Merrill- 53
Palmer Quarterly: Journal of Developmental Psychology, 39(1): 1-21.

Dispositions are beginning to register
within the learning analytics community

Brown, M., Learning Analytics: Moving from Concept to Practice. EDUCAUSE Learning Initiative
Briefing, 2012. http://www.educause.edu/library/resources/learning-analytics-moving-concept-practice 54

In your experience, what are the qualities
shown by the most effective learners?

Think about the most effective learners you’ve met/
mentored/taught

Not necessarily the highest grade scorers, but the ones
who showed a sustained appetite for learning

What qualities/dispositions/attitudes did they bring?

Type a few key words
into the textchat…
55

A ‘visual learning analytic’
7-dimensional spider diagram of how the learner sees themself

Basis for a mentored-
discussion on how the
learner sees him/herself,
and strategies for
strengthening the profile

56
Bristol and Open University are now embedding ELLI in learning software.

ELLI: Effective Lifelong Learning Inventory
Web questionnaire 72 items (children and adult versions: used
in schools, universities and workplace)

57

Validated as loading onto
7 dimensions of “Learning Power”

Being Stuck & Static Changing & Learning
Data Accumulation Meaning Making
Passivity Critical Curiosity
Being Rule Bound Creativity
Isolation & Dependence Learning Relationships
Being Robotic Strategic Awareness
Fragility & Dependence Resilience

Univ. Bristol and Vital Partnerships provides practitioner resources
and tools to support their application in schools and the workplace 58

Learning to Learn: 7 Dimensions of Learning Power
Factor analysis of the literature plus expert interviews: identified seven
dimensions of effective learning power , since validated empirically with
learners at many levels. (Deakin Crick, Broadfoot and Claxton, 2004)

Learning to Learn: 7 Dimensions of Learning Power
Factor analysis of the literature plus expert interviews: identified seven
dimensions of effective learning power , since validated empirically with
learners at many levels. (Deakin Crick, Broadfoot and Claxton, 2004)

60

Learning Warehouse 2.0 analytics platform

User experience:
Research-validated assessment tools
Researcher interface
Learning Communities
Analytics:
Real time ELLI Analytics reports
Bespoke research reports

Datasets:
>40,000 ELLI profiles
(data from other hosted apps)

61

Adding imagery to ELLI dimensions to
connect with learner identity

62

Working with Gappuwiyak School, N. Territory AUS
(Ruth Deakin Crick, University of Bristol) http://bit.ly/srUSHE

Changing & Learning: Strategic Awareness:
The Drongo - Guwak Emu - Wurrpan

Meaning Making:
The Pigeon - Nabalawal

Critical Curiosity:
Sea Eagle - Djert

Resilience:
Brolga - Gudurrku

Learning Relationships: Creativity:
The Cockatoo - Ngerrk Bower Bird - Djurwirr 63

Cohort analytics for
educators and
organizational leaders

64

EnquiryBlogger:
Tuning Wordpress as an ELLI-based learning journal

Standard Wordpress editor

Categories from ELLI

Plugin visualizes
blog categories,
mirroring the ELLI
spider

65

Primary School EnquiryBloggers
Bushfield School, Wolverton, UK

EnquiryBlogger: blogging for Learning Power & Authentic Enquiry
http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry

Could a platform generate an
ELLI profile from user traces?

Different social
network patterns
Questioning and
in different
challenging may
contexts may
load onto Critical
load onto
Curiosity
Learning
Relationships

Repeated
Sharing relevant attempts to pass
resources from an online test
other contexts may load onto
may load onto Resilience
Meaning Making

Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn
http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium

SocialLearn provides new possibilities of
looking at learners learning

ELLI works from what Now we can observe what
learners say they do they actually do…

69

ELLI feedbacks inform development of
learning

Educator or
leader s
interventions

Mentored
discussions

70

How about SocialLearn learning disposition
analytics?

How do these
feedbacks help
people learn?

What and where What kind of feedback
should we look at? should we provide?
Will we still have What is the most appropriate
seven dimensions? way to do it?

71

LearningEmergence.net: embedding
dispositional analytics into practice + tools
EnquiryBlogger: Wordpress plugins for reflective learning journals

72

Analytics for
learning conversations

73

Socio-cultural discourse analysis
(Mercer et al, OU)

•  Disputational talk, characterised by disagreement and
individualised decision making.

•  Cumulative talk, in which speakers build positively but
uncritically on what the others have said.

•  Exploratory talk, in which partners engage critically but
constructively with each other's ideas.

Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
74

Socio-cultural discourse analysis
(Mercer et al, OU)

•  Exploratory talk, in which partners engage critically but
constructively with each other's ideas.
•  Statements and suggestions are offered for joint consideration.

•  These may be challenged and counter-challenged, but challenges are
justified and alternative hypotheses are offered.

•  Partners all actively participate and opinions are sought and considered
before decisions are jointly made.

•  Compared with the other two types, in Exploratory talk knowledge is made
more publicly accountable and reasoning is more visible in the talk.

Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
75

Analytics for identifying Exploratory talk

Elluminate sessions can
be very long – lasting for
hours or even covering
days of a conference

It would be useful if we could
identify where quality learning
conversations seem to be taking
place, so we can recommend
those sessions, and not have to
sit through online chat about
virtual biscuits

Ferguson, R. and Buckingham Shum, S. Learning analytics to identify exploratory dialogue within synchronous text chat. 76
1st International Conference on Learning Analytics & Knowledge (Banff, Canada, 27 Mar-1 Apr, 2011)

Defining indicators of Exploratory Talk

Category Indicator
Challenge But if, have to respond, my view
Critique However, I’m not sure, maybe
Discussion of Have you read, more links
resources
Evaluation Good example, good point
Explanation Means that, our goals
Explicit reasoning Next step, relates to, that’s why
Justification I mean, we learned, we observed
Reflections of Agree, here is another, makes the
perspectives of others point, take your point, your view
77

Extract classified as Exploratory Talk

Time Contribution
2:42 PM I hate talking. :-P My question was whether "gadgets" were just
basically widgets and we could embed them in various web sites,
like Netvibes, Google Desktop, etc.
2:42 PM Thanks, that's great! I am sure I understood everything, but looks
inspiring!
2:43 PM Yes why OU tools not generic tools?
2:43 PM Issues of interoperability
2:43 PM The "new" SocialLearn site looks a lot like a corkboard where you
can add various widgets, similar to those existing web start pages.
2:43 PM What if we end up with as many apps/gadgets as we have social
networks and then we need a recommender for the apps!
2:43 PM My question was on the definition of the crowd in the wisdom of
crowds we acsess in the service model?
2:43 PM there are various different flavours of widget e.g. Google gadgets,
W3C widgets etc. SocialLearn has gone for Google gadgets 78

Discourse analytics on webinar
textchat
Given a 2.5 hour webinar, where in the live
textchat were the most effective learning
conversations?

Not at the start and end of a webinar
Sheffield, UK not as sunny but if we zoom in on a peak… See you!
as yesterday - still warm
bye for now!
Greetings from Hong Kong
bye, and thank you
Morning from Wiltshire,
80
sunny here! Bye all for now

60

40

20

0
9:28
9:32

10:13

11:48

12:00

12:05
12:04
9:36
9:40
9:41
9:46
9:50
9:53
9:56
10:00
10:05
10:07
10:07
10:09

10:17
10:23
10:27
10:31
10:35
10:40
10:45
10:52
10:55
11:04
11:08
11:11
11:17
11:20
11:24
11:26
11:28
11:31
11:32
11:35
11:36
11:38
11:39
11:41
11:44
11:46

11:52
11:54

12:03
-20

-40
Average Exploratory
-60

Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous
Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

Discourse analytics on webinar
textchat

Given a 2.5 hour
webinar, where in the
live textchat were the
most effective learning
conversations?
Classified as
“exploratory
talk”

(more
substantive
100 for learning)

50

0
9:28

“non-
9:40
9:50
10:00
10:07
10:17
10:31
10:45
11:04
11:17
11:26
11:32
11:38
11:44
11:52
12:03

-50 exploratory”

Averag
-100

Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous
Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

Discourse analysis (Xerox Incremental Parser)
Detection of salient sentences in scholarly reports,
based on the rhetorical signals authors use:
BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:
Recent studies indicate … ... new insights provide direct evidence ... … little is known …

… the previously proposed … ... we suggest a new ... approach ... … role … has been elusive
Current data is insufficient …
… is universally accepted ... ... results define a novel role ...

CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:
… unorthodox view resolves … studies ... have provided important The goal of this study ...
paradoxes … advances Here, we show ...
In contrast with previous Knowledge ... is crucial for ... Altogether, our results ... indicate
hypotheses ... understanding
... inconsistent with past findings ... valuable information ... from studies

GENERALIZING: SURPRISE:
... emerging as a promising approach We have recently observed ...
surprisingly
Our understanding ... has grown
exponentially ... We have identified ... unusual
... growing recognition of the The recent discovery ... suggests Ágnes Sándor & OLnet Project:
http://olnet.org/node/512
intriguing roles
importance ...

De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine
Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

Human and machine analysis of a text for key
contributions

Document 1 19 sentences annotated 22 sentences annotated
11 sentences same as human annotation
Document 2 71 sentences annotated 59 sentences annotated
42 sentences same as human annotation

http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine
Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

KMi’s Cohere:
a web deliberation platform enabling semantic social
network and discourse network analytics

Rebecca is playing
the role of broker,
connecting 2 peers’
contributions in
meaningful ways

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st
International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

Discourse Network Analytics =
Concept Network + Social Network Analytics

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st
International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

Closing thoughts

85

“The basic question is not
what can we measure?

The basic question is
what does a good education look like?”

(Gardner Campbell)

http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college
http://lak12.wikispaces.com/Recordings 86

Our analytics promote
values, pedagogy and
assessment regimes.

Are we clear which master
our analytics serve? Are we
happy to be judged by them?

87

LAnoirblanc.tumblr.com
reactions to Learning Analytics in image and story

Choose an image and email it to the site with your story…
Instructions: h"p://www.educause.edu/sites/default/ﬁles/library/presenta7ons/ELI124/GS13/LAnoirblanc.pdf

ascilite-webinar-oct2012

More Related Content

What's hot

Similar to ascilite-webinar-oct2012

More from Simon Buckingham Shum

Recently uploaded

ascilite-webinar-oct2012