This document discusses learner engagement and how it can be used as a metric for learning. It defines engagement as an alternative view of the learner experience that goes beyond just performance or skills. Engagement is described as a developmental process of discovery, evaluation, use, and affinity. The document outlines a research project to develop web dashboards based on an engagement process that analyzes learner data and activity using analytics to help identify effective teaching practices and support learning design. It proposes measuring engagement through involvement, interaction, and intimacy data.

1. Learner Engagement
A Metric for Learning
James Ballard
jameslballard
JamesBallard2
@jameslballard

2. LONDON -> MELBOURNE: NOV 2013 SE ASIA: DEC 2012 – SEP 2013
About Me

3. e-Learning Specialist (2005-
2012)
DfES ICT Test Bed Project
JISC/BECTA Transforming Curriculum
Delivery
CETIS Interoperability: Repository and
Portfolio
JISC Distributed VLEs
HEFCE/JISC Assessment and Feedback
About Me
E-Learning Solution Design and
Evaluation

4. Overview

5. Education is not a
technological issue,
it is a cultural
challenge
We-learning
http://goo.gl/h3HdND

6. Drivers
Shifting Paradigms

7. Personalisation
‘The logic of education systems should be
reversed so that it is the
, rather than the
learner to the system. This is the essence
of personalisation’.
Personalisation and Digital Technologies

8. Harnessing Technology: Next Generation Learning
A shift towards more personalised learning is fundamental and will entail significant
development of practice with technology.
BECTA (2008)

9. 21st Century Learning
It is a world in which knowing what and how to learn the next thing is as important as
what has already been learnt.
Jackson and Ward, 2004

10. 21st Century Work
Internet enables hyper-competition and destroys geographical barriers. Longer term
value today resides in non-standardized work that requires creativity, imagination
and innovation.

11. We-Learning
Is there a widening gap between a technologically constraining society and its
alienated individuals in search of an authentic community?
Shōnen Sunday
(1969)
Computopia
An improved life for
all
Big Brother
Society
Norms reproduced
by elusive forces

12. Personalised
Learning
A Framework for e-Learning

13. Personalised Learning
Social
Personal
Institutional
National
Professional

14. Personalised Learning
Social
Personal
Institutional
National
Professional

15. Personalised Learning
Social
Personal
Institutional
National
Professional

16. Personalised Learning
Social
Personal
Institutional
National
Professional

17. Personalised Learning
Social
Personal
Institutional
National
Professional

18. Personalised Learning
Social
Personal
Institutional
National
Professional

19. Personalised Learning
Social
Personal
Institutional
National
Professional

20. Personalised Learning Framework
Professional
Social
National
Institutional
Personal
Portal
VLE /
LMS
Portfolio
Personal
Learning
Plan (PLP)
Accreditation

21. Transforming
Curriculum Delivery
Lewisham College
eme.lewisham.ac.uk

22. Embracing
change
through
innovation
Through the use of existing and
emerging technologies we will
re-model learning, teaching,
support and business practices
resulting in outstanding,
efficient and effective
performance.
e-
College
Recruitment
Diagnostics
Learning
Assessment
Support
Systems
Digital
Literacy
Comms

23. PLP
Plan – enable learners to work with tutors and support staff to articulate
personal goals, set realistic targets and evaluate their progress towards these.

24. VLE
Learn – offer learners access to course materials, the learning resource centre,
assessment tools and collaboration using forums, discussion boards and chat.

25. E-Portfolio
Reflect – allow learners to store their work, create CVs, reflect on learning
through blogs and showcase what they have produced to tutors, peers and
prospective employers.
Reflection
Mixed Media
Learning
Logs
(process)

26. Outcomes
1. Participation, retention and
achievement
2. Quality improvement
3. Assessment and feedback
‘One of the most important
achievements of the
programme was to instil into
institutions the importance of
sustaining and embedding their
projects.’
Professor Peter Chatterton
JISC Report

27. Ownership
Empowering environments for learning

28. Empowerment
Developing empowering cultures
requires skill, thought and a
commitment to valuing and
exploring processes, not just
focusing on the ‘product’
- Moss and Huntingdon (2004)

29. Applying
market
consumerism
to education
may
compromise
the principles
of equity on
which it is
based
Values such as
self-motivation,
self-regulation,
and
educational
progress, are
not equally
distributed
among cultures
Disadvantaged
learners are
least likely to
seek help
Poorly
structured
choice may
actively reduce
the scope for
the collective
action
Contradictions

30. Ownership Matrix
An independent learner becomes an outcome of education and not a pre-
requisite for it.

31. COST OF PERSONALISATION
In their desire to become customer
driven, many companies have resorted
to inventing new programs and
procedures to meet every customer’s
request. But as customers and their
needs grow increasingly diverse, such
an approach will add unnecessary cost
and complexity to operations.
Pine & Gilmore (1997)
FOUR APPROACHES TO CUSTOMISATION
Transpare
nt
Differentiate
activities
based on
learning
styles/needs.
Collaborati
ve
Learners
participate in
design and
reflection.
Adaptive
Standard
tools create a
unique
learning path.
Cosmetic
Presented
differently, not
fundamentally
altered.
Fours Faces of Customisation
No change Change
NochangeChange
Representation
ActivityDesign

32. Expanding the Framework
Tool Process
(Decision Making)
Ownership
(Customisation)
Assessment
VLE Rational Transparent Dynamic
E-PLP Judgemental Adaptive Formative
Accreditation Political Cosmetic Summative
E-Portofolio Complex Collaborative Portfolio
Ballard and Butler (2011)

33. Enhancing Assessment
• Engagement Tracking
• Log activity analysis
• Completion/Progress
Tracking
Dynamic
• Learning/Development
Plans
• Peer Review
• Feedback
Formative • Blind/Double Marking
• Group Submission
• Peer Assessment
• Criteria-based
assessment
Summative
• Creative processes
• Reflective practice
• Lifelong Learning
Portfolio
Martin (2008) Black (2001)

34. Distributed Learning
Environments
Manchester Metropolitan University
lrt.mmu.ac.uk

35. Distributed Learning Environment

36. Integration
Strategy
Recognised: user’s choice -
watch and learn
Recommended: user’s choice
– train and integrate
Arranged: institutional
provision with training
Core: institutional provision
with compulsory use and
consistent access
LRT @ MMU
Recognised
Recommend
ed
Arrange
d
Core
Integrated
VLE
Live@ED
U
Turnitin
Twitter
YouTub
e
SlideSha
re
Diigo
Faceboo
k
Ning
GoogleDo
cs
Blogge
r

37. My MMU
Automated spaces for faculty, programme, study skills and module areas

38. Seamless System Integration
Integrated notifications, calendar and tailored assessment, reading and resource
lists.

39. Outcomes
1. Courses are well organised and
running smoothly
2. University resources are appropriate
to my learning needs
3. Enhance capability of VLE to meet
user emerging user requirements
4. Heighten awareness about patterns
of student engagement and success
‘Incremental change is not
enough. You have to drive
large-scale change by
changing the environment in
which people work’
– Kevin Bonnett, Deputy Vice
Chancellor Student Experience
JISC Report
MMU Review

40. Learner Engagement
A metric for learning

41. Engagement
Common activities such as checking announcements, viewing grades and uploading
assignments represent little time investment from the user and may not be useful indicators of
engagement.

42. Engagement
Engagement has emerged as an
alternative view of the learner experience
that can enrich the often reductionist
language of performance, skills and
competence.
HEA, Trowler and Trowler (2010)

43. Engagement Process
Engagement is the new metric that supersedes previous linear metaphors, through
a developmental process of discovery, evaluation, use, and affinity.
Haven (2007)

44. Research
Project
Web dashboards based on engagement
process accessing a data warehouse
model developed from Activity Theory.
Utilises new and existing analytics and
supports multiple learning design
approaches.
1. How can student activity help identify
and promote effective teaching
practices?
2. Understand the role that analytics can
play in learning design, feedback and
assessment.
If patterns of nonparticipation
(disengagement) are to be
disrupted an improved
conceptual framework may be
necessary.

45. Involvement
The presence of a learner
within the institution including
data such as physical or virtual
visits.
 Overall Activity
 Locations
 Time of day

46. Involvement
The presence of a learner
within the institution including
data such as physical or virtual
visits.
 Overall Activity
 Locations
 Time of day

47. Interaction
Provides a depth of
understanding: where
involvement measures touches,
interaction measures actions.
 Activity types
 Action analysis
 Connectivity maps
Conole (2007)

48. Activity
Analysis
Engeström’s (1987, 1999)
approach allows us to
overcome oppositions between
activity and communication and
highlight subject-community
relations.
Modelling pedagogy with
Activity Theory
Stevenson (2008)
http://goo.gl/vOuiqp

49. Exposing
Activity
The intention of this is to reveal
the nature of the system,
allowing designers (e.g.
teachers) to evaluate the
system in the wider context of
their teaching and learning
practice.

50. Data Model
Dimension Fact
Action Post to forum
Tool Forum
Instance Discussion topic
User Oliver Twist
Role Student
Course Introduction to English
Date 02/10/2013
Time 9:45
System Moodle
Enables multi-dimensional
tagging to explore data from
different perspectives.

51. Coding
One can then begin to
distinguish the possible actions
that are generated through the
use of tools from the operations
needed to access them and
code these via learning design
theories.

52. Interaction
Provides a depth of
understanding: where
involvement measures touches,
interaction measures actions.
 Activity types
 Action analysis
 Connectivity maps
Conole (2007)

53. Interaction
Provides a depth of
understanding: where
involvement measures touches,
interaction measures actions.
 Activity types
 Action analysis
 Connectivity maps

54. Intimacy
Helps understand sentiment
and or affection; the most
common way to collect this
type of data is through
interviews or surveys.
 Learning Power
 Self-theory
 Motivated Strategies for
Learning Questionnaire,
MSLQ
 Self-determination theory
Deakin Crick, Broadfoot, and Claxton (2004)

55. Intimacy
Helps understand sentiment
and or affection; the most
common way to collect this
type of data is through
interviews or surveys.
 Learning Power
 Self-theory
 Motivated Strategies for
Learning Questionnaire,
MSLQ
 Self-determination theory
0
1
2
3
4
5
6
Rehearsal Elaboration Organisation Self-Regulation Critical Thinking
MSLQ
Pre Post
Pintrich (1990)

56. Influence
Determines the likelihood of the
individual recommending
learning to others and
contributing to local culture(s).
 Social Network Analysis
 Distributed Cognition
 Collective Intelligence
 Pathway of Participation
Dawson (2010)

57. Influence
Determines the likelihood of the
individual recommending
learning to others and
contributing to local culture(s).
 Social Network Analysis
 Distributed Cognition
 Collective Intelligence
 Pathway of Participation
School Leader Network
Harré (1983)

58. Summary
There’s probably a long way to go with
learners generally
... I think it’s simply that we haven’t got far
enough down the line yet with the whole
situation.