The document discusses how online learner activity analytics can be used to measure and understand student engagement, and how this data can inform learning design. It presents various activities aimed at analyzing learner behavior, the significance of personalized learning frameworks, and explores the implementation outcomes of such approaches. Additionally, it emphasizes identifying and tracking engagement metrics to enhance educational experiences.

1. Learner Enhanced
Technology
Can activity analytics support understanding engagement a measurable process?
James Ballard – HETL SOTE 2015, Utah Valley University
jameslballard
JamesBallard2
@jameslballard

2. My Research
Questions
What can online
learner activity
tell us about
engagement?
How can this be
used to inform
learning design?
“As we start a new school year, Mr. Smith, I just want you
to know that I’m an Abstract-Sequential learner and trust
that you’ll conduct yourself accordingly!”

3. Overview
Icebreaker activity
Introduction
Activity 1 – Classification
Activity 2 – Engagement Profile
Conclusion/Discussion

4. Icebreaker Activity
Should engagement constitute a formal assessment of
teaching?
5 mins – introduce yourselves and give initial reactions to this question

5. Introduction
Personalised Learning and Engagement

6. The challenge
Course
Delivery
(narrow
curriculum)
Learner
Experience
(broad
curriculum)
Personalised
Learning
Framework

7. Personalised Learning Framework
Ballard & Butler (2011)

8. Integrated VLE
Live@EDUTurnitin
Twitter
YouTube SlideShare
Diigo
Facebook
Ning
GoogleDocs
Blogger
Recognised: user’s choice - watch and
learn
Recommended: user’s choice – train and
integrate
Arranged: institutional provision with
training
Core: institutional provision with
compulsory use and consistent access
LRT @ MMU
Implementation Approach

9. Outcomes
JISC Transforming
Curriculum Delivery
Retention: 62% to 92%
Achievement: 69% to 73%
Anecdotal evidence that students felt more
supported
JISC Distributed Learning
Environments
Student satisfaction for course organisation
up 1.8%
Student satisfaction for appropriate
resource selection up 3.5%
Guardian University Award for Student
Experience 2013
EUNIS Elite Award for Excellence 2014
LeSoCo College 2008-2010 MMU 2010-2011

10. PERFORMATIVITY AND POWER
Constant supervision and correction
constrains learners through the need
to achieve conformity
HUMANISM AND DEMOCRATIC EDUCATION
Reflection
Ballard (2014)
Problem
solving
Human
expression
Lived
experience
Can we design learning
systems that promote more of
the right than the left?

11. Learner Voice – Design for Engagement
Teacher Learner

12. Involvement Interaction Intimacy Influence
Engagement as Metric
What to track
• Site visits
• Time spent
• Pages viewed
• Search
keywords
• Navigation
paths
• Site logins
• Contributed
comments to
blogs
• Quantity /
frequency of
written
reviews, blog
comments,
forum posts
• Sentiment
tracking on 3rd
party sites
• Sentiment
tracking of
internal sites
• Opinions
expressed in
customer calls
• Net Promoter
(NP) score
• Service
satisfaction
ratings
• Brand affinity
• Content
forwarded to
friends
How to track
• Web Analytics • eCommerce
• Social Media
• Surveys
• Brand
monitoring
• Service calls
• Surveys
• Brand
monitoring
Haven & Vittel (2008)
Haven (2007)

13. Engagement
Profile
Can this be
appropriated to
educational
technologies?
Haven (2007)

14. Delineation
• Extract logs
• Summary statistics
Activity
Analysis
• Activity Structure
• Activity System
Instrument
Formation
• Pedagogical Model
• Task types
Application
• Engagement profile
Methodology

15. Further Research

16. Activity 1 – Instrument
Formation
What might we measure?

17. Conole’s Task Types
Assimilative
• attending and understanding content;
Information Handling
• gathering and classifying resource or manipulating data;
Adaptive
• use of modelling or simulation software;
Communicative
• dialogic activities (e.g. pair dialogues or group based discussion);
Productive
• construction of an artefact such as a written essay, new chemical compound or a sculpture;
Experiential
• practising skills in a particular context or undertaking an investigation

18. Extracted Logs Sample
Date Course Tool Action
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Course View
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Course View section
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Forum View discussions
03/01/2014 10:03 CIH L3 Cert in Housing
Practice Programme
Forum Add post
03/01/2014 10:04 CIH L3 Cert in Housing
Practice Programme
Forum View discussion
09/01/2014 08:51 CIH L3 Cert in Housing
Practice Programme
Assignment View
09/01/2014 08:52 CIH L3 Cert in Housing
Practice Programme
Assignment View confirmation form
09/01/2014 08:53 CIH L3 Cert in Housing
Practice Programme
Assignment Submit

19. Case Study Activity 1
• Extracted log data is
provided for a learner;
• For each log decide which of
Conole’s task types this
represents - keep a tally in
the worksheet;
• A log may be none of the
types and excluded but
cannot more than one type;
• (Optional) Complete the pie
and/or column charts
Activity 1 Discussion
• What type of tasks is this
learner undertaking?
• How might this be used to
inform learning design?
• What other classifications
might you use?
In Small Groups: 20 minutes

20. Activity 2 – Practical
Application
Creating an Engagement Profile

21. Involvement Interaction Intimacy Influence
Appropriating Engagement Metrics
What to track
• Site visits
• Time spent
• Pages viewed
• Search
keywords
• Navigation
paths
• Site logins
• Contributions
to class
• Quantity and
frequency of
content
• Type of
content
accessed
• Learning
dispositions
• Discourse
tracking Social
network
analysis
• Evaluation
surveys
• Links to
outside world
• Extra-
curricular
contributions
• Practice based
research and
reflection
How to track
• System Logs
• Web Analytics
• System Logs
• Learning
Design
• Surveys
• Content
analysis
• Student
Development
Model

22. Case Study Activity 2
• Data for two students on are
provided – focus on student 1.
• Use the engagement metric to
make notes under each phase
of the relevant profile in the
worksheet;
• Rate each phase from low to
high at the bottom;
• (Optional) Make notes on
extra data you would like to
have for each phase
Activity 2 Discussion
• What can online learner
activity tell us about
engagement?
• If we extend the data we
collect can this be used to
design for engagement?
• What other types of data
would be useful for
engagement?
In Small Groups: 35 minutes

23. INVOLVEMENT INTERACTION INTIMACY INFLUENCE
• Above average
activity on site, but
below in course.
• Regular access
throughout year
• Primarily course
access
• Accessed
throughout day
• Interacts with peers
via forums
• Submitted
assignments
• Apparent focus on
low commitment
activities
• Potential strategic
learning strategy
• Forum posts
directed to or in
response to teacher
• Low peer-to-peer
posts
• No shared links
High Low-Medium Low-Medium Low
Student Profile 1: Jo March

24. INVOLVEMENT INTERACTION INTIMACY INFLUENCE
• Anomalously
high activity
profile
• Intermittent
access
• Interacts
predominantly
with course
materials
• No assignment
submissions
• Potential surface
learning strategy
• Peer interactions
not present
• No shared links
Low Low Low Low
Student Profile 2: Oliver Twist

25. Learner Enhanced Technology
Understanding engagement as a
measurable process
Thank You
James Ballard
jameslballard@outlook.com
JamesBallard2
@jameslballard
jameslballard