Understanding engagement as a measurable process workshop at #ellta2014

1. Learner Enhanced
Technology
Understanding engagement as a measurable process
James Ballard
JamesBallard2
@jameslballard
jameslballard

2. My Research
Questions
Is engagement
something we can
measure?
Should we be
measuring it?
Should it
constitute formal
assessment of
teaching?
“As we start a new school year, Mr. Smith, I just want you
to know that I’m an Abstract-Sequential learner and trust
that you’ll conduct yourself accordingly!”

3. Overview
Introduction
• Icebreaker activity
• Why is engagement important?
• Where does the metric idea originate?
Activity 1 – What should we look for?
• Introduce an engagement matrix
• Each group to identify and discuss engagement patterns
Activity 2 – How can we measure this?
• Introduce an engagement framework metric
• Each group to explore a data driven case study
Conclusion/Discussion

4. Icebreaker Activity
Should engagement constitute a formal assessment of
teaching?
5 mins – introduce yourselves and give initial reactions to this question

5. Introduction 1
Why is engagement important?

6. Improved Learning Outcomes
Practical competence
and skills transferability
Cognitive
development
General abilities and critical
thinking
Self-esteem, psychosocial
development, productive
racial and gender identity
formation
Trowler and Trowler (2010)

7. Multicultural Classrooms
UNESCO (2014): Global Flow of Tertiary Students

8. Learner Voice
Teacher Learner
Ballard & Butler (2015)

9. Introduction 2
Where does measuring engagement come from?

10. Engagement Surveys
During the current school year, about how often have you had discussions about the following?
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Economic or social inequality
Issues of race, ethnicity, or nationality
Religious or philosophical differences
Different political viewpoints
Issues of gender or sexual orientation
Never Sometimes Often Very often
NSSE 2014: Experiences with different perspectives

11. Social Marketing: Engagement as Metric
Involvement Interaction Intimacy Influence
What to track
• Site visits
• Time spent
• Pages viewed
• Search
keywords
• Navigation
paths
• Site logins
• Contributed
comments to
blogs
• Quantity /
frequency of
written
reviews, blog
comments,
forum posts
• Sentiment
tracking on 3rd
party sites
• Sentiment
tracking of
internal sites
• Opinions
expressed in
customer calls
• Net Promoter
(NP) score
• Service
satisfaction
ratings
• Brand affinity
• Content
forwarded to
friends
How to track
• Web Analytics • eCommerce
• Social Media
• Surveys
• Brand
monitoring
• Service calls
• Surveys
• Brand
monitoring
Haven & Vittel (2008)
Haven (2007)

12. Social Marketing
Haven (2007)

13. Activity 1
What would we want to measure?

14. Dimensions of Engagement
ENGAGEMENT TYPES
FREDRICKS, BLUMENFELD AND PARIS (2004)
ENGAGEMENT POLES
TROWLER AND TROWLER (2010)
• Tends to result in expected
learning behaviours
Positive
Engagement
• A passive rejection of such
behaviours – alienation or inertia
Non-engagement
• An active and hostile rejection of
expectations
Negative
Engagement
• Participation in academic and
social activities Behavioural
Emotional • Feelings towards staff and peers
• Willingness to invest effort in
complex tasks and mastery of
skills
Cognitive

15. Discussion Activity: Small Groups
15 mins – Discuss: “how do we recognise when students are engaged or not?”
POSTIVE ENGAGEMENT NON-ENGAGEMENT NEGATIVE ENGAGEMENT
BEHAVIOURAL Attends lectures, participates with
enthusiasm
Skips lectures without excuse Boycotts, pickets or disrupts lectures
EMOTIONAL Interest Boredom Rejection
COGNITIVE Meets or exceeds assignment
requirements
Assignments late, rushed or
absent
Redefines parameters for
assignments
5 mins - Feedback an example to the wider audience
Examples and prompts:

16. Activity 2
How can we measure this?

17. Appropriating Engagement Metrics
Involvement Interaction Intimacy Influence
What to track
• Site visits
• Time spent
• Pages viewed
• Search
keywords
• Navigation
paths
• Site logins
• Contributions
to class
• Quantity and
frequency of
content
• Type of
content
accessed
• Learning
dispositions
• Discourse
tracking Social
network
analysis
• Evaluation
surveys
• Links to
outside world
• Extra-curricular
contributions
• Practice based
research and
reflection
How to track
• System Logs
• Web Analytics
• System Logs
• Learning
Design
• Surveys
• Content
analysis
• Student
Development
Model

18. Discussion Activity: Small Groups
ACTIVITY CASE STUDY AND STUDENT PROFILE
Provides real student data
Make notes in the student profile
Mark each component from low to high
30 mins – Discuss the case study and
complete the 2 student profiles in the
worksheet provided
10 mins - Feedback for the whole
group

19. Student Profile 1: Jo March
INVOLVEMENT INTERACTION INTIMACY INFLUENCE
• Average activity
• Interacts with
• Enthused by
profile
peers via
university visit
• Regular access
messaging
and tour – not
throughout year
• Some portfolio
sustained
• Accessed
building
• Potential surface
different support
• Few course
learning strategy
areas
based activities
• Limited to small
social group -
connected to hub
• Use of 1-to-1
messaging
• No shared links
Medium Low Medium Low

20. Student Profile 2: Oliver Twist
INVOLVEMENT INTERACTION INTIMACY INFLUENCE
• High activity
profile
• Regular access
across 2 years
• Interacts
predominantly
with course
materials
• Several
assignment
submissions
• Potential
strategic learning
strategy
• Peer interactions
not present
• No shared links
High Medium Low Low

21. Further Research

22. Final thoughts
Is engagement something we can measure?
Should we be measuring it?
Should it constitute formal assessment of teaching?

23. Thank You
Learner Enhanced Technology
Understanding engagement as a
measurable process
James Ballard
jameslballard@outlook.com
JamesBallard2
@jameslballard
jameslballard