The Many Flavours of Productive Engagement

We are on the unceded territories of the
xʷməθkwəy̓əm (Musqueam) people

The promise of learning analytics
Learners System Data

Graham Gibbs
What best predicts
educational gain is measures
of educational process
Gibbs, G. (2010). Dimensions of quality. York:
Higher EducaOon Academy.

Graham Gibbs
What best predicts
educational gain is measures
of educational process
…Essentially the crucial
variable is ‘student
engagement’ Dimensions of quality (2010)

Student engagement
?
• Engagement is complex
• …and oSen does not emit data

Outline
• What is engagement?
• Flavours of user engagement
• Affect, behaviour & cognition
• Value
• Interpretation
• Implications for learning analytics

Student engagement is…
…the time and energy students devote to
educationally sound activities inside and outside
of the classroom (Kuh, 2003).
… intended to optimise the student experience
and enhance the learning outcomes and
development of students and the performance,
and reputation of the institution (Trowler, 2010).
Kuh, G. D. (2003). What we’re learning about student engagement from NSSE: Benchmarks for effective educational practices.
Change: The Magazine of Higher Learning, 35(2), 24–32.
Trowler, V. (2010). Student engagement literature review. The Higher Education Academy, 11(1), 1-15.

User engagement
• “A user’s response to an interaction that gains, maintains,
and encourages their attention, particularly when they are
intrinsically motivated” (Jacques, 1996, p. 103)
Low
High
Attention | Motivation |Perceived time | Control |Needs & Attitudes
Jacques, R. D. (1996). The nature of engagement and its role in hypermedia evaluation and design (Doctoral dissertation, South
Bank University).

User engagement
• A quality of user experience (UX) that is
characterized by the depth of an actor’s
cognitive, temporal and/or emotional
investment in an interaction with a digital
system (O’Brien, 2016)
O’Brien, HL. (2016). Translating theory into methodological practice. In HL. O’Brien & P. Cairns (Eds). Why Engagement
Matters: Cross-Disciplinary Perspectives and Innovations on User Engagement with Digital Media (pp. 1-26). Berlin
Heidelberg: Springer.

Student engagement User engagement
Interaction of time,
resources and effort
Investment of time,
emotions and cognition
Within and outside of the
classroom
Within a digitally-mediated
environment
Goal à optimize learning
outcomes
Goal à optimize learning,
spending, searching,
health behaviours, etc.
Reflects institutions’
investment in “student
experience”
Reflects quality of “user
experience”
Kuh, 2003; Trowler, 2010 O’Brien, 2016

THE MANY FLAVOURS OF
ENGAGEMENT

#1. NOW WE KNOW OUR ABC’S
(Affect, Behaviour and Cognition, that is)

Student engagement is…
• Affective à interest, enjoyment, sense of
belonging
• Behavioural à complying with norms such as
attendance and non-disruptive behaviours
• Cognitive à seeking challenge, investing in
learning, doing more than the minimum
required
• Fredricks, Blumenfeld & Paris, 2004
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of
the evidence. Review of educational research, 74(1), 59-109.

A view of student engagement
Low
High
Based on Trowler, 2010
Jacques, 1996
NEGATIVE
A: Frustrated
B: Not attending
class, being
disruptive
C: Not invested or
making an effort
POSITIVE
A: Interested
B: Attending class
C: Persisting,
reflecting, seeking
challenge

Rethinking engagement as
positive vs. negative
• Example 1
– Engagement is a process
• Example 2
– Examining measures of affect, behaviour and
cognition in concert

Ex. 1 - Information Search Process
Model
Tasks Initiation Selection Exploration Formulation Collection Presentation
A Uncertainty Optimism
Doubt
Confusion
Frustration
Clarity Sense of
Direction/
Lack of
confidence
Satisfaction/
Disappoint-
ment
B Seeking Relevant Information-------------------------àSeeking Pertinent Information
Exploring Documenting
C Vague -----------------------------------à Focused
Increased Interest ----------------à
Kuhlthau, C. C. (1991). Inside the search process: Information seeking from the user's perspective. Journal of the American
Society for Information Science, 42(5), 361-371.

Process Model of User Engagement
Point of
engagement
Engagement Disengagement
Re-engagement
O’Brien, HL., & Toms, E. (2008). What is user engagement? A conceptual framework for defining user engagement with
technology. Journal of the American Society for Information Science and Technology, 59(6), 938–955.
(O’Brien & Toms, 2008)
Online shoppers, searchers, learners and
gamers

ISP and User Engagement
A Uncertainty Optimism
Doubt
Confusion
Frustration
Clarity Sense of
Direction/
Lack of
confidence
Satisfaction/
Disappoint-
ment
B Seeking Relevant Information-------------------------àSeeking Pertinent Information
Exploring Documenting
C Vague -----------------------------------à Focused
Increased Interest ----------------à
Point of
engagement
Engagement
Disengagement
Re-
engagement
Point of
engagement
Engagement
Disengagement
Re-
engagement Point of
engagement
Engagement
Disengagement
Re-
engagement
Point of engagement à Engagement à Disengagement à Re-engagement

Ex. 2 – Relating feelings and thoughts
to actions
• Perceived task complexity a better predictor of
engagement than actual complexity
– Post-task knowledge +
– Post-task determinability +
– Pre- and post-task ease +
– Change in self-reported ease −
O’Brien, H.L., Arguello, J. & Capra, R. (under review). An empirical study of task
characteristics and user task perceptions on search engagement.

Making sense of the behavioural data
Rank
- Avg. bookmark
rank
- Avg. click rank
Querying
- Queries without clicks
- unique queries
- Queries
- Query reformulations
- …
Pace of
interaction
- Between events
- To 1st
bookmarký
- Completion time
Bookmarks
SERP
Exploration
- Clicks
- Unique SERP
clicks
- Total Scrolls
- …

Making sense of the behavioural data
Rank
- Avg. bookmark
rank
- Avg. click rank
Querying
- Queries without clicks
- unique queries
- Queries
- Query reformulations
- …
Pace of
interaction
- Between events
- To 1st
bookmarký
- Completion time
Bookmarks
SERP
Exploration
- Clicks
- Unique SERP
clicks
- Total Scrolls
- …
Low user
engagement
High user
engagement

#2. THE VALUE OF ENGAGEMENT
(Making a case for a range of experiences)

Engagement ßà Disengagement
• Student disengagement: a lack of affective
(e.g. decline in interest), behavioural (e.g. lack
of participation) and cognitive (e.g. lack of
attention) engagement (Fredricks, Blumenfeld, & Paris 2004).
• The negative node on the engagement
spectrum
– > detrimental for learning.

Value: Behavioral (dis)engagement
• Pauses and learning
– Physics simulations
(Perez, 2017)
– Help seeking (Shih, 2008)
• Offline engagement
(Dodson, submitted)
– Notebooks
– Peers
Roll, I., Yee, N., & Briseno, A. (2014). Students’ adaptation and transfer of strategies across levels of scaffolding in an exploratory
environment. Proceedings of ITS, pp. 348–353.
Dodson, S., Roll, I., Harandi, N. M., Fels, S., & Yoon, D. (Submitted). Weaving together media, technologies, and people: Students’
information practices in flipped classrooms. Submitted to Information and Learning Sciences.
Shih, B., Koedinger, K. R., & Scheines, R. (2008). A response time model for bottom-out hints as worked examples. Proceedings of
the 1st International Conference on Educational Data Mining, 117–126.
actions.
The topic of the ﬁrst activity was light-bulbs. All students worked with the same
simulation and were given the following focus for their inquiry:
Use the DC Circuit PhET to explore how voltage, current, and the brightness of light
bulbs depend on: 1. The number of light bulbs in a circuit; 2. The arrangement of
light bulbs in circuit. For example, a. What happens when several light bulbs are
placed in the same loop? (that is, all the electrons move through the same
Fig. 1 The PhET D/C Circuit Construction Kit (CCK) simulation. Snapshots taken during the study
123

Value: Affective (dis)engagement
• Frustrated or bored?
– boredom is associated with more detrimental
behaviours, and is more persistent (Baker, 2010)
• Pleasant or unpleasant?
– Depend on our goal (Halbert, 2015)
– Transformative learning (Mezirow)
Baker, R. S. J. d., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored:
The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three
different computer-based learning environments. International Journal of Human Computer Studies 68, 223-
241.
Halbert, H., & Nathan, L. P. (2015). Designing for Discomfort : Supporting Critical Reflection through Interactive
Tools. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social
Computing (CSCW ’15) (Vol. 11, pp. 334–349).

Value: Cognitive (dis)engagement
• Productive failure
• Test enhanced learning
• Impasse-driven learning
Chowrira, S. G., Smith, K. M., Dubois, P. J., & Roll, I. (2019). DIY productive failure: boosting performance in a large undergraduate biology
course. Npj Science of Learning, 4(1), 1. https://doi.org/10.1038/s41539-019-0040-6
Loibl, K., Roll, I., & Rummel, N. (2017). Towards a Theory of When and How Problem Solving Followed by Instruction Supports Learning.
Educational Psychology Review, 29(4), 1–23.
VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. B. (2003). Why do only some events cause learning during human tutoring?
Cognition and Instruction, 2(3), 209–249.
Kornell, N., Hays, M. J., & Bjork, R. A. (2009). Unsuccessful Retrieval Attempts Enhance Subsequent Learning. Journal of Experimental
Psychology: Learning Memory and Cognition, 35(4), 989–998.
p< 0.001. This difference was coincidental and went unnoticed
until analysis of study data. Possibly due to differences in sample
composition, significant differences (by condition) were observed
in regular course exam marks (unrelated to the study). On
those marks, students in the PF condition outperformed their
counterparts on Midterm 1 (5.14 ***p< 0.001, 95% CI = [3.23,
7.06], t(559.31) = 5.28; d= 0.44) and final exam (7.63 ***p< 0.001,
95% CI = [4.75, 10.52], t(567.02) = 5.20; d= 0.44), but not on
Midterm 2, (−1.51, p= 0.23, 95% CI = [-4.01, 0.98], t(561.03) =
−1.19; d= −0.10), as shown in Table 1. In order to account for
these differences, performance on these “other’’ items served as
covariates in our model.
Effects of condition on learning
Student performance on the midterm and final exams was
analyzed, blind to condition. The differences described above
were not unexpected for a quasi-experimental student sample. To
isolate the effects of learning condition on student performance,
we used linear regression to predict scores (0–100) on topics where
teaching approach varied, controlling for scores on Midterm 1,
gender, university year, enrollment in the science program, and
exam scores for questions in the same exam on topics where
teaching was not varied (referred to as “other items”). Thus,
differences between student populations or instructors, as
reflected by performance on other items, were accounted for,
isolating the impact of PF vs. AL on the studied topics. For Midterm
2, (following the implementation of PF) the resulting coefficient for
the PF condition was significant, b= 4.78, 95% CI [2.19, 7.36], t
(567) = 3.63, p< 0.001, with an effect size of 0.32, 95% CI [0.15,
0.49] (derived by dividing the coefficient by the residual standard
error). This shows that students in the PF condition had grades for
those topics almost five percentage-points higher (Fig. 3). The
overall model fit was R2
= 0.43, F(6,567) = 71.95, p< 0.001, which
means that 43% of those condition-topic scores could be explained
by this model. To account for the large number of covariates, we
ran regression models with stepwise variable selection. Results
were nearly identical to those reported above, and thus we focus
on the more straightforward, comprehensible, and complete
points higher than their counterparts in the AL group, even after
controlling for gender, faculty, year, and performance on other
items. This effect is considerable for that group.
The effect of PF on Med and High achieving students does not
reach significance, likely because splitting the population reduces
statistical power. As the coefficients for Med and High were
similar, we examined the combined data for these two groups.
The coefficient for the combined group reaches significance, b=
3.25, 95% CI [0.07, 6.43], t(373) = 2.01, p= 0.045. This suggests
that the PF technique benefitted the lower-performing students
Table 1. Raw exam scores, by condition, showing mean (standard deviation)
Study items Other items
Condition N Midterm 2 Final Midterm 1 Midterm 2 Final
AL (active learning) 279 67.96 (19.81) 66.09 (28.11) 76.23 (12.18) 73.50 (15.82) 64.15 (17.89)
PF (productive failure) 295 73.97 (19.35) 74.31 (25.13) 81.38 (11.08) 71.98 (14.54) 71.78 (17.22)
Fig. 3 Adjusted mean study item grades (0–100) and Standard Error
by Condition, controlling for gender, university year, program,
Midterm 1 grades and Midterm 2 non-study items. Error bars show
standard error. ***p< .001
S.G. Chowrira et al.
4
Impact on Midterm
4.8***
(Chowrira, 2019)
Instruction first Problems first

Value: complete (dis)engagement
• Spacing effect
– Spaced practice is superior to “cramming”.
Kang, S. H. K., & Pashler, H. (2011). Learning painting styles: Spacing is advantageous when it promotes discriminative contrast.
Applied Cognitive Psychology, 26(1), 97-103.

↓ Engagement
↓ Learning
↓ Engagement
↑ Learning
↑ Engagement
↓ Learning
↑ Engagement
↑ Learning
Some forms of (dis)engagement are
productive
Kapur, M. (2016). Examining productive failure… Educational Psychologist, 51(2), 289-299.
Loibl, K., Roll, I., & Rummel, N. (2017). Towards a theory of when and how problem solving followed by
instruction supports learning. Educational Psychology Review, 29(4), 693-715.
Engagement
Learning
Over-
estimate
learning
Under-
estimate
learning

#3. THE TRICKY LANDSCAPE OF
INTERPRETING DATA
("All happy families are alike; each unhappy family is unhappy in
its own way.” - Tolstoy’s Anna Karenina)

Context
• Student factors
– Knowledge, goals, motivations,…
• Task factors
– Interface, relevance, sequence,…
• Environmental factors
– Day and time, peers, space, …
Learners System

Interpretation depends on…
student knowledge
• Should students ask for hints as
they struggle to solve a problem?
Roll, I., Baker, R. S. J. d., Aleven, V., & Koedinger, K. R. (2014). On the benefits of seeking (and avoiding) help in
online problem-solving environments. Journal of the Learning Sciences, 23(4), 537–560.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
High-skill Medium-skill Low-skill
Hints Failed attempts

data representation
• Representation affects
meaning
– implicitly or explicitly.
• How to describe ”clicking
through hints”?
Aleven, V., Mclaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a cognitive
tutor. International Journal of Artificial Intelligence in Education, 16(2), 101–128.
Clicking Through Hints. However, when counting each click
46% of all hint requests were clicking through clusters. Of th
Avoidance (26%) is now far more frequent than either Help
(15%). Further, as shown in Table 6, Help Avoidance is now
with post-test (when controlling for pre-test), whereas before
Combined with the data presented above showing th
Avoidance and performance with the tutor, this analysis len
is captured in the model.
Table 6
Correlations between the bug categories and post-test, with
clusters of Clicking Through Hints either as single actions
(“not collapsed”). The “Not collapsed” row repeats Table 3
that are significant at the 0.05 level are marke
Help
Abuse
Help
Avoidance
Try-S
Abu
Clicking Through Hints
clusters collapsed -0.46* –0.38* 0.11
Not collapsed (-0.48*) (-0.12) (0.26

sequencing
• Who determines whether
“instruction first” or
”problem first”?
• Who interprets?
Loibl, Roll & Rummel, 2017; VanLehn, Siler, Murray, Yamauchi & Baggett, 2003; Kornell, Hays & Bjork, 2009
p< 0.001. This difference was coincidental and went unnoticed
until analysis of study data. Possibly due to differences in sample
composition, significant differences (by condition) were observed
in regular course exam marks (unrelated to the study). On
those marks, students in the PF condition outperformed their
counterparts on Midterm 1 (5.14 ***p< 0.001, 95% CI = [3.23,
7.06], t(559.31) = 5.28; d= 0.44) and final exam (7.63 ***p< 0.001,
95% CI = [4.75, 10.52], t(567.02) = 5.20; d= 0.44), but not on
Midterm 2, (−1.51, p= 0.23, 95% CI = [-4.01, 0.98], t(561.03) =
−1.19; d= −0.10), as shown in Table 1. In order to account for
these differences, performance on these “other’’ items served as
covariates in our model.
Effects of condition on learning
Student performance on the midterm and final exams was
analyzed, blind to condition. The differences described above
were not unexpected for a quasi-experimental student sample. To
isolate the effects of learning condition on student performance,
we used linear regression to predict scores (0–100) on topics where
teaching approach varied, controlling for scores on Midterm 1,
gender, university year, enrollment in the science program, and
exam scores for questions in the same exam on topics where
teaching was not varied (referred to as “other items”). Thus,
differences between student populations or instructors, as
reflected by performance on other items, were accounted for,
isolating the impact of PF vs. AL on the studied topics. For Midterm
2, (following the implementation of PF) the resulting coefficient for
the PF condition was significant, b= 4.78, 95% CI [2.19, 7.36], t
(567) = 3.63, p< 0.001, with an effect size of 0.32, 95% CI [0.15,
0.49] (derived by dividing the coefficient by the residual standard
error). This shows that students in the PF condition had grades for
those topics almost five percentage-points higher (Fig. 3). The
overall model fit was R2
= 0.43, F(6,567) = 71.95, p< 0.001, which
means that 43% of those condition-topic scores could be explained
by this model. To account for the large number of covariates, we
ran regression models with stepwise variable selection. Results
were nearly identical to those reported above, and thus we focus
on the more straightforward, comprehensible, and complete
points higher than their counterparts in the AL group, even after
controlling for gender, faculty, year, and performance on other
items. This effect is considerable for that group.
The effect of PF on Med and High achieving students does not
reach significance, likely because splitting the population reduces
statistical power. As the coefficients for Med and High were
similar, we examined the combined data for these two groups.
The coefficient for the combined group reaches significance, b=
3.25, 95% CI [0.07, 6.43], t(373) = 2.01, p= 0.045. This suggests
that the PF technique benefitted the lower-performing students
Table 1. Raw exam scores, by condition, showing mean (standard deviation)
Study items Other items
Condition N Midterm 2 Final Midterm 1 Midterm 2 Final
AL (active learning) 279 67.96 (19.81) 66.09 (28.11) 76.23 (12.18) 73.50 (15.82) 64.15 (17.89)
PF (productive failure) 295 73.97 (19.35) 74.31 (25.13) 81.38 (11.08) 71.98 (14.54) 71.78 (17.22)
Fig. 3 Adjusted mean study item grades (0–100) and Standard Error
by Condition, controlling for gender, university year, program,
Midterm 1 grades and Midterm 2 non-study items. Error bars show
standard error. ***p< .001
S.G. Chowrira et al.
4
Impact on Midterm
4.8***
(Chowrira, 2019)
Instruction first Problems first

IMPLICATIONS FOR LEARNING
ANALYTICS
https://jmich23.wordpress.com/2014/09/11/ice-cream-graph/

The ABC of Engagement
• Engagement is holistic
– We are more than our actions.
– Use multiple methods, including self-reports, to
make sense of the user experience
– Interpret multiple behaviours or behavioural
patterns in context
• The ABCs are highly intertwined
– Cycles within trajectories, shifts over time

Engagement vs. learning
• Involve users in meaning making
– Qualitative methodologies
• Understand “productive engagement”
– Beyond counting clicks
– Many roads to the same goal (e.g., download vs.
watch video)
• Rely on and inform theory

Interpretation: it depends…
• Capture and include in your models:
– Variability in learners
– Resources
– Sequences
– Context
• Triangulate:
– Across systems à Look for semantic meaning
– Across learners à Cohorts, groups

We study icebergs from a satellite
• Engagement often does not leave
data traces
• Engagement is complex
• ABCs
• Productive (dis)engagement
• Highly contextual
• And this is okay…
• We triangulate
• We close the loop
• We make sense

AcknowledgementsFaculty of Education
Office of Research in Education
(http://educ.ubc.ca)
Hampton Research Grant
Sponsor: Office of VP-Research & International
Sponsor program link: http://www.ors.ubc.ca/node/1265 (http://www.ors.ubc.ca/node/1265)
Value: $10,000 – $25,000 for individual projects for maximum of 2 years
Objectives: designed to fund highly meritorious research or scholarly activity in the Social
Sciences and Humanities. Grants for the Creative and Performing Arts will also be awarded to
support innovative, integrative and/or critically grounded production, reinterpretation or
performances of creative works. It is the intention of the Hampton Committee to approach the
awarding of these grants in a flexible manner, to recognize innovation, experimental and unique
approaches.
Application Deadlines:
9:00 AM Monday February 2: Optional OGPR internal review service. Applicants please
email Robert Olaj (mailto:robert.olaj@ubc.ca) your full proposal. Proposals submitted after
this date may be considered for review, if time permits.9:00 AM Monday February 16: OGPR
internal signature deadline. Applicants please email Robert Olaj (mailto:robert.olaj@ubc.ca)
your full proposal and your Department-signed RPIF
(http://www.research.ubc.ca/vpri/research-project-information-form) .
3:00 PM Monday February 16: The time at which Robert will email the Hampton Coordinator,
ORE
Images: http://clipart-library.com/picture-of-a-ice-cream-cone.html

The Many Flavours of Productive Engagement

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The Many Flavours of Productive Engagement

Similar to The Many Flavours of Productive Engagement (20)

Recently uploaded

Recently uploaded (20)

The Many Flavours of Productive Engagement