AERA talk on Nudge, a Computer support for studying at the right times. Goes through design process and system evaluation.
Some slides with animation will be confusing or missing pieces because of Slideshare's importer.
1. +
1
Nudge
Computer support for
studying at the right times
Aleahmad, Koedinger and
Zimmerman
2. + 2
Usability
Research often difficult to translate into practice
Need usable knowledge (Lagemann, 2002)
Tension between internal and ecological validity
Modern computing lets us do controlled lab-style
research in real-world settings
Lower the costs of deploying research in broad contexts
to understand contextual factors
Usable systems
AERA 2012 — Aleahmad, Koedinger and Zimmerman
3. + 5
Process
Evaluation: Evaluation:
Informed
Enactment Local Broader
Exploration
Impact Impact
Integrative Learning Design Framework (Bannan-Ritland 2003)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
4. + 6
Informed Exploration
Used HCI methods for usability:
Focus setting College lecture courses
Contextual inquiry Interviews with stakeholders
Ideation ~60 ideas for systems
Scenario development 17 that express needs
Needs validation Interview stakeholders to check
Example observation for Nudge need:
“by second semester freshman year I was trying to learn how to
study, pretty much teaching myself.”
AERA 2012 — Aleahmad, Koedinger and Zimmerman
6. + 8
Needs Validation
IES practice guides to see what practice
recommendations could be operationalized usably in
college lecture courses
E.g. “help students allocate study time efficiently” from
Organizing Instruction and Study to Improve Student Learning
(Pashler et al., 2007) principle #6
Ruled out all but a few systems because either
1. not interesting theoretically
2. required too much change by stakeholders
3. rejected by the culture
e.g. study leaderboard. students didn't want competition.
AERA 2012 — Aleahmad, Koedinger and Zimmerman
7. + 9
Principles of Nudge Design
Feature Claim Warrant
Course task assigned Explicit and salient External deadlines boost task performance
to dates and dates more likely to more than self-determined deadlines (Ariely &
organized centrally be met Wertenbroch, 2002);
Students generally do whatever’s due soonest
(Kornell & Bjork, 2007)
Break-down of study Decomposition of Smaller tasks abate the planning fallacy (Forsyth
activity into smaller tasks improves time & Burt, 2008; Kruger & Evans, 2003);
actions allocation and Students procrastinate largely due to fear of
decreases failure (Solomon & Rothblum, 1984);
aversiveness In shared task lists, vague information preferred
(Blandford, 2001)
Maintaining and Recording task Self-monitoring of study behaviors improves
tracking status increases learning (Richards, 1975)
assignments, study awareness and
time and progress inclination
AERA 2012 — Aleahmad, Koedinger and Zimmerman
8. + 10
Iterative Development: Mockup
C✓ | Home
http:/ / coursecheck.org/ Google
Joe Student Dashboard Settings Log out
CourseCheck Thu Jan 13
Introduction to Modern Chemistry Calculus in 3D English 101 Pottery
Prof. Paul Karol 01-234
Full course
Past due
Mastery Exam 1 Ch2: exercises 21, 23, 30, 31, 37, 38 Jan 11
Mastery Exam 1 Ch2: exercises 55, 63 Jan 13
Coming week
Minutes spent Save
Notes to self
Notes to instructor
Ch3: exercises 23, 25, 27, 31, 37, 41 Jan 13
for Quiz 1
Ch3: exercises 47, 55 Jan 18
Ch4: exercises 17, 19 Jan 18
AERA 2012 — Aleahmad, Koedinger and Zimmerman
10. + 12
Final Iteration for Evaluation
AERA 2012 — Aleahmad, Koedinger and Zimmerman
11. + 13
Study
Large lecture-based intro chemistry class
15 week fall semester
60 tasks defined for the semester
(14 required, 43 advised, 3 suppl.)
Do and submit HW-n (required)
Take a practice text for Exam-n (advised)
Review notes for Exam IV (advised)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
12. + 14
Conditions
Introductory Chemistry Course
N=136
All nudges polled on tasks before due (n=45)
Nudge?
No nudges, tasks batched after exam to poll (n=48)
Everyone else gets message before due, at a
schedule they choose. (n=42)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
13. + 15
Hypotheses
H-grades: Student sent all Nudge messages perform better
on exams than students sent no Nudge messages.
H-disposition: Students with poor study time use benefit
more from Nudge messages.
Time/Environment scale from Motivated Strategies for
Learning Questionnaire (Pintrich et al. 2001)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
14. + 16
Evaluation of Operation
H-grade: No main effects…
H-disposition: Better time
management led to better exam
scores (F(1,76.9)=6.4, p=.014) but
Nudge interacts to help students
with poor management
(F(1,76.9)=4.6, p=.036).
Nudge helps students with poor management
…and hinders with good management?
AERA 2012 — Aleahmad, Koedinger and Zimmerman
15. + 17
Mechanism
The worse a student’s time use the greater
the benefit of opening each Nudge
message.
AERA 2012 — Aleahmad, Koedinger and Zimmerman
16. + 18
Broader Impact
75% of nudged respondents (n=28) rated “Email reminders
about course work” as “Good” or “Great”
40% agreed, “The reminder emails helped me in the class”
46% agreed, “I wish I could have email reminders for all my
classes”
44% agreed, “Without the reminders I would have forgotten
to do something.”
80% of students who could stop the message didn’t
Group Replied to task polls Agreement with “What I enter is accurate”.
ever (7pt Likert)
No nudges 83% (40/48) 5.5 (n=26, sd=1.4)
All nudges 87% (39/45) 5.8 (n=29, sd=1.3)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
17. + 19
Discussion
Need validation ruled out systems I thought would work;
saved resources
Indications that it is easy to adopt and would be used
voluntarily by students and teachers
Need to test adoption empirically
Usability focus needs more methods for validation
Examplify
Future work
AERA 2012 — Aleahmad, Koedinger and Zimmerman
18. + 20
Thanks for your attention!
@tfunq
Turadg Aleahmad
http://coursecheck.org
http://turadg.aleahmad.net
http://openeducationresearch.org
AERA 2012 — Aleahmad, Koedinger and Zimmerman
Editor's Notes
Hi I’m Turadg Aleahmad and I’d like to share with you the design and evaluation of Nudge, a computer support for studying at the right times.I’ve approached this work from an HCI perspective with my PhD advisors Ken Koedinger and John Zimmerman.
We all know that research is often difficult to translate into practice. We need more usable knowledge!Difficult to span the tension between internal and ecological validityModern computing lets helpsMy big goal is to use computing and design processes to lower the costs of deploying research in broad contexts, to understand contextual factorsThis requires systems that people can and WANT to use
Nudge was developed as part of a design-based inquiry in college lecture courses.The goal was to develop systems that fit into the design context and
I’ve applied some methods to build Nudge,, a system I built to help students allocate their study time more effectivelyI don’t have time to go into all the methods but I will skim over results from three phases of the design:Informed ExplorationEnactmentLocal Impact evaluationAt the end I will speculate on the fourth, Broader Impact
In the Informed Exploration phase I used HCI best practices to try honing in on a design that would fit First I chose a context that may adopt technologies easily: college lecture courses20 million college students every year, mostly in lecturesSorry I have to zoom through but one example result from interviews:by second semester…
So there appeared to be a need among students to have explicit support in how to study.Here is the scenario I used to validate whether that was true.In this hypothetical system, students are asked to report their study activities.Students record on their phones when they are doing different course activities like listening to lecture, or reading over notes.At the end of the course, students can see what different study activities predicted what grades and see evidence for trying new activities to improve their grades.Students really liked having explicit recommendations for how they could study better, but didn’t like having to enter everything in at the moment.
I developed 17 such explicit scenariosIn designing the scenarios,I consulted the IES practice guides to see what I could operationalize that fit these needs.One relevant to Nudge is #6, “help students allocate study time efficiently”Also ruled out many designs because… [read three]e.g. leaderboard
With the results of the interviews and literature review, I had a basic set of design principles for nudge.Basicallya software enhanced interactive syllabus[Walk through features without claims]
More usability testing[should be under 6 min at this point]
pilot, tested in a lecture course through voluntary useExam grades went up for students who used sooner than laterworth studyinglearned that students didn't like logging into the web sitelots of features not used much like a dashboard to track their progressoptimized the email interface
final iteration for evaluation[walk through the UI]progress shown by crossing out tasks skipped or completed
HypothesesH-grades: Student sent all Nudge messages perform better on assessments than students sent no Nudge messages.H-disposition: Students with poor study time use benefit more from Nudge messages.
What explains these gains?Lots of factors here. The task data was noisy and biased by condition.However we know the number of emails they opened and they were a positive predictor of exam scores for students with poor Time skills and were irrelevant for students with good time skills.[discuss more here before going to future work]
So what about the Broader Impact? Don’t have direct evidence but some indications point to the feasibility for broader impact.1. students liked it [data]2. cheap to author for by converting syllabus (quick and no domain expertise)3. didn't require any instructor changeseager to work with others on further deployments