Download to read offline
Uploaded byTuradg Aleahmad
AERA 2012 presentation on Nudge
The document discusses the development and evaluation of nudge systems to enhance study behaviors in college courses, emphasizing the gap between usability research and practical application. Through informed exploration and validation of needs, the authors identified successful nudging strategies that improve student performance, particularly benefiting those with poor time management. The findings suggest that email reminders and structured task management are effective, and future research is needed to empirically test the adoption and usability of these systems.
AERA 2012 presentation on Nudge
- 1. + 1 Nudge Computer support for studying at the right times Aleahmad, Koedinger and Zimmerman
- 2. + 2 Usability Research often difficult to translate into practice Need usable knowledge (Lagemann, 2002) Tension between internal and ecological validity Modern computing lets us do controlled lab-style research in real-world settings Lower the costs of deploying research in broad contexts to understand contextual factors Usable systems AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 3. + 5 Process Evaluation: Evaluation: Informed Enactment Local Broader Exploration Impact Impact Integrative Learning Design Framework (Bannan-Ritland 2003) AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 4. + 6 Informed Exploration Used HCI methods for usability: Focus setting College lecture courses Contextual inquiry Interviews with stakeholders Ideation ~60 ideas for systems Scenario development 17 that express needs Needs validation Interview stakeholders to check Example observation for Nudge need: “by second semester freshman year I was trying to learn how to study, pretty much teaching myself.” AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 5. + 7 Nudge Scenario AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 6. + 8 Needs Validation IES practice guides to see what practice recommendations could be operationalized usably in college lecture courses E.g. “help students allocate study time efficiently” from Organizing Instruction and Study to Improve Student Learning (Pashler et al., 2007) principle #6 Ruled out all but a few systems because either 1. not interesting theoretically 2. required too much change by stakeholders 3. rejected by the culture e.g. study leaderboard. students didn't want competition. AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 7. + 9 Principles of Nudge Design Feature Claim Warrant Course task assigned Explicit and salient External deadlines boost task performance to dates and dates more likely to more than self-determined deadlines (Ariely & organized centrally be met Wertenbroch, 2002); Students generally do whatever’s due soonest (Kornell & Bjork, 2007) Break-down of study Decomposition of Smaller tasks abate the planning fallacy (Forsyth activity into smaller tasks improves time & Burt, 2008; Kruger & Evans, 2003); actions allocation and Students procrastinate largely due to fear of decreases failure (Solomon & Rothblum, 1984); aversiveness In shared task lists, vague information preferred (Blandford, 2001) Maintaining and Recording task Self-monitoring of study behaviors improves tracking status increases learning (Richards, 1975) assignments, study awareness and time and progress inclination AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 8. + 10 Iterative Development: Mockup C✓ | Home http:/ / coursecheck.org/ Google Joe Student Dashboard Settings Log out CourseCheck Thu Jan 13 Introduction to Modern Chemistry Calculus in 3D English 101 Pottery Prof. Paul Karol 01-234 Full course Past due Mastery Exam 1 Ch2: exercises 21, 23, 30, 31, 37, 38 Jan 11 Mastery Exam 1 Ch2: exercises 55, 63 Jan 13 Coming week Minutes spent Save Notes to self Notes to instructor Ch3: exercises 23, 25, 27, 31, 37, 41 Jan 13 for Quiz 1 Ch3: exercises 47, 55 Jan 18 Ch4: exercises 17, 19 Jan 18 AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 9.
- 10. + 12 Final Iteration for Evaluation AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 11. + 13 Study Large lecture-based intro chemistry class 15 week fall semester 60 tasks defined for the semester (14 required, 43 advised, 3 suppl.) Do and submit HW-n (required) Take a practice text for Exam-n (advised) Review notes for Exam IV (advised) AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 12. + 14 Conditions Introductory Chemistry Course N=136 All nudges polled on tasks before due (n=45) Nudge? No nudges, tasks batched after exam to poll (n=48) Everyone else gets message before due, at a schedule they choose. (n=42) AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 13. + 15 Hypotheses H-grades: Student sent all Nudge messages perform better on exams than students sent no Nudge messages. H-disposition: Students with poor study time use benefit more from Nudge messages. Time/Environment scale from Motivated Strategies for Learning Questionnaire (Pintrich et al. 2001) AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 14. + 16 Evaluation of Operation H-grade: No main effects… H-disposition: Better time management led to better exam scores (F(1,76.9)=6.4, p=.014) but Nudge interacts to help students with poor management (F(1,76.9)=4.6, p=.036). Nudge helps students with poor management …and hinders with good management? AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 15. + 17 Mechanism The worse a student’s time use the greater the benefit of opening each Nudge message. AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 16. + 18 Broader Impact 75% of nudged respondents (n=28) rated “Email reminders about course work” as “Good” or “Great” 40% agreed, “The reminder emails helped me in the class” 46% agreed, “I wish I could have email reminders for all my classes” 44% agreed, “Without the reminders I would have forgotten to do something.” 80% of students who could stop the message didn’t Group Replied to task polls Agreement with “What I enter is accurate”. ever (7pt Likert) No nudges 83% (40/48) 5.5 (n=26, sd=1.4) All nudges 87% (39/45) 5.8 (n=29, sd=1.3) AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 17. + 19 Discussion Need validation ruled out systems I thought would work; saved resources Indications that it is easy to adopt and would be used voluntarily by students and teachers Need to test adoption empirically Usability focus needs more methods for validation Examplify Future work AERA 2012 — Aleahmad, Koedinger and Zimmerman
- 18. + 20 Thanks for your attention! @tfunq Turadg Aleahmad http://coursecheck.org http://turadg.aleahmad.net http://openeducationresearch.org AERA 2012 — Aleahmad, Koedinger and Zimmerman
Editor's Notes
- #2 Hi I’m Turadg Aleahmad and I’d like to share with you the design and evaluation of Nudge, a computer support for studying at the right times.I’ve approached this work from an HCI perspective with my PhD advisors Ken Koedinger and John Zimmerman.
- #3 We all know that research is often difficult to translate into practice. We need more usable knowledge!Difficult to span the tension between internal and ecological validityModern computing lets helpsMy big goal is to use computing and design processes to lower the costs of deploying research in broad contexts, to understand contextual factorsThis requires systems that people can and WANT to use
- #4 Nudge was developed as part of a design-based inquiry in college lecture courses.The goal was to develop systems that fit into the design context and
- #6 I’ve applied some methods to build Nudge,, a system I built to help students allocate their study time more effectivelyI don’t have time to go into all the methods but I will skim over results from three phases of the design:Informed ExplorationEnactmentLocal Impact evaluationAt the end I will speculate on the fourth, Broader Impact
- #7 In the Informed Exploration phase I used HCI best practices to try honing in on a design that would fit First I chose a context that may adopt technologies easily: college lecture courses20 million college students every year, mostly in lecturesSorry I have to zoom through but one example result from interviews:by second semester…
- #8 So there appeared to be a need among students to have explicit support in how to study.Here is the scenario I used to validate whether that was true.In this hypothetical system, students are asked to report their study activities.Students record on their phones when they are doing different course activities like listening to lecture, or reading over notes.At the end of the course, students can see what different study activities predicted what grades and see evidence for trying new activities to improve their grades.Students really liked having explicit recommendations for how they could study better, but didn’t like having to enter everything in at the moment.
- #9 I developed 17 such explicit scenariosIn designing the scenarios,I consulted the IES practice guides to see what I could operationalize that fit these needs.One relevant to Nudge is #6, “help students allocate study time efficiently”Also ruled out many designs because… [read three]e.g. leaderboard
- #10 With the results of the interviews and literature review, I had a basic set of design principles for nudge.Basicallya software enhanced interactive syllabus[Walk through features without claims]
- #11 More usability testing[should be under 6 min at this point]
- #12 pilot, tested in a lecture course through voluntary useExam grades went up for students who used sooner than laterworth studyinglearned that students didn't like logging into the web sitelots of features not used much like a dashboard to track their progressoptimized the email interface
- #13 final iteration for evaluation[walk through the UI]progress shown by crossing out tasks skipped or completed
- #16 HypothesesH-grades: Student sent all Nudge messages perform better on assessments than students sent no Nudge messages.H-disposition: Students with poor study time use benefit more from Nudge messages.
- #18 What explains these gains?Lots of factors here. The task data was noisy and biased by condition.However we know the number of emails they opened and they were a positive predictor of exam scores for students with poor Time skills and were irrelevant for students with good time skills.[discuss more here before going to future work]
- #19 So what about the Broader Impact? Don’t have direct evidence but some indications point to the feasibility for broader impact.1. students liked it [data]2. cheap to author for by converting syllabus (quick and no domain expertise)3. didn't require any instructor changeseager to work with others on further deployments