Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments


Published on

Paper at CHI 2019, PDF at

Digital educational resources could enable the use of randomized
experiments to answer pedagogical questions that
instructors care about, taking academic research out of the
laboratory and into the classroom. We take an instructorcentered
approach to designing tools for experimentation that
lower the barriers for instructors to conduct experiments. We
explore this approach through DynamicProblem, a proof-ofconcept
system for experimentation on components of digital
problems, which provides interfaces for authoring of experiments
on explanations, hints, feedback messages, and learning
tips. To rapidly turn data from experiments into practical improvements,
the system uses an interpretable machine learning
algorithm to analyze students’ ratings of which conditions are
helpful, and present conditions to future students in proportion
to the evidence they are higher rated. We evaluated the system
by collaboratively deploying experiments in the courses
of three mathematics instructors. They reported benefits in
reflecting on their pedagogy, and having a new method for
improving online problems for future students.

Published in: Education
  • Login to see the comments

  • Be the first to like this

CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments

  1. 1. Enhancing Online Problems Through Instructor- Centered Tools for Randomized Experiments Joseph Jay Williams University of Toronto Computer Science ( Nat. U. of Singapore), Anna Rafferty, Andrew Ang, Dustin Tingley, Walter Lasecki, Juho Kim [I’m originally from the Caribbean, Trinidad and Tobago] Postdoc at U of T ( Computer Science PhD positions to do Education Research CHI 2019 subcommittee on “Learning/Families” (Amy Ogan & I are SCs)
  2. 2. How Can We Help Instructors Conduct A/B Experiments? • Opportunity: Collect data about alternative instructional approaches, instead of relying on intuition • Obstacle: Time and effort to program experiments • Elaboration Messages in online problems x = matrix(rnorm(m*n),m,n) What is the standard error? Answer: A z-score is defined as the number of standard deviations a specific point is away from the mean. Elaboration Messages: Explanations, Hints, Learning Tips
  3. 3. Related Work • Technology for A/B Experimentation (Optimizely, edX, ASSISTments) (Heffernan & Heffernan, 2014) • Involving instructors in research (Barab & Squire, 2014) • Elaboration messages in online problems (Shute, 2008)
  4. 4. Overview • Design goals for instructor-centered experimentation • DynamicProblem, an end-user tool (on-campus courses & MOOCs) • Reinforcement learning for dynamic experimentation • Insights from deployment with 3 instructors
  5. 5. Goals for Instructor Centered Experimentation • 1. Deploy experiments and obtain data with minimal programming – Provide end-user plug-in, DynamicProblem • 2. Use data for practical improvement – Use reinforcement learning to automatically give more effective conditions to future students
  6. 6. DynamicProblem Plug-In for Courses • Embed into any Learning Management System (e.g. Canvas) or MOOC, via Learning Tools Interoperability Standard
  7. 7. Student View of DynamicProblem Linda is training for a marathon, which is a race that is 26 miles long. Her average training time for the 26 miles is 208 minutes, but the day of the marathon she was x minutes faster than her average time. What was Linda's running speed for the marathon in miles per minute? Elaboration Message Linda's speed is the distance she ran divided by the time it took. The distance Linda ran was 26 miles. The time it took her was 208 – x. Linda's speed was 26/(208 - x) 26/(208 - x) How helpful was the above information for your learning? Completely Perfectly Unhelpful Helpful 0 1 2 3 4 5 6 7 8 9 10 A B ACM Learning @ Scale 2016
  8. 8. Instructor View of DynamicProblem
  9. 9. View Elaboration Messages
  10. 10. Add Elaboration Messages
  11. 11. View Elaboration Messages Learning Tip
  12. 12. Data Dashboard (Instructor 3) Learning Tip Probability of Message
  13. 13. Observations from Deployment with 3 Instructors • Lowered Barriers: “not aware of any tools that do this sort of thing”, “even if I found one, wouldn’t have the technical expertise to incorporate it in my course” • Reflection on pedagogy: “I never really seriously considered [testing] multiple versions as we are now doing. So even if we don't get any significant data, that will have been a benefit in my mind” • Making research practical: “a valuable tool. Putting in the hands of the teacher to understand how their students learn. Not just in broad terms, but specifically in their course”….
  14. 14. 2. Use Data For Practical Improvement • Instructor concerns: – Experiments advance researchers’ goals, but do not directly help their students – Ethics of giving students unhelpful conditions • Approach of Dynamic Experimentation: – Analyze data in real-time – Give higher-rated messages to future students
  15. 15. Model Action a Dynamic Experimentation: Exploration vs Exploitation • Multi-Armed Bandit (Reinforcement Learning) A Reward R Policy Elaboration Message A The probability is 3/7 * 5/8, because the number of cookies is changing. Rating How helpful was the above information for your learning? 0 1 2 3 4 5 6 7 8 9 10 A B 70% 30% (Probability of Message being Helpful) (0 to 10 Rating by Student) Elaboration Message B The number of cookies is changing.. Randomized Probability Matching (Thompson Sampling)
  16. 16. Dynamically Weighted Randomization • 50/50 probability of assignment • 60/40 • 70/30 • … 100/0 • Probability a student assigned to a message = • Probability of that message being highest rated
  17. 17. Instructor 2’s Experiment Probability of Message
  18. 18. Observations from Deployment with 3 Instructors • Directly helping students: “improved the experience of many of the students by giving them answers that are more helpful… the earlier ones can help improve the experience of the later students. That’s pretty neat”
  19. 19. Student Perceptions? • Students weren’t surprised by, and appreciated the approach: • “I assume companies are always A/B testing on me” • “now the data I provide can help other people learn”
  20. 20. Limitations & Future Work • Conduct better experiments – You can use the hosted tool: • Go beyond subjective student judgments • Instructor-researcher collaboration • Personalization of Elaboration Messages –MOOClet WebService for dynamic A/B testing (
  21. 21. Review • Design goals for instructor experimentation • DynamicProblem, an end-user tool (Use it at • Multi-armed bandits for dynamic experimentation • Insights from deployment with 3 instructors • Limitations & Future Work • PhD students can do education at U of T Comp Sci • • Learning/Families subcommittee for CHI 2019 (Amy Ogan & I are SCs)
  22. 22. Thank you! • National University of Singapore Information Systems & Analytics • Harvard VPAL Research Group