Report

Share

Follow

•1 like•478 views

•1 like•478 views

Report

Share

Download to read offline

Paper at CHI 2019, PDF at tiny.cc/icepdf. Digital educational resources could enable the use of randomized experiments to answer pedagogical questions that instructors care about, taking academic research out of the laboratory and into the classroom. We take an instructorcentered approach to designing tools for experimentation that lower the barriers for instructors to conduct experiments. We explore this approach through DynamicProblem, a proof-ofconcept system for experimentation on components of digital problems, which provides interfaces for authoring of experiments on explanations, hints, feedback messages, and learning tips. To rapidly turn data from experiments into practical improvements, the system uses an interpretable machine learning algorithm to analyze students’ ratings of which conditions are helpful, and present conditions to future students in proportion to the evidence they are higher rated. We evaluated the system by collaboratively deploying experiments in the courses of three mathematics instructors. They reported benefits in reflecting on their pedagogy, and having a new method for improving online problems for future students.

Follow

- 1. Enhancing Online Problems Through Instructor- Centered Tools for Randomized Experiments Joseph Jay Williams University of Toronto Computer Science ( Nat. U. of Singapore) www.josephjaywilliams.com/papers, tiny.cc/icepdf Anna Rafferty, Andrew Ang, Dustin Tingley, Walter Lasecki, Juho Kim [I’m originally from the Caribbean, Trinidad and Tobago] Postdoc at U of T (www.josephjaywilliams.com/postdoc) Computer Science PhD positions to do Education Research CHI 2019 subcommittee on “Learning/Families” (Amy Ogan & I are SCs)
- 2. How Can We Help Instructors Conduct A/B Experiments? • Opportunity: Collect data about alternative instructional approaches, instead of relying on intuition • Obstacle: Time and effort to program experiments • Elaboration Messages in online problems x = matrix(rnorm(m*n),m,n) What is the standard error? Answer: A z-score is defined as the number of standard deviations a specific point is away from the mean. Elaboration Messages: Explanations, Hints, Learning Tips
- 3. Related Work • Technology for A/B Experimentation (Optimizely, edX, ASSISTments) (Heffernan & Heffernan, 2014) • Involving instructors in research (Barab & Squire, 2014) • Elaboration messages in online problems (Shute, 2008)
- 4. Overview • Design goals for instructor-centered experimentation • DynamicProblem, an end-user tool (on-campus courses & MOOCs) • Reinforcement learning for dynamic experimentation • Insights from deployment with 3 instructors
- 5. Goals for Instructor Centered Experimentation • 1. Deploy experiments and obtain data with minimal programming – Provide end-user plug-in, DynamicProblem • 2. Use data for practical improvement – Use reinforcement learning to automatically give more effective conditions to future students
- 6. DynamicProblem Plug-In for Courses • Embed into any Learning Management System (e.g. Canvas) or MOOC, via Learning Tools Interoperability Standard
- 7. Student View of DynamicProblem Linda is training for a marathon, which is a race that is 26 miles long. Her average training time for the 26 miles is 208 minutes, but the day of the marathon she was x minutes faster than her average time. What was Linda's running speed for the marathon in miles per minute? Elaboration Message Linda's speed is the distance she ran divided by the time it took. The distance Linda ran was 26 miles. The time it took her was 208 – x. Linda's speed was 26/(208 - x) 26/(208 - x) How helpful was the above information for your learning? Completely Perfectly Unhelpful Helpful 0 1 2 3 4 5 6 7 8 9 10 A B ACM Learning @ Scale 2016
- 8. Instructor View of DynamicProblem
- 12. Data Dashboard (Instructor 3) Learning Tip Probability of Message
- 13. Observations from Deployment with 3 Instructors • Lowered Barriers: “not aware of any tools that do this sort of thing”, “even if I found one, wouldn’t have the technical expertise to incorporate it in my course” • Reflection on pedagogy: “I never really seriously considered [testing] multiple versions as we are now doing. So even if we don't get any significant data, that will have been a benefit in my mind” • Making research practical: “a valuable tool. Putting in the hands of the teacher to understand how their students learn. Not just in broad terms, but specifically in their course”….
- 14. 2. Use Data For Practical Improvement • Instructor concerns: – Experiments advance researchers’ goals, but do not directly help their students – Ethics of giving students unhelpful conditions • Approach of Dynamic Experimentation: – Analyze data in real-time – Give higher-rated messages to future students
- 15. Model Action a Dynamic Experimentation: Exploration vs Exploitation • Multi-Armed Bandit (Reinforcement Learning) A Reward R Policy Elaboration Message A The probability is 3/7 * 5/8, because the number of cookies is changing. Rating How helpful was the above information for your learning? 0 1 2 3 4 5 6 7 8 9 10 A B 70% 30% (Probability of Message being Helpful) (0 to 10 Rating by Student) Elaboration Message B The number of cookies is changing.. Randomized Probability Matching (Thompson Sampling)
- 16. Dynamically Weighted Randomization • 50/50 probability of assignment • 60/40 • 70/30 • … 100/0 • Probability a student assigned to a message = • Probability of that message being highest rated
- 18. Observations from Deployment with 3 Instructors • Directly helping students: “improved the experience of many of the students by giving them answers that are more helpful… the earlier ones can help improve the experience of the later students. That’s pretty neat”
- 19. Student Perceptions? • Students weren’t surprised by, and appreciated the approach: • “I assume companies are always A/B testing on me” • “now the data I provide can help other people learn”
- 20. Limitations & Future Work • Conduct better experiments – You can use the hosted tool: www.josephjaywilliams.com/dynamicproblem github.com/kunanit/adaptive-mooclet-lti • Go beyond subjective student judgments • Instructor-researcher collaboration • Personalization of Elaboration Messages –MOOClet WebService for dynamic A/B testing (www.josephjaywilliams.com/mooclet)
- 21. Review • Design goals for instructor experimentation • DynamicProblem, an end-user tool (Use it at www.josephjaywilliams.com/dynamicproblem) • Multi-armed bandits for dynamic experimentation • Insights from deployment with 3 instructors • Limitations & Future Work • PhD students can do education at U of T Comp Sci • www.josephjaywilliams.com/postdoc • Learning/Families subcommittee for CHI 2019 (Amy Ogan & I are SCs)
- 22. Thank you! • National University of Singapore Information Systems & Analytics • Harvard VPAL Research Group