CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments

Joseph Jay Williams
Joseph Jay WilliamsVPAL Online Education Research Group
Enhancing Online Problems Through Instructor-
Centered Tools for Randomized Experiments
Joseph Jay Williams
University of Toronto Computer Science ( Nat. U. of Singapore)
www.josephjaywilliams.com/papers, tiny.cc/icepdf
Anna Rafferty, Andrew Ang, Dustin Tingley, Walter Lasecki, Juho Kim
[I’m originally from the Caribbean,
Trinidad and Tobago]
Postdoc at U of T (www.josephjaywilliams.com/postdoc)
Computer Science PhD positions to do Education Research
CHI 2019 subcommittee on “Learning/Families” (Amy Ogan &
I are SCs)
How Can We Help Instructors Conduct A/B Experiments?
• Opportunity: Collect data about alternative instructional
approaches, instead of relying on intuition
• Obstacle: Time and effort to program experiments
• Elaboration Messages in online problems
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
A z-score is defined as the number of
standard deviations a specific point is
away from the mean.
Elaboration Messages:
Explanations,
Hints,
Learning Tips
Related Work
• Technology for A/B Experimentation (Optimizely,
edX, ASSISTments) (Heffernan & Heffernan, 2014)
• Involving instructors in research (Barab & Squire, 2014)
• Elaboration messages in online problems (Shute, 2008)
Overview
• Design goals for instructor-centered
experimentation
• DynamicProblem, an end-user tool (on-campus
courses & MOOCs)
• Reinforcement learning for dynamic
experimentation
• Insights from deployment with 3 instructors
Goals for Instructor Centered Experimentation
• 1. Deploy experiments and obtain data with
minimal programming
– Provide end-user plug-in, DynamicProblem
• 2. Use data for practical improvement
– Use reinforcement learning to automatically give
more effective conditions to future students
DynamicProblem Plug-In for Courses
• Embed into any Learning Management System (e.g. Canvas)
or MOOC, via Learning Tools Interoperability Standard
Student View of DynamicProblem
Linda is training for a marathon, which is a race that is 26
miles long.
Her average training time for the 26 miles is 208 minutes,
but the day of the marathon she was x minutes faster than
her average time.
What was Linda's running speed for the marathon in miles
per minute?
Elaboration Message
Linda's speed is the distance she ran divided by the time it took. The
distance Linda ran was 26 miles. The time it took her was 208 – x.
Linda's speed was 26/(208 - x)
26/(208 - x)
How helpful was the above information for your learning?
Completely Perfectly
Unhelpful Helpful
0 1 2 3 4 5 6 7 8 9 10
A
B
ACM Learning @ Scale 2016
Instructor View of DynamicProblem
View Elaboration Messages
Add Elaboration Messages
View Elaboration Messages
Learning Tip
Data Dashboard (Instructor 3)
Learning Tip
Probability
of
Message
Observations from Deployment with 3 Instructors
• Lowered Barriers: “not aware of any tools that do this
sort of thing”, “even if I found one, wouldn’t have the
technical expertise to incorporate it in my course”
• Reflection on pedagogy: “I never really seriously
considered [testing] multiple versions as we are now
doing. So even if we don't get any significant data, that
will have been a benefit in my mind”
• Making research practical: “a valuable tool. Putting in
the hands of the teacher to understand how their
students learn. Not just in broad terms, but specifically
in their course”….
2. Use Data For Practical Improvement
• Instructor concerns:
– Experiments advance researchers’ goals, but do not
directly help their students
– Ethics of giving students unhelpful conditions
• Approach of Dynamic Experimentation:
– Analyze data in real-time
– Give higher-rated messages to future students
Model
Action a
Dynamic Experimentation: Exploration vs Exploitation
• Multi-Armed Bandit (Reinforcement Learning)
A
Reward R
Policy
Elaboration Message A
The probability is 3/7 * 5/8, because the number of
cookies is changing.
Rating
How helpful was the above information for your learning?
0 1 2 3 4 5 6 7 8 9 10
A B
70% 30%
(Probability of Message being Helpful)
(0 to 10 Rating by Student)
Elaboration Message B
The number of cookies is changing..
Randomized Probability
Matching (Thompson Sampling)
Dynamically Weighted Randomization
• 50/50 probability of assignment
• 60/40
• 70/30
• … 100/0
• Probability a student assigned to a message =
• Probability of that message being highest rated
Instructor 2’s Experiment
Probability
of
Message
Observations from Deployment with 3 Instructors
• Directly helping students: “improved the experience
of many of the students by giving them answers that
are more helpful… the earlier ones can help improve
the experience of the later students. That’s pretty
neat”
Student Perceptions?
• Students weren’t surprised by, and appreciated the
approach:
• “I assume companies are always A/B testing on me”
• “now the data I provide can help other people learn”
Limitations & Future Work
• Conduct better experiments
– You can use the hosted tool: www.josephjaywilliams.com/dynamicproblem
github.com/kunanit/adaptive-mooclet-lti
• Go beyond subjective student judgments
• Instructor-researcher collaboration
• Personalization of Elaboration Messages
–MOOClet WebService for dynamic A/B testing
(www.josephjaywilliams.com/mooclet)
Review
• Design goals for instructor experimentation
• DynamicProblem, an end-user tool (Use it at
www.josephjaywilliams.com/dynamicproblem)
• Multi-armed bandits for dynamic experimentation
• Insights from deployment with 3 instructors
• Limitations & Future Work
• PhD students can do education at U of T Comp Sci
• www.josephjaywilliams.com/postdoc
• Learning/Families subcommittee for CHI 2019 (Amy
Ogan & I are SCs)
Thank you!
• National University of Singapore Information
Systems & Analytics
• Harvard VPAL Research Group
1 of 22

Recommended

Michael Gage SOED 2016 by
Michael Gage SOED 2016Michael Gage SOED 2016
Michael Gage SOED 2016Colleen Ganley
1.4K views37 slides
Infuse Student Engagement and Assessment Using InfuseLearning by
Infuse Student Engagement and Assessment Using InfuseLearningInfuse Student Engagement and Assessment Using InfuseLearning
Infuse Student Engagement and Assessment Using InfuseLearningAndrew Steinman
577 views31 slides
Adaptive Multilevel Clustering Model for the Prediction of Academic Risk by
Adaptive Multilevel Clustering Model for the Prediction of Academic RiskAdaptive Multilevel Clustering Model for the Prediction of Academic Risk
Adaptive Multilevel Clustering Model for the Prediction of Academic RiskXavier Ochoa
851 views30 slides
Clicker technologies as a tool to increase student engagement and motivation by
Clicker technologies as a tool to increase student engagement and motivationClicker technologies as a tool to increase student engagement and motivation
Clicker technologies as a tool to increase student engagement and motivationWSSU CETL
1.4K views34 slides
micro testing teaching learning analytics by
micro testing teaching learning analyticsmicro testing teaching learning analytics
micro testing teaching learning analyticsMartin Schön
1.6K views37 slides
Digging deep by
Digging deepDigging deep
Digging deeprlgiggy
143 views6 slides

More Related Content

What's hot

Teaching electronic resources to students with computer anxiety by
Teaching electronic resources to students with computer anxietyTeaching electronic resources to students with computer anxiety
Teaching electronic resources to students with computer anxietyAurélie Gandour
1.5K views35 slides
Observed to Improve by
Observed to ImproveObserved to Improve
Observed to Improveolumisti
549 views21 slides
Surviving (and thriving) in the Age of Technology-Enhanced Instruction by
Surviving (and thriving) in the Age of Technology-Enhanced InstructionSurviving (and thriving) in the Age of Technology-Enhanced Instruction
Surviving (and thriving) in the Age of Technology-Enhanced InstructionMaria H. Andersen
7.5K views73 slides
Multimodal Learning Analytics by
Multimodal Learning AnalyticsMultimodal Learning Analytics
Multimodal Learning AnalyticsXavier Ochoa
514 views83 slides
Some experiences from evaluating and stress testing digital examination systems by
Some experiences from evaluating and stress testing digital examination systemsSome experiences from evaluating and stress testing digital examination systems
Some experiences from evaluating and stress testing digital examination systemsPer Olof Arnäs
520 views53 slides
Introduction to College Studies by
Introduction to College StudiesIntroduction to College Studies
Introduction to College Studiesbtjt97
326 views10 slides

What's hot(20)

Teaching electronic resources to students with computer anxiety by Aurélie Gandour
Teaching electronic resources to students with computer anxietyTeaching electronic resources to students with computer anxiety
Teaching electronic resources to students with computer anxiety
Aurélie Gandour1.5K views
Observed to Improve by olumisti
Observed to ImproveObserved to Improve
Observed to Improve
olumisti549 views
Surviving (and thriving) in the Age of Technology-Enhanced Instruction by Maria H. Andersen
Surviving (and thriving) in the Age of Technology-Enhanced InstructionSurviving (and thriving) in the Age of Technology-Enhanced Instruction
Surviving (and thriving) in the Age of Technology-Enhanced Instruction
Maria H. Andersen7.5K views
Multimodal Learning Analytics by Xavier Ochoa
Multimodal Learning AnalyticsMultimodal Learning Analytics
Multimodal Learning Analytics
Xavier Ochoa514 views
Some experiences from evaluating and stress testing digital examination systems by Per Olof Arnäs
Some experiences from evaluating and stress testing digital examination systemsSome experiences from evaluating and stress testing digital examination systems
Some experiences from evaluating and stress testing digital examination systems
Per Olof Arnäs520 views
Introduction to College Studies by btjt97
Introduction to College StudiesIntroduction to College Studies
Introduction to College Studies
btjt97326 views
DIT_Clickers_Workshop Sept 2011 by Barry_Ryan
DIT_Clickers_Workshop Sept 2011DIT_Clickers_Workshop Sept 2011
DIT_Clickers_Workshop Sept 2011
Barry_Ryan1K views
Poll everywhere by drew_grace
Poll everywherePoll everywhere
Poll everywhere
drew_grace595 views
Experience Sharing of Introducing Free Software to Campus: Example of Geng-fa... by 光章 林
Experience Sharing of Introducing Free Software to Campus: Example of Geng-fa...Experience Sharing of Introducing Free Software to Campus: Example of Geng-fa...
Experience Sharing of Introducing Free Software to Campus: Example of Geng-fa...
光章 林455 views
Multimodal Learning Analytics by Xavier Ochoa
Multimodal Learning AnalyticsMultimodal Learning Analytics
Multimodal Learning Analytics
Xavier Ochoa1.8K views
Blog virtual manipulatives research ppt by castilloe6086
Blog virtual manipulatives research pptBlog virtual manipulatives research ppt
Blog virtual manipulatives research ppt
castilloe60861.5K views
Netbook trial 21st century learning 2010 by cummingscraig
Netbook trial 21st century learning 2010Netbook trial 21st century learning 2010
Netbook trial 21st century learning 2010
cummingscraig266 views
Analysis on ABCSTTU Model of Homdepot by MJ Ferdous
Analysis on ABCSTTU Model of HomdepotAnalysis on ABCSTTU Model of Homdepot
Analysis on ABCSTTU Model of Homdepot
MJ Ferdous955 views
Elac team project2 by Unjoo Lee
Elac team project2Elac team project2
Elac team project2
Unjoo Lee284 views
AMATYC Ignite 2017 2nd Half by Fred Feldon
AMATYC Ignite 2017 2nd HalfAMATYC Ignite 2017 2nd Half
AMATYC Ignite 2017 2nd Half
Fred Feldon147 views
Teaching coding workshops by Whitni Watkins
Teaching coding workshops Teaching coding workshops
Teaching coding workshops
Whitni Watkins317 views
Data collection and analysis by educw200
Data collection and analysisData collection and analysis
Data collection and analysis
educw200519 views

Similar to CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments

Learner Analytics and the “Big Data” Promise for Course & Program Assessment by
Learner Analytics and the “Big Data” Promise for Course & Program AssessmentLearner Analytics and the “Big Data” Promise for Course & Program Assessment
Learner Analytics and the “Big Data” Promise for Course & Program AssessmentJohn Whitmer, Ed.D.
988 views65 slides
How online educational resources provide novel affordances for conducting pra... by
How online educational resources provide novel affordances for conducting pra...How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...Joseph Jay Williams
1K views29 slides
The why and what of testa by
The why and what of testaThe why and what of testa
The why and what of testaTansy Jessop
55 views75 slides
Defining Adaptive Learning Technology: What it is, how it works, and why it’s... by
Defining Adaptive Learning Technology: What it is, how it works, and why it’s...Defining Adaptive Learning Technology: What it is, how it works, and why it’s...
Defining Adaptive Learning Technology: What it is, how it works, and why it’s...DreamBox Learning
4.4K views96 slides

Similar to CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments(20)

Learner Analytics and the “Big Data” Promise for Course & Program Assessment by John Whitmer, Ed.D.
Learner Analytics and the “Big Data” Promise for Course & Program AssessmentLearner Analytics and the “Big Data” Promise for Course & Program Assessment
Learner Analytics and the “Big Data” Promise for Course & Program Assessment
How online educational resources provide novel affordances for conducting pra... by Joseph Jay Williams
How online educational resources provide novel affordances for conducting pra...How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...
The why and what of testa by Tansy Jessop
The why and what of testaThe why and what of testa
The why and what of testa
Tansy Jessop55 views
Defining Adaptive Learning Technology: What it is, how it works, and why it’s... by DreamBox Learning
Defining Adaptive Learning Technology: What it is, how it works, and why it’s...Defining Adaptive Learning Technology: What it is, how it works, and why it’s...
Defining Adaptive Learning Technology: What it is, how it works, and why it’s...
DreamBox Learning4.4K views
CCC-M F2F meeting_141007 by McGill_RSB
CCC-M F2F meeting_141007CCC-M F2F meeting_141007
CCC-M F2F meeting_141007
McGill_RSB451 views
Lessons Learned from Moodle VLE/LMS Data in the Field by John Whitmer, Ed.D.
Lessons Learned from Moodle VLE/LMS Data in the FieldLessons Learned from Moodle VLE/LMS Data in the Field
Lessons Learned from Moodle VLE/LMS Data in the Field
TESTA to FASTECH Presentation by Tansy_Jessop
TESTA to FASTECH PresentationTESTA to FASTECH Presentation
TESTA to FASTECH Presentation
Tansy_Jessop492 views
WCOL2019: Learning analytics for learning design or learning design for learn... by Marko Teräs
WCOL2019: Learning analytics for learning design or learning design for learn...WCOL2019: Learning analytics for learning design or learning design for learn...
WCOL2019: Learning analytics for learning design or learning design for learn...
Marko Teräs203 views
CCD Jump into JiTT - Feb 2014 by Jeff Loats
CCD Jump into JiTT - Feb 2014CCD Jump into JiTT - Feb 2014
CCD Jump into JiTT - Feb 2014
Jeff Loats389 views
The power of learning analytics to unpack learning and teaching: a critical p... by Bart Rienties
The power of learning analytics to unpack learning and teaching: a critical p...The power of learning analytics to unpack learning and teaching: a critical p...
The power of learning analytics to unpack learning and teaching: a critical p...
Bart Rienties281 views
Changing the assessment narrative by Tansy Jessop
Changing the assessment narrativeChanging the assessment narrative
Changing the assessment narrative
Tansy Jessop324 views
TESTA to FASTECH (November 2011) by TESTA winch
 TESTA to FASTECH (November 2011) TESTA to FASTECH (November 2011)
TESTA to FASTECH (November 2011)
TESTA winch252 views
Learning analytics overview: Building evidence based practice by Shane Dawson
Learning analytics overview: Building evidence based practiceLearning analytics overview: Building evidence based practice
Learning analytics overview: Building evidence based practice
Shane Dawson4.1K views
An evidence based model by Jisc
An evidence based modelAn evidence based model
An evidence based model
Jisc436 views
Defining Adaptive Learning Technology by DreamBox Learning
Defining Adaptive Learning TechnologyDefining Adaptive Learning Technology
Defining Adaptive Learning Technology
DreamBox Learning2.1K views
CHECO Retreat - Changing landscape of teaching by Jeff Loats
CHECO Retreat - Changing landscape of teachingCHECO Retreat - Changing landscape of teaching
CHECO Retreat - Changing landscape of teaching
Jeff Loats309 views
Learning Analytics: Realizing their Promise in the California State University by John Whitmer, Ed.D.
Learning Analytics:  Realizing their Promise in the California State UniversityLearning Analytics:  Realizing their Promise in the California State University
Learning Analytics: Realizing their Promise in the California State University

More from Joseph Jay Williams

Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear... by
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Joseph Jay Williams
697 views43 slides
Experiments in Educational Research and Practice by
Experiments in Educational Research and PracticeExperiments in Educational Research and Practice
Experiments in Educational Research and PracticeJoseph Jay Williams
898 views19 slides
Learning innovation at scale chi 2014 workshop extended abstract by
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractJoseph Jay Williams
931 views4 slides
Experiments in Educational Research & Practice by
Experiments in Educational Research & PracticeExperiments in Educational Research & Practice
Experiments in Educational Research & PracticeJoseph Jay Williams
424 views19 slides
Supporting Instructors in MOOCs: Using cognitive science research to guide pe... by
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Joseph Jay Williams
1K views22 slides
Using Experiments and Cognitive Science Research to Improve the Design of Onl... by
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Joseph Jay Williams
998 views32 slides

More from Joseph Jay Williams(11)

Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear... by Joseph Jay Williams
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Experiments in Educational Research and Practice by Joseph Jay Williams
Experiments in Educational Research and PracticeExperiments in Educational Research and Practice
Experiments in Educational Research and Practice
Learning innovation at scale chi 2014 workshop extended abstract by Joseph Jay Williams
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstract
Supporting Instructors in MOOCs: Using cognitive science research to guide pe... by Joseph Jay Williams
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Using Experiments and Cognitive Science Research to Improve the Design of Onl... by Joseph Jay Williams
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Doing online learning research with both scientific and financial value by Joseph Jay Williams
Doing online learning research with both scientific and financial valueDoing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial value
How can Cognitive Science improve Online Learning & Education? by Joseph Jay Williams
How can Cognitive Science improve Online Learning & Education?How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?
Joseph Jay Williams1.1K views
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting by Joseph Jay Williams
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
Explanation & learning slides (talk @ pittsburgh science of learning center) by Joseph Jay Williams
Explanation & learning slides (talk @ pittsburgh science of learning center)Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)

Recently uploaded

Dance KS5 Breakdown by
Dance KS5 BreakdownDance KS5 Breakdown
Dance KS5 BreakdownWestHatch
86 views2 slides
Recap of our Class by
Recap of our ClassRecap of our Class
Recap of our ClassCorinne Weisgerber
81 views15 slides
MIXING OF PHARMACEUTICALS.pptx by
MIXING OF PHARMACEUTICALS.pptxMIXING OF PHARMACEUTICALS.pptx
MIXING OF PHARMACEUTICALS.pptxAnupkumar Sharma
82 views35 slides
MercerJesse2.1Doc.pdf by
MercerJesse2.1Doc.pdfMercerJesse2.1Doc.pdf
MercerJesse2.1Doc.pdfjessemercerail
237 views5 slides
Psychology KS5 by
Psychology KS5Psychology KS5
Psychology KS5WestHatch
103 views5 slides
unidad 3.pdf by
unidad 3.pdfunidad 3.pdf
unidad 3.pdfMarcosRodriguezUcedo
106 views38 slides

Recently uploaded(20)

Dance KS5 Breakdown by WestHatch
Dance KS5 BreakdownDance KS5 Breakdown
Dance KS5 Breakdown
WestHatch86 views
Psychology KS5 by WestHatch
Psychology KS5Psychology KS5
Psychology KS5
WestHatch103 views
Drama KS5 Breakdown by WestHatch
Drama KS5 BreakdownDrama KS5 Breakdown
Drama KS5 Breakdown
WestHatch87 views
Structure and Functions of Cell.pdf by Nithya Murugan
Structure and Functions of Cell.pdfStructure and Functions of Cell.pdf
Structure and Functions of Cell.pdf
Nithya Murugan701 views
Ch. 7 Political Participation and Elections.pptx by Rommel Regala
Ch. 7 Political Participation and Elections.pptxCh. 7 Political Participation and Elections.pptx
Ch. 7 Political Participation and Elections.pptx
Rommel Regala105 views
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx by Ms. Pooja Bhandare
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptxPharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx
The basics - information, data, technology and systems.pdf by JonathanCovena1
The basics - information, data, technology and systems.pdfThe basics - information, data, technology and systems.pdf
The basics - information, data, technology and systems.pdf
JonathanCovena1126 views
Education and Diversity.pptx by DrHafizKosar
Education and Diversity.pptxEducation and Diversity.pptx
Education and Diversity.pptx
DrHafizKosar177 views
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively by PECB
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks EffectivelyISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively
PECB 598 views
Ch. 8 Political Party and Party System.pptx by Rommel Regala
Ch. 8 Political Party and Party System.pptxCh. 8 Political Party and Party System.pptx
Ch. 8 Political Party and Party System.pptx
Rommel Regala53 views
7 NOVEL DRUG DELIVERY SYSTEM.pptx by Sachin Nitave
7 NOVEL DRUG DELIVERY SYSTEM.pptx7 NOVEL DRUG DELIVERY SYSTEM.pptx
7 NOVEL DRUG DELIVERY SYSTEM.pptx
Sachin Nitave61 views
EIT-Digital_Spohrer_AI_Intro 20231128 v1.pptx by ISSIP
EIT-Digital_Spohrer_AI_Intro 20231128 v1.pptxEIT-Digital_Spohrer_AI_Intro 20231128 v1.pptx
EIT-Digital_Spohrer_AI_Intro 20231128 v1.pptx
ISSIP379 views

CHI (Computer Human Interaction) 2019 enhancing online problems through instructor centered tools for randomized experiments

  • 1. Enhancing Online Problems Through Instructor- Centered Tools for Randomized Experiments Joseph Jay Williams University of Toronto Computer Science ( Nat. U. of Singapore) www.josephjaywilliams.com/papers, tiny.cc/icepdf Anna Rafferty, Andrew Ang, Dustin Tingley, Walter Lasecki, Juho Kim [I’m originally from the Caribbean, Trinidad and Tobago] Postdoc at U of T (www.josephjaywilliams.com/postdoc) Computer Science PhD positions to do Education Research CHI 2019 subcommittee on “Learning/Families” (Amy Ogan & I are SCs)
  • 2. How Can We Help Instructors Conduct A/B Experiments? • Opportunity: Collect data about alternative instructional approaches, instead of relying on intuition • Obstacle: Time and effort to program experiments • Elaboration Messages in online problems x = matrix(rnorm(m*n),m,n) What is the standard error? Answer: A z-score is defined as the number of standard deviations a specific point is away from the mean. Elaboration Messages: Explanations, Hints, Learning Tips
  • 3. Related Work • Technology for A/B Experimentation (Optimizely, edX, ASSISTments) (Heffernan & Heffernan, 2014) • Involving instructors in research (Barab & Squire, 2014) • Elaboration messages in online problems (Shute, 2008)
  • 4. Overview • Design goals for instructor-centered experimentation • DynamicProblem, an end-user tool (on-campus courses & MOOCs) • Reinforcement learning for dynamic experimentation • Insights from deployment with 3 instructors
  • 5. Goals for Instructor Centered Experimentation • 1. Deploy experiments and obtain data with minimal programming – Provide end-user plug-in, DynamicProblem • 2. Use data for practical improvement – Use reinforcement learning to automatically give more effective conditions to future students
  • 6. DynamicProblem Plug-In for Courses • Embed into any Learning Management System (e.g. Canvas) or MOOC, via Learning Tools Interoperability Standard
  • 7. Student View of DynamicProblem Linda is training for a marathon, which is a race that is 26 miles long. Her average training time for the 26 miles is 208 minutes, but the day of the marathon she was x minutes faster than her average time. What was Linda's running speed for the marathon in miles per minute? Elaboration Message Linda's speed is the distance she ran divided by the time it took. The distance Linda ran was 26 miles. The time it took her was 208 – x. Linda's speed was 26/(208 - x) 26/(208 - x) How helpful was the above information for your learning? Completely Perfectly Unhelpful Helpful 0 1 2 3 4 5 6 7 8 9 10 A B ACM Learning @ Scale 2016
  • 8. Instructor View of DynamicProblem
  • 12. Data Dashboard (Instructor 3) Learning Tip Probability of Message
  • 13. Observations from Deployment with 3 Instructors • Lowered Barriers: “not aware of any tools that do this sort of thing”, “even if I found one, wouldn’t have the technical expertise to incorporate it in my course” • Reflection on pedagogy: “I never really seriously considered [testing] multiple versions as we are now doing. So even if we don't get any significant data, that will have been a benefit in my mind” • Making research practical: “a valuable tool. Putting in the hands of the teacher to understand how their students learn. Not just in broad terms, but specifically in their course”….
  • 14. 2. Use Data For Practical Improvement • Instructor concerns: – Experiments advance researchers’ goals, but do not directly help their students – Ethics of giving students unhelpful conditions • Approach of Dynamic Experimentation: – Analyze data in real-time – Give higher-rated messages to future students
  • 15. Model Action a Dynamic Experimentation: Exploration vs Exploitation • Multi-Armed Bandit (Reinforcement Learning) A Reward R Policy Elaboration Message A The probability is 3/7 * 5/8, because the number of cookies is changing. Rating How helpful was the above information for your learning? 0 1 2 3 4 5 6 7 8 9 10 A B 70% 30% (Probability of Message being Helpful) (0 to 10 Rating by Student) Elaboration Message B The number of cookies is changing.. Randomized Probability Matching (Thompson Sampling)
  • 16. Dynamically Weighted Randomization • 50/50 probability of assignment • 60/40 • 70/30 • … 100/0 • Probability a student assigned to a message = • Probability of that message being highest rated
  • 18. Observations from Deployment with 3 Instructors • Directly helping students: “improved the experience of many of the students by giving them answers that are more helpful… the earlier ones can help improve the experience of the later students. That’s pretty neat”
  • 19. Student Perceptions? • Students weren’t surprised by, and appreciated the approach: • “I assume companies are always A/B testing on me” • “now the data I provide can help other people learn”
  • 20. Limitations & Future Work • Conduct better experiments – You can use the hosted tool: www.josephjaywilliams.com/dynamicproblem github.com/kunanit/adaptive-mooclet-lti • Go beyond subjective student judgments • Instructor-researcher collaboration • Personalization of Elaboration Messages –MOOClet WebService for dynamic A/B testing (www.josephjaywilliams.com/mooclet)
  • 21. Review • Design goals for instructor experimentation • DynamicProblem, an end-user tool (Use it at www.josephjaywilliams.com/dynamicproblem) • Multi-armed bandits for dynamic experimentation • Insights from deployment with 3 instructors • Limitations & Future Work • PhD students can do education at U of T Comp Sci • www.josephjaywilliams.com/postdoc • Learning/Families subcommittee for CHI 2019 (Amy Ogan & I are SCs)
  • 22. Thank you! • National University of Singapore Information Systems & Analytics • Harvard VPAL Research Group