How online educational resources provide novel
affordances for conducting practical interventions
and doing psychology experiments
www.josephjaywilliams.com/experiments-online-education
Joseph Jay Williams, Stanford University
Lytics Lab (lytics.stanford.edu), Graduate School of Education
Office of the Vice Provost of Online Learning (vpol.stanford.edu)
josephjaywilliams@stanford.edu
www.josephjaywilliams.com
Originally from Trinidad – apologies for thick accent!
1

Overview
•
•
•
•
•
•
MOOCs VS Digital Online Resources
Experiments: Link Real World & Laboratory
Motivational Messages Experiment
Simultaneous Basic & Applied Research
Ed-Surge Meetup to link Research & Practice
(Wed Feb 19 @ 7)
Current Directions
Teaching Learning Strategies
Learning Coach
MOOC Experiment
•
MOOClets Research & Practice Group
2

Online Education

Moving from Physical to Digital World

Experiments: Link Real World & Laboratory
• Randomized assignment
• Experimental Control
• Rich data
• Real-world
environment
• Authentic activities
• Practical Challenges

Experiment
• Practical goal on Khan Academy
• Extend motivation research
6

Motivate students to practice math
• Online (Math) Exercise
Motivational Message: Keep up the good work!
1. Number of
Problems
Practiced
2. Improve
Accuracy
7

Add motivational messages
Practice-as-usual
Growth Mindset Message
Remember, the more you practice the smarter you become!
8

Which messages to add?
•
•
•
•
•
•
"This might be a tough problem, but we know
you can do it!”
Remember, the more you practice the smarter
you become!
Some of these problems are hard. Just do your
best.”
Encourage “Growth Mindset” beliefs about
intelligence (Dweck, 2008)
Fixed Mindset
Growth Mindset
9

Embedded in vivo Experiment
Jascha Sohl-Dickstein
• Practice-as-usual
• Growth Mindset
Message
•
•
"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”
10

Results: More motivated?
• Growth Mindset Message > Practice-asUsual
• reduction in dropout rate
• increase in Number of problems
practiced
• increase in Number Correct
• Accuracy: Problems correct/Problems
attempted
• Accuracy improvement
11

Interpreting results
• Obvious?
• Skepticism: Anything’s statistically
signficant in a MOOC
12

Different motivational messages
Practice-as-usual Message
Growth Mindset Message
Positive
Remember, the more you practice the smarter you become!
Some of these problems are hard. Do your
best!
13

Does any positive message work?
• Practice-as-usual
• Growth Mindset
Message
•
•
"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”
• Positive Message
•
•
"Some of these problems are
hard. Just do your best."
"This might be a tough problem,
but we know you can do it.”
14

Effects of Positive Messages?
• Positive Messages not better than
Practice-as-Usual!
• Growth Mindset better than Positive
• reduction in dropout rate
• increase in Number of problems
practiced
• increase in Number Correct
• Accuracy improvement
15

Next: Situationally appropriate messages
• Message blast, can be ignored
• Providing motivational message Justin-Time: after incorrect answer
• Laboratory experiments
• www.assistments.org Neil Heffernan
16

Simultaneous Applied & Basic Research
• Start with “Product”
• Constraints
Randomization (Technical, use Qualtrics)
Behavioral/Learning Measures
Williams & Williams (2013). Using Interventions to improve Online Learning. NIPS
Data Driven Education workshop. tiny.cc/experimentsonlinelearning
• Consider Synthesis
Williams, J.J. (2013) Improving Learning in MOOCs by Applying Cognitive
Science. AIED MOOCshop & NIPS Data-Driven Workshop.
tiny.cc/cognitivescienceinmoocs
Reading List: www.josephjaywilliams.com/education
Improving outcome of psychosocial treatments by enhancing memory and
learning.(in press) Perspectives in Psychological Science.
• High Investment but High Return
17

Multidisciplinary – Relevant Communities
•
•
•
•
•
Association for Psychological Science 2013. Online Learning and Psychological
Science: Opportunities to integrate research and practice. Williams, Saxberg,
Means, Mitros.
Cognitive Science Society 2013. Online Education: A Unique Opportunity for
Cognitive Scientists to Integrate Research and Practice. Williams, Renkl,
Koedinger, Stamper.
AERA 2014– American Education Research Association. How online resources can
facilitate interdisciplinary collaboration. Williams, Thille, Trumbore, Stigler, Siemens.
ICLS (under review) – International Conference for the Learning Sciences. Online &
Blended Education: Opportunities and Obstacles for the Learning Sciences.
Williams, Linn, Slotta, Rose, Dillenbourg.
CHI 2014 – Computer Human Interaction. Learning Innovations at Scale. Williams,
Kizilcec, Russell, Klemmer.
•
•
•
E-learning guild’s DevLearn
Educational Data Mining
Artificial Intelligence in Education
•
•
•
Computer Supported Collaborative Learning
Professional & Organizational Development Network
American Society for Training & Development

Laboratory Paradigms & Online Products
• A matching problem
• www.edsurge.com,
www.edudemic.com
• josephjaywilliams@stanford.edu
•
Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, & Aleahmad,
T. (2014). Leveraging the Internet to do Laboratory Research in the
Real World. Symposium to be conducted at the annual convention
of the Association for Psychological Science. San Francisco, CA.
19

Example: Explanation & Learning
•
Williams & Lombrozo (2010, Cognitive Science; 2013,
Cognitive Psychology; w Rehder, JEP: General)
Glorp
Right
Drent
Wrong
20

EdSurge Meetup Wed Feb 19 7 pm CERAS
• “Improving Ed-Tech using Research on
Learning & Advancing Research on
Learning using Ed-Tech”
• Ed-Tech Startup Survey
• Researcher Survey
• Featured Company & Researcher
• Panelist & Question suggestions?
21

Completely New Horizons for Experimentation
• “Bridging Studies”
• www.josephjaywilliams.com/experime
nts-online-education
•
Williams, J. J. (2013). Enhancing Educational Research &
Practice using Experiments on Online Educational
Resources. Talk at Pittsburgh Science of Learning Center
LearnLab Summer School, Pittsburgh, PA.
tiny.cc/onlineexperiments
22

Next: Teaching Learning Strategies
• Limited teaching time: Specific content
or general strategies?
• Online Advantage:
Easier to run RCTs & collect longitudinal data
Reinforce new educational habits & behaviors
• Teach metacognitive self-questioning
strategies
(Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams &
Lombrozo, 2010, Cognitive Science)
23

Drop-down text about strategies
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]

Learning Coach
• Tiny.cc/kalearningcoach
25

MOOC experiment
• Motivational Video
Control
Energetic Thank you & Encouragement
Growth Mindset (by Instructor)
Growth Mindset (outside Instructor)
“Belonging”
• Questions for reflection after video
None VS Just Think VS Write response
26

MOOClets Group – Develop Digital Modules
•
•
•
•
•
•
•
•
•
E.g. MOOClet: Math exercise, 5 minute interactive video, 30
minute lesson, Study Strategy coach, Quiz/Assessment
Instructors can use for blended classroom education (High
School, Community College, University)
Continuously in use and being improved
Easy to build, prototype, test & iterate
Researchers directly implement & collect data
Small scale & online access facilitates multidisciplinary
collaboration & research-practitioner development (see
AERA 2014 invited panel)
Illustrate/Operationalize instructional design principles
MOOClet Reading & Discussion Group meets Tue 11-12 PST @
Barnum (Stanford) or tiny.cc/lyticshangout
Multidisciplinary group of researchers & practitioners
(tiny.cc/mooclet)
27

Review
•
•
•
•
•
•
•
MOOCs VS Digital Online Resources
Experiments: Link Real World & Laboratory
Motivational Messages Experiment
Simultaneous Basic & Applied Research
Ed-Surge Meetup to link Research & Practice (Wed
Feb 19 @ 7)
Current Directions
Teaching Learning Strategies
Learning Coach
MOOC Experiment
MOOClets Research & Practice Group Meets
Tuesdays 11-12 PST (tiny.cc/lyticshangout,
tiny.cc/mooclet)
28

Acknowledgements
•
•
•
•
•
•
Jascha Sohl-Dickstein
Jace Kohlmeier & Khan Academy
PERTS
Sam Maldonado
Lytics Lab (lytics.stanford.edu)
VPOL (Vice Provost of Online Learning,
online.stanford.edu)
29