Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How online educational resources provide novel
affordances for conducting practical interventions
and doing psychology exp...
Overview
•
•
•
•
•
•

MOOCs VS Digital Online Resources
Experiments: Link Real World & Laboratory
Motivational Messages Ex...
Online Education
Moving from Physical to Digital World
Experiments: Link Real World & Laboratory

• Randomized assignment
• Experimental Control
• Rich data

• Real-world
enviro...
Experiment
• Practical goal on Khan Academy
• Extend motivation research

6
Motivate students to practice math
• Online (Math) Exercise
Motivational Message: Keep up the good work!

1. Number of
Pro...
Add motivational messages
Practice-as-usual
Growth Mindset Message
Remember, the more you practice the smarter you become!...
Which messages to add?
•
•
•
•
•
•

"This might be a tough problem, but we know
you can do it!”
Remember, the more you pra...
Embedded in vivo Experiment
Jascha Sohl-Dickstein

• Practice-as-usual
• Growth Mindset
Message
•
•

"Remember, the more y...
Results: More motivated?
• Growth Mindset Message > Practice-asUsual
• reduction in dropout rate
• increase in Number of p...
Interpreting results
• Obvious?
• Skepticism: Anything’s statistically
signficant in a MOOC

12
Different motivational messages
Practice-as-usual Message
Growth Mindset Message
Positive
Remember, the more you practice ...
Does any positive message work?

• Practice-as-usual
• Growth Mindset
Message
•
•

"Remember, the more you practice
the sm...
Effects of Positive Messages?
• Positive Messages not better than
Practice-as-Usual!
• Growth Mindset better than Positive...
Next: Situationally appropriate messages

• Message blast, can be ignored
• Providing motivational message Justin-Time: af...
Simultaneous Applied & Basic Research
• Start with “Product”
• Constraints

Randomization (Technical, use Qualtrics)
Behav...
Multidisciplinary – Relevant Communities
•

•
•
•

•

Association for Psychological Science 2013. Online Learning and Psyc...
Laboratory Paradigms & Online Products
• A matching problem
• www.edsurge.com,
www.edudemic.com
• josephjaywilliams@stanfo...
Example: Explanation & Learning
•

Williams & Lombrozo (2010, Cognitive Science; 2013,
Cognitive Psychology; w Rehder, JEP...
EdSurge Meetup Wed Feb 19 7 pm CERAS

• “Improving Ed-Tech using Research on
Learning & Advancing Research on
Learning usi...
Completely New Horizons for Experimentation

• “Bridging Studies”
• www.josephjaywilliams.com/experime
nts-online-educatio...
Next: Teaching Learning Strategies
• Limited teaching time: Specific content
or general strategies?
• Online Advantage:
Ea...
Drop-down text about strategies
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]
Learning Coach
• Tiny.cc/kalearningcoach

25
MOOC experiment
• Motivational Video
Control
Energetic Thank you & Encouragement
Growth Mindset (by Instructor)
Growth Min...
MOOClets Group – Develop Digital Modules
•

•
•
•
•
•
•
•
•

E.g. MOOClet: Math exercise, 5 minute interactive video, 30
m...
Review
•
•
•
•
•

•

•

MOOCs VS Digital Online Resources
Experiments: Link Real World & Laboratory
Motivational Messages ...
Acknowledgements
•
•
•
•
•
•

Jascha Sohl-Dickstein
Jace Kohlmeier & Khan Academy
PERTS
Sam Maldonado
Lytics Lab (lytics.s...
Upcoming SlideShare
Loading in …5
×

How online educational resources provide novel affordances for conducting practical interventions and doing psychology experiments

859 views

Published on

Published in: Education, Technology
  • Hello there! Get Your Professional Job-Winning Resume Here! http://bit.ly/topresum
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

How online educational resources provide novel affordances for conducting practical interventions and doing psychology experiments

  1. 1. How online educational resources provide novel affordances for conducting practical interventions and doing psychology experiments www.josephjaywilliams.com/experiments-online-education Joseph Jay Williams, Stanford University Lytics Lab (lytics.stanford.edu), Graduate School of Education Office of the Vice Provost of Online Learning (vpol.stanford.edu) josephjaywilliams@stanford.edu www.josephjaywilliams.com Originally from Trinidad – apologies for thick accent! 1
  2. 2. Overview • • • • • • MOOCs VS Digital Online Resources Experiments: Link Real World & Laboratory Motivational Messages Experiment Simultaneous Basic & Applied Research Ed-Surge Meetup to link Research & Practice (Wed Feb 19 @ 7) Current Directions Teaching Learning Strategies Learning Coach MOOC Experiment • MOOClets Research & Practice Group 2
  3. 3. Online Education
  4. 4. Moving from Physical to Digital World
  5. 5. Experiments: Link Real World & Laboratory • Randomized assignment • Experimental Control • Rich data • Real-world environment • Authentic activities • Practical Challenges
  6. 6. Experiment • Practical goal on Khan Academy • Extend motivation research 6
  7. 7. Motivate students to practice math • Online (Math) Exercise Motivational Message: Keep up the good work! 1. Number of Problems Practiced 2. Improve Accuracy 7
  8. 8. Add motivational messages Practice-as-usual Growth Mindset Message Remember, the more you practice the smarter you become! 8
  9. 9. Which messages to add? • • • • • • "This might be a tough problem, but we know you can do it!” Remember, the more you practice the smarter you become! Some of these problems are hard. Just do your best.” Encourage “Growth Mindset” beliefs about intelligence (Dweck, 2008) Fixed Mindset Growth Mindset 9
  10. 10. Embedded in vivo Experiment Jascha Sohl-Dickstein • Practice-as-usual • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” 10
  11. 11. Results: More motivated? • Growth Mindset Message > Practice-asUsual • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy: Problems correct/Problems attempted • Accuracy improvement 11
  12. 12. Interpreting results • Obvious? • Skepticism: Anything’s statistically signficant in a MOOC 12
  13. 13. Different motivational messages Practice-as-usual Message Growth Mindset Message Positive Remember, the more you practice the smarter you become! Some of these problems are hard. Do your best! 13
  14. 14. Does any positive message work? • Practice-as-usual • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” • Positive Message • • "Some of these problems are hard. Just do your best." "This might be a tough problem, but we know you can do it.” 14
  15. 15. Effects of Positive Messages? • Positive Messages not better than Practice-as-Usual! • Growth Mindset better than Positive • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy improvement 15
  16. 16. Next: Situationally appropriate messages • Message blast, can be ignored • Providing motivational message Justin-Time: after incorrect answer • Laboratory experiments • www.assistments.org Neil Heffernan 16
  17. 17. Simultaneous Applied & Basic Research • Start with “Product” • Constraints Randomization (Technical, use Qualtrics) Behavioral/Learning Measures Williams & Williams (2013). Using Interventions to improve Online Learning. NIPS Data Driven Education workshop. tiny.cc/experimentsonlinelearning • Consider Synthesis Williams, J.J. (2013) Improving Learning in MOOCs by Applying Cognitive Science. AIED MOOCshop & NIPS Data-Driven Workshop. tiny.cc/cognitivescienceinmoocs Reading List: www.josephjaywilliams.com/education Improving outcome of psychosocial treatments by enhancing memory and learning.(in press) Perspectives in Psychological Science. • High Investment but High Return 17
  18. 18. Multidisciplinary – Relevant Communities • • • • • Association for Psychological Science 2013. Online Learning and Psychological Science: Opportunities to integrate research and practice. Williams, Saxberg, Means, Mitros. Cognitive Science Society 2013. Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. Williams, Renkl, Koedinger, Stamper. AERA 2014– American Education Research Association. How online resources can facilitate interdisciplinary collaboration. Williams, Thille, Trumbore, Stigler, Siemens. ICLS (under review) – International Conference for the Learning Sciences. Online & Blended Education: Opportunities and Obstacles for the Learning Sciences. Williams, Linn, Slotta, Rose, Dillenbourg. CHI 2014 – Computer Human Interaction. Learning Innovations at Scale. Williams, Kizilcec, Russell, Klemmer. • • • E-learning guild’s DevLearn Educational Data Mining Artificial Intelligence in Education • • • Computer Supported Collaborative Learning Professional & Organizational Development Network American Society for Training & Development
  19. 19. Laboratory Paradigms & Online Products • A matching problem • www.edsurge.com, www.edudemic.com • josephjaywilliams@stanford.edu • Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, & Aleahmad, T. (2014). Leveraging the Internet to do Laboratory Research in the Real World. Symposium to be conducted at the annual convention of the Association for Psychological Science. San Francisco, CA. 19
  20. 20. Example: Explanation & Learning • Williams & Lombrozo (2010, Cognitive Science; 2013, Cognitive Psychology; w Rehder, JEP: General) Glorp Right Drent Wrong 20
  21. 21. EdSurge Meetup Wed Feb 19 7 pm CERAS • “Improving Ed-Tech using Research on Learning & Advancing Research on Learning using Ed-Tech” • Ed-Tech Startup Survey • Researcher Survey • Featured Company & Researcher • Panelist & Question suggestions? 21
  22. 22. Completely New Horizons for Experimentation • “Bridging Studies” • www.josephjaywilliams.com/experime nts-online-education • Williams, J. J. (2013). Enhancing Educational Research & Practice using Experiments on Online Educational Resources. Talk at Pittsburgh Science of Learning Center LearnLab Summer School, Pittsburgh, PA. tiny.cc/onlineexperiments 22
  23. 23. Next: Teaching Learning Strategies • Limited teaching time: Specific content or general strategies? • Online Advantage: Easier to run RCTs & collect longitudinal data Reinforce new educational habits & behaviors • Teach metacognitive self-questioning strategies (Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science) 23
  24. 24. Drop-down text about strategies Clickable link. [Click here to learn about the “What? Why? How?” strategy]
  25. 25. Learning Coach • Tiny.cc/kalearningcoach 25
  26. 26. MOOC experiment • Motivational Video Control Energetic Thank you & Encouragement Growth Mindset (by Instructor) Growth Mindset (outside Instructor) “Belonging” • Questions for reflection after video None VS Just Think VS Write response 26
  27. 27. MOOClets Group – Develop Digital Modules • • • • • • • • • E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment Instructors can use for blended classroom education (High School, Community College, University) Continuously in use and being improved Easy to build, prototype, test & iterate Researchers directly implement & collect data Small scale & online access facilitates multidisciplinary collaboration & research-practitioner development (see AERA 2014 invited panel) Illustrate/Operationalize instructional design principles MOOClet Reading & Discussion Group meets Tue 11-12 PST @ Barnum (Stanford) or tiny.cc/lyticshangout Multidisciplinary group of researchers & practitioners (tiny.cc/mooclet) 27
  28. 28. Review • • • • • • • MOOCs VS Digital Online Resources Experiments: Link Real World & Laboratory Motivational Messages Experiment Simultaneous Basic & Applied Research Ed-Surge Meetup to link Research & Practice (Wed Feb 19 @ 7) Current Directions Teaching Learning Strategies Learning Coach MOOC Experiment MOOClets Research & Practice Group Meets Tuesdays 11-12 PST (tiny.cc/lyticshangout, tiny.cc/mooclet) 28
  29. 29. Acknowledgements • • • • • • Jascha Sohl-Dickstein Jace Kohlmeier & Khan Academy PERTS Sam Maldonado Lytics Lab (lytics.stanford.edu) VPOL (Vice Provost of Online Learning, online.stanford.edu) 29

×