Using Experiments and Cognitive Science
Research to Improve the Design of Online
Resources for Learning
Joseph Jay William...
Online Education & Learning Online
• New research area?
• Convergence of
computational &
behavioral science
NIPS “Data-Dri...
Novel Research Opportunity: Real-World +
Laboratory
Integrate Research & Practice

• Randomized assignment
• Experimental Control
• Rich data

• Generalizable theories
• “in ...
Overview
•
•
•
•
•

Explanation & Learning
Teaching Learning Strategies
Motivational Messages
Experimental Paradigm
Experi...
Why does explaining “why?” help learning?
• General boost to Learning Engagement vs.
• The Subsumptive Constraints Account...
Explanation and Learning: Lab  Online
• Discovery & transfer

GLORP
(Williams & Lombrozo, 2010, Cognitive

Science)

• Us...
Learning Task & Experimental Paradigm
• Online (Math) Exercise
1. Number of
Problems
Completed
2. Percent
Correct

8
Logic of my Previous Research
Explain why this is
correct.
Elaborate on
what you are
thinking now.

•
•
•
•

Post-Study te...
Future: Generate, Receive, Compare
Generate

E.g. Explain
why this is
correct.
Receive

Compare

Geza Kovacs

Simultaneous...
Teaching Learning Strategies
• Spend time teaching specific content, or
general strategies?
• Online: Collect data that wo...
Experimentally manipulate additional prompts
Clickable link. + Prompts embedded into hints
[Click here to learn about the ...
Self-questioning strategy: What? Why? How?
Embedded Prompts between Hint/Solution
Steps
Prompts
Prompt
type

Promptedunderstanding

What?

What does this step
mean to you?

Why?

Why is what you are
currently d...
Explanation study
Practice-asusual

Control

Self-Regulating
Thinking

Reflecting on
Meaning

Problems
Attempted
Problems
...
Explanation study
Practice-as-usual

No Textbox

Textbox

Problems Attempted
Problems Correct
Practice Tasks
Completed
Mas...
Learning Behavior Support
• Clickable link to Drop-Down text with
suite of strategies:
Are you stuck?
Click here for some ...
Natural Link to Learning Strategy Training
• If you want to learn more about
strategies to keep motivated and
learn well, ...
3. Add motivational messages
Practice-as-usual
Growth Mindset Message
Remember, the more you practice the smarter you beco...
3. Embedded in vivo Experiment
• Benefit of Growth Mindset Message?
• Practice-as-usual

Jascha Sohl-Dickstein

• Growth M...
Results: More motivated?
• Growth Mindset Message > Practice-asUsual
• extra problems attempted
• more problems correct
• ...
3. Add motivational messages
Practice-as-usual Message
Growth Mindset Message
Positive
Remember, the more you practice the...
Does any positive message work?

• Practice-as-usual
• Growth Mindset
Message
•
•

"Remember, the more you practice
the sm...
Effects of Positive Messages?
• Positive Messages ~= Practice-asUsual
• Growth Mindset > Positive
• extra problems attempt...
Computational Modeling
•

•
•
•
•
•

Williams, Mitchell, Heffernan. MOOC Research Initiative grant from
Gates Foundation &...
Synthesize Scientific Findings
•

Williams, J.J. (2013)Improving Learning in MOOCs
by Applying Cognitive Science. Paper pr...
Experimental Paradigm: R.E.P.E.A.T.
•

•

Williams, J. J. (2013). Finding connections between
basic experimental research ...
Experiments
•

•

Williams, J.J. & Williams, B.A. (under review). Online
A/B Tests & Experiments: A Practical But
Scientif...
Experiment-Focused Design
• Williams, J.J. & Williams, B. A. (2013).
Using Interventions to Improve Online
Learning. Paper...
Review
•
•
•
•
•
•

Explanation & Learning
Teaching Learning Strategies
Motivational Messages
Experimental Paradigm
Experi...
Acknowledgements
•
•
•
•
•

Jascha Sohl-Dickstein
Jace Kohlmeier & Khan Academy
Sam Maldonado
Lytics Lab (lytics.stanford....
Upcoming SlideShare
Loading in …5
×

Using Experiments and Cognitive Science Research to Improve the Design of Online Resources for Learning

818 views

Published on

The recent explosion of online educational resources has the potential to reorganize how we learn – from K-12 and university to the workplace and the informal learning we do every day. It also raises new questions and opportunities for research that crosses the many disciplines relevant to designing computer programs that help people learn. For example, HCI and cognitive science can provide complementary perspectives in investigating how to design the content and instructional features of an online course, such that a person processes and stores that information in a way that successfully guides their future behavior. Online educational environments provide new optimism in tackling challenges like these because they can be instrumented to collect an unprecedented scale and diversity of data, and allow iterative sequences of experiments to be embedded in authentic educational contexts with real students.
This talk presents one approach to this kind of research, using experimental comparisons to test the effects of modifying online mathematics exercises to include motivational messages and question prompts for people to explain, the design of which is guided by the psychological literature on motivation and learning. A combination of laboratory experiments and experiments embedded in real-world online education platforms (like www.KhanAcademy.org) reveal that prompting people to explain “why?” facts are true drives them beyond memorization to uncover underlying principles and patterns, and that teaching such self-questioning strategies may accelerate student learning. Motivational messages appear to have limited benefits if they are simply encouraging or aimed at raising confidence, but do increase how much effort students invest if the messages emphasize that aptitude is malleable and can be improved through persistence. Several planned experiments are presented which also use this paradigm of adding minimal but effective textual changes to online exercises to achieve practical impact and explore basic cognitive science questions about learning.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
818
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Originally from Trinidad. I’m a Research Fellow in theLytics Lab & Office of Online Learning. Cognitive Science background – experimental psychology, statistical modeling and machine learning.Core interest is being a knowledge broker – reviewing and synthesizing the thousands of published studies in cognitive science, education research, the learning sciences to see which ones are relevant to real-world outcomes, and building products or conducting experiments that improve practically ad financially valuable outcomes. Illustrate this approach with an experiment that shows how to improve students’ motivation while learning from mathematics problems on Khan Academy.
  • Not traditional, but ubiquitous
  • Not traditional, but ubiquitous
  • Using Experiments and Cognitive Science Research to Improve the Design of Online Resources for Learning

    1. 1. Using Experiments and Cognitive Science Research to Improve the Design of Online Resources for Learning Joseph Jay Williams josephjaywilliams@stanford.edu www.josephjaywilliams.com/researchoverview 1
    2. 2. Online Education & Learning Online • New research area? • Convergence of computational & behavioral science NIPS “Data-Driven Education” New ACM conference “Learning at Scale” CHI
    3. 3. Novel Research Opportunity: Real-World + Laboratory
    4. 4. Integrate Research & Practice • Randomized assignment • Experimental Control • Rich data • Generalizable theories • “in vivo” experiments • Diverse populations • Real-world environment • Authentic activities • Practical Challenges • Practical improvements • Disseminate research • Generate Funding
    5. 5. Overview • • • • • Explanation & Learning Teaching Learning Strategies Motivational Messages Experimental Paradigm Experiment-focused Design 5
    6. 6. Why does explaining “why?” help learning? • General boost to Learning Engagement vs. • The Subsumptive Constraints Account: Interpret target of why-explanation in terms of a broader generalization (Williams & Lombrozo, 2010) • • 2x3=6 Why? 6
    7. 7. Explanation and Learning: Lab  Online • Discovery & transfer GLORP (Williams & Lombrozo, 2010, Cognitive Science) • Use of prior knowledge (Williams & Lombrozo, 2013, Cog. Psych.) • Erroneously overgeneralize at expense of exceptions • Promotes belief revision – given sufficient anomalies (Williams et al, 2013, JEP: General) (Williams, Walker, Maldonado & Lombrozo, 2012; 2013, Cog Sci Conference; in prep) • Prompts in online (math) exercises (Williams, Paunesku, Haley, Sohl-Dickstein, 2013, AIED Moocshop; ongoing) DRENT
    8. 8. Learning Task & Experimental Paradigm • Online (Math) Exercise 1. Number of Problems Completed 2. Percent Correct 8
    9. 9. Logic of my Previous Research Explain why this is correct. Elaborate on what you are thinking now. • • • • Post-Study test questions Transfer/Generalization questions Questions about key principle Memory for details 9
    10. 10. Future: Generate, Receive, Compare Generate E.g. Explain why this is correct. Receive Compare Geza Kovacs Simultaneously Learning AND Crowdsourcing Improvement of Learning Resources Williams, Thille, Siemens, Trumbore, Stigler. How online resources can facilitate interdisciplinary collaboration. Invited talk to be presented at SIG on Computer and Internet Applications in Education, AERA 10 2014.
    11. 11. Teaching Learning Strategies • Spend time teaching specific content, or general strategies? • Online: Collect data that would be extremely difficult to get in the real world • Online: Repeatedly reinforce habits & educational behaviors • Teach “What? Why? How?” selfquestioning/explanation strategies (Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science) • Understanding vs. Problem-solving • vs. Interpreting vs. Practice-as-Usual 11
    12. 12. Experimentally manipulate additional prompts Clickable link. + Prompts embedded into hints [Click here to learn about the “What? Why? How?” strategy]
    13. 13. Self-questioning strategy: What? Why? How?
    14. 14. Embedded Prompts between Hint/Solution Steps
    15. 15. Prompts Prompt type Promptedunderstanding What? What does this step mean to you? Why? Why is what you are currently doing Why is it helpful to helpful? Why is it take this step? useful for achieving your goal? How? How well is your current approach to this problem working? How do you know this step is right? Promptedproblem-solving Promptedinterpretation What are you doing What is this step or thinking right saying? Restate it in now? your own words.
    16. 16. Explanation study Practice-asusual Control Self-Regulating Thinking Reflecting on Meaning Problems Attempted Problems Correct Practice Tasks Completed Mastery Tasks Completed Problem Accuracy 16
    17. 17. Explanation study Practice-as-usual No Textbox Textbox Problems Attempted Problems Correct Practice Tasks Completed Mastery Tasks Completed Problem Accuracy 17
    18. 18. Learning Behavior Support • Clickable link to Drop-Down text with suite of strategies: Are you stuck? Click here for some tips. • Provide previously examined prompts. • Use mouseover and drop-down text to reveal information “as requested”, rich traversal of options, guided by student
    19. 19. Natural Link to Learning Strategy Training • If you want to learn more about strategies to keep motivated and learn well, go to tiny.cc/learningassistant or XX or YY
    20. 20. 3. Add motivational messages Practice-as-usual Growth Mindset Message Remember, the more you practice the smarter you become! 20
    21. 21. 3. Embedded in vivo Experiment • Benefit of Growth Mindset Message? • Practice-as-usual Jascha Sohl-Dickstein • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” 21
    22. 22. Results: More motivated? • Growth Mindset Message > Practice-asUsual • extra problems attempted • more problems correct • Percent Correct: Problems correct/Problems attempted • increase in Percent Correct 22
    23. 23. 3. Add motivational messages Practice-as-usual Message Growth Mindset Message Positive Remember, the more you practice the smarter you become! Some of these problems are hard. Do your best! 23
    24. 24. Does any positive message work? • Practice-as-usual • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” • Positive Message • • "Some of these problems are hard. Just do your best." "This might be a tough problem, but we know you can do it.” 24
    25. 25. Effects of Positive Messages? • Positive Messages ~= Practice-asUsual • Growth Mindset > Positive • extra problems attempted • more problems correct • increase in Percent Correct 25
    26. 26. Computational Modeling • • • • • • Williams, Mitchell, Heffernan. MOOC Research Initiative grant from Gates Foundation & Athabasca. Investigating the benefits of embedding motivational messages in online exercises. 2 million users on 12 kinds of fractions exercises, ~100 problems each Moderators & Mediators Item Response Theory Non-parametric Bayesian clustering of Users (CrossCat, JMLR) Model latent knowledge states 26
    27. 27. Synthesize Scientific Findings • Williams, J.J. (2013)Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN. • www.josephjaywilliams.com/education 27
    28. 28. Experimental Paradigm: R.E.P.E.A.T. • • Williams, J. J. (2013). Finding connections between basic experimental research and realistic online education contexts. In J. J. Williams (chair), Online Learning and Psychological Science: Opportunities to integrate research and practice. Symposium conducted at the annual convention of the Association for Psychological Science. Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. 28 Austin, TX: Cognitive Science Society. (pdf)
    29. 29. Experiments • • Williams, J.J. & Williams, B.A. (under review). Online A/B Tests & Experiments: A Practical But Scientifically Informed Introduction. Course proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. (pdf) Williams, J.J., Heffernan, N., & Koedinger, K. Experiments at Scale: Instrumenting MOOCs for experimentation and course-improving data analysis. Tutorial proposal submitted to the First Annual ACM Conference on Learning at Scale. 29
    30. 30. Experiment-Focused Design • Williams, J.J. & Williams, B. A. (2013). Using Interventions to Improve Online Learning. Paper to be presented at the NIPS 2013 Workshop on Data Driven Education. 30
    31. 31. Review • • • • • • Explanation & Learning Teaching Learning Strategies Motivational Messages Experimental Paradigm Experiment-focused Design Williams, J.J., Klemmer, S., Kizilcec, R., & Russel, D. (under review). Learning Innovations at Scale. Workshop proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. 31
    32. 32. Acknowledgements • • • • • Jascha Sohl-Dickstein Jace Kohlmeier & Khan Academy Sam Maldonado Lytics Lab (lytics.stanford.edu) VPOL (Vice Provost of Online Learning, online.stanford.edu) 32

    ×