Learning Engineering of MOOClets: Simultaneously
benefiting Professional Learning, Financial Success,
and Cognitive & Lear...
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
I. Motivation
• Beliefs about Intelligence
• To what extent do you agree?
How smart you are can be significantly
changed b...
MOOClet 1: Exercise (Khan Academy)
5
1. Number of
Problems
Practiced
2. Accuracy
[TEXT SHOWN HERE – Randomize]
Embed motivational messages
Practice-as-usual
Remember, the more you practice the smarter you become!
Growth Mindset Messa...
Growth Mindset Message helpful?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "M...
Results: More motivated?
• Growth Mindset Message > Practice-as-
Usual
• reduction in dropout rate
• increase in Number of...
Interpreting results
• Obvious?
• Skeptical? Anything’s statistically
significant with large N…
• Second experiment (run s...
Practice-as-usualGrowth Mindset MessagePositive Message
Different motivational messages
Some of these problems are hard. D...
Does any positive message work?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "M...
Effects of Positive Messages?
• Positive Messages not better than Practice-as-
Usual!
• Growth Mindset > Positive Message
...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Constraint Satisfaction
Practical Product
Research
Financial Value 14
“MOOClets” or Modular Online Resources
• E.g. MOOClet: Math exercise, 5 minute
interactive video, 30 minute lesson, Study
...
Integrate Diverse Expertise & Innovations
16
Where is relevant research, best practice, technology, funding,
market? tiny.cc/mapofspace, tiny.cc/stanfordmap
MOOCs
(Cou...
Relevant Resources
• www.josephjaywilliams.com/home/useful-
software
• www.josephjaywilliams.com/education
• www.josephjay...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Messages in Coursera?
20
Week 6 Quiz
Whether or not you’re correct on the first try, each attempt at a tough problem can
m...
Motivational Introductions
21
Practice-as-usual
Have fun in the course!
Anne Friedlander Growth MindsetText version of Min...
Instructor Guide to foster Growth Mindset
• tiny.cc/idstrategymindset
22
Where do you think you will apply this?
Do you ha...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
II. Learning through Questions/Assessments
• Discovery of patterns
(Williams & Lombrozo, 2010, Cognitive Science)
• Use of...
Adding questions/activities
• tiny.cc/augmentedvideo
• tiny.cc/augmentedexercise
26
III. General Learning & Problem-Solving Skills
27
Learning/Self-Questioning Strategies
• Specific Prompt VS.
• General Strategy of Self-Questioning?
• Teach strategies
(McN...
Embedded “On-Request” Strategy Instruction
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]
29
Self-questioning strategy: What? Why? How?
30
Cues – Reflect & Remind
31
Preliminary Results
32N = ~ 800 per condition
• Prompted Problem-solving better than
Practice-as-usual.
• Increased Accura...
Learning Assistant
• tiny.cc/learningassistant
33
Learning Assistant – e.g. 1
34
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
IV. Performance Support Tools & Checklists
37
Coach/Support Learning Behaviors
tiny.cc/kalearningcoach
38
A Lesson: Concept + Digital Tool to apply
Concept in Future
39
Tool for Difficult Conversations
40
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Acknowledgements
• Tania Lombrozo & Tom Griffiths
• John Mitchell & Candace Thille
• Jascha Sohl-Dickstein
• Jace Kohlmeie...
Upcoming SlideShare
Loading in …5
×

Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research

287 views
244 views

Published on

Invited talk given at Declara

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
287
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide







  • Originally from Trinidad. Thank you for having me here, it’s a pleasure because of how this entire movement has impacted me. Two years ago was finishing my dissertation work in cognitive science – psychology experiments on learning, & computational psychology, or using Bayesian statistics & machine learning to model human thinking, but I had been doing work in the education school, and wanted to have a real-world impact, and actually build products. Seemed impossible to do both, but all of a sudden – online education exploded!

    Tremendously excited, started research in collaboration with Khan Academy, and have been trying to add value by bringing to bear scientific knowledge about learning in a way that’s actually practically useful, and linking researchers to these practical platforms, like writing grants with Khan Academy & Stanford Online. So I will present a subset of that work.

    But it will be different from a typical research talk, because I took such an amazing group as a chance to share explicitly on the approach I take to developing online educational materials, point to resources people here might also find useful, and emphasize some broad themes in online learning where I think real-world impact and science converge.

    Interested in discussion & feedback from you on these – you have thought about it a tremendous deal here at Coursera, and much more familiar with your and others’ platforms. Don’t hesitate to make suggestions or state different opinion, I’d like to learn more about your thinking on these issues.



    Start with practical or “product-first”. Not necessarily a typical focus for researchers, except Candace Thille, Ken Koedinger, ?
    50% is primary research, which I’ll present. 50% synthesizing existing work, having broad view of technology, entrepreneurship. I like collecting these list, and think there’s a lot of value to be tapped.

    Take a business perspective – where is there unrealized value, because of lack of information?

    So I’m going to discuss ongoing work with Khan Academy & MOOCs and residential education.



    Atypical Trajectory, which means the talk has many moving pieces: Use digital online educational resources to do use-inspired basic research.

    Extremely tough to link research and practice, so the last few years I’ve taken positions that allowed me to invest a lot of time into having a broad view of the emerging trends – understanding technology and goals of the MOOC platforms, Khan Academy, hundreds of Startups in Bay Area, as well as e-learning in the corporate training & workforce development world.

    Also been trying to get a broad view of which research topics and themes are particularly well suited to these contexts, from cognitive psychology to educational topics, social & clinical, health behavior change. Methodologies from HCI, education policy.
  • Collaborative Development
    Guided by Practical and Research Goals

    ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  • o.32
    0.27
  • Collaborative Development
    Guided by Practical and Research Goals

    ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  • Have an unusual background, so I thought it might help to provide some context.
    Approach: Use digital educational resources to do inspired basic research (e.g. math exercises), understand how cognitive processes like explaining can guide learning & how people’s beliefs influence motivation. (Pasteur’s Quadrant).
    Principles & Demonstrations for doing this kind of work laid by PSLC.


    ust to provide some context about Use digital online An interdisciplinary approach to linking research and practice,PhD training is a mixture of Experimental Psychology, Computational Psychology (using methods from machine learning & Bayesian statistics to model cognition),


  • Exhausting… but exciting!

    Criticized for not using literature… but don’t have time to.
    Researchers haven’t looked at & thought enough about platforms, to synthesize the key ideas. Also rare to have the time to take a broad view.
  • Collaborative Development
    Guided by Practical and Research Goals

    ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  • Add textbox below first two questions for people to type into.

    Add some hyperlink looking things below “Click … case studies” with something like these fields:
    Encouraging students to see literary skills as malleable through class emails
    Improving persistence in Database Course programming assignments
    Enhancing receptiveness to peer feedback in management courses

    Under would you like to, same kind of thing, but with:
    Learn more about the research behind this concept
    Ask a question or share thoughts with the Teach-Net mailing list
    Request a future reminder for applying this guide
  • Collaborative Development
    Guided by Practical and Research Goals

    ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  • Not traditional, but ubiquitous

    A challenging choice… Take the next theoretical step in this line of research, with a new set of materials.

    On the other hand, this is venturing into new territory. PSLC & others been successful, they have full control.
    Also, since there’s little direct past research in many of these contexts, don’t even know yet what form the phenomenon takes.

    Start with a practical challenge first, seek to improve it by bringing to bear research. Establishing whether there is an effect, sets the stage for using this online context to probe its underlying mechanisms. Jump to the other side, use-inspired basic research.
  • Collaborative Development
    Guided by Practical and Research Goals

    ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  • Collaborative Development
    Guided by Practical and Research Goals

    ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  • Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research

    1. 1. Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research Joseph Jay Williams Lytics Lab, Graduate School of Education Office of the Vice Provost for Online Learning, Stanford josephjaywilliams@stanford.edu www.josephjaywilliams.com 1 [Originally from Trinidad & Tobago – apologies for difficulty in following accent.]
    2. 2. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 2
    3. 3. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 3
    4. 4. I. Motivation • Beliefs about Intelligence • To what extent do you agree? How smart you are can be significantly changed by your effort and hard work. You have a certain amount of intelligence, and what you do doesn’t really change that much. • Growth Mindset vs. Fixed Mindset • Dweck, 2008; Yeager & Walton, 2012 • Williams, Paunesku, Haley & Sohl-Dickstein, 2013 4
    5. 5. MOOClet 1: Exercise (Khan Academy) 5 1. Number of Problems Practiced 2. Accuracy [TEXT SHOWN HERE – Randomize]
    6. 6. Embed motivational messages Practice-as-usual Remember, the more you practice the smarter you become! Growth Mindset Message 6
    7. 7. Growth Mindset Message helpful? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Practice-as-usual Jascha Sohl-Dickstein 7
    8. 8. Results: More motivated? • Growth Mindset Message > Practice-as- Usual • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy: Problems correct/Problems attempted • Accuracy improvement 8
    9. 9. Interpreting results • Obvious? • Skeptical? Anything’s statistically significant with large N… • Second experiment (run simultaneously) 9
    10. 10. Practice-as-usualGrowth Mindset MessagePositive Message Different motivational messages Some of these problems are hard. Do your best! Remember, the more you practice the smarter you become! 10
    11. 11. Does any positive message work? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Positive Message • "Some of these problems are hard. Just do your best." • "This might be a tough problem, but we know you can do it.” • Practice-as-usual 11
    12. 12. Effects of Positive Messages? • Positive Messages not better than Practice-as- Usual! • Growth Mindset > Positive Message • How intuitive is this result? • Teachers? • Ongoing: After incorrect answer • www.assistments.org Neil Heffernan • Gates Grant for Modeling Research 12Rapid Experimentation: tiny.cc/mechanicalturk
    13. 13. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 13
    14. 14. Constraint Satisfaction Practical Product Research Financial Value 14
    15. 15. “MOOClets” or Modular Online Resources • E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment • www.josephjaywilliams.com/mooclets • MOOClets Group Tue 12-1 PST (remote) • tiny.cc/joinmooclets mailing list & updates • Williams, J.J. & Williams, B. A. (2013). Randomized Experiments as a Methodological and Conceptual Tool for Improving Design of Online Learning. Data Driven Education, NIPS Workshop. tiny.cc/experimentsonlinelearning 15
    16. 16. Integrate Diverse Expertise & Innovations 16
    17. 17. Where is relevant research, best practice, technology, funding, market? tiny.cc/mapofspace, tiny.cc/stanfordmap MOOCs (Coursera, EdX, Udacity) www.KhanAcademy.org www.Assistments.org K-12 Ed-Tech space (Bay Area, Boston...) (www.edsurge.com) (www.edudemic.com ) Higher-Ed Technology/LMS E-Learning & Training for Corporate, State, Federal, Army Psychological Scientists Computer Scientists Education Corporate Training Funding Agencies: IES, Dept. of Education, Dept. of Labor, NIH, NSF Practical Reports: What Works Clearinghouse IES Practice Guides University Teaching & Learning Centers Department of Education reports Think Tanks: SRI International WestEd Pittsburgh Science of Learning Center Learning Sciences Mooc Research Initiative Open Learning Initiative Distance Learning Athabasca, Western Governor’s University Economics of Education Habit & Behavior Change Medical Education Centers/Scholarship of Teaching & Learning Assessment Pearson ETS Open Education Resources Creative Commons Foundations: Gates, Spencer, Hewlett, Kresge Venture Capital Private VCs New Schools Impact Investing Accelerators: Imagine K-12, HCI, Design Thinking Workforce Development 17
    18. 18. Relevant Resources • www.josephjaywilliams.com/home/useful- software • www.josephjaywilliams.com/education • www.josephjaywilliams.com/online-education- experiments • tiny.cc/mapofspace • Applying Cognitive Science to Online Learning. Paper presented at NIPS Workshop on Data Driven Education (tiny.cc/cognitivescienceinmoocs) • Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science. 18
    19. 19. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 19
    20. 20. Messages in Coursera? 20 Week 6 Quiz Whether or not you’re correct on the first try, each attempt at a tough problem can make you better at statistics.
    21. 21. Motivational Introductions 21 Practice-as-usual Have fun in the course! Anne Friedlander Growth MindsetText version of Mindset MessageMindset Message – Another InstructorPositive EncouragementBelonging
    22. 22. Instructor Guide to foster Growth Mindset • tiny.cc/idstrategymindset 22 Where do you think you will apply this? Do you have any questions or want clarification? Click to see any of the Case Studies below Would you like to?
    23. 23. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 23
    24. 24. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 24
    25. 25. II. Learning through Questions/Assessments • Discovery of patterns (Williams & Lombrozo, 2010, Cognitive Science) • Use of prior knowledge (Williams & Lombrozo, 2013, Cognitive Psychology) • Children’s causal learning (Walker, Williams, Lombrozo, Gopnik, 2012) • Overgeneralize at expense of specific facts (Williams et al, 2013, Journal of Experimental Psychology: General) • Contradictions & belief revision (Williams, et al 2013; 2014; Cog Sci, CHI WiP) • Comparison of examples (Edwards, Williams, Lombrozo, Gentner, 2013) GLORP DRENT Online/Blended Educational Resources
    26. 26. Adding questions/activities • tiny.cc/augmentedvideo • tiny.cc/augmentedexercise 26
    27. 27. III. General Learning & Problem-Solving Skills 27
    28. 28. Learning/Self-Questioning Strategies • Specific Prompt VS. • General Strategy of Self-Questioning? • Teach strategies (McNamara, 2004, Discourse Processes; Palinscar & Brown, 1984, Cognition and instruction; Williams & Lombrozo, 2010, Cognitive Science) • Habit & Behavior Change • Theory & Technology: Embed & Remind • Methodology: “longitudinal”, large N 28e.g. Weinstein & Mayer, 1986 Rothman, Baldwin, & Hertel, 2004
    29. 29. Embedded “On-Request” Strategy Instruction Clickable link. [Click here to learn about the “What? Why? How?” strategy] 29
    30. 30. Self-questioning strategy: What? Why? How? 30
    31. 31. Cues – Reflect & Remind 31
    32. 32. Preliminary Results 32N = ~ 800 per condition • Prompted Problem-solving better than Practice-as-usual. • Increased Accuracy on intervention and post-intervention problems.
    33. 33. Learning Assistant • tiny.cc/learningassistant 33
    34. 34. Learning Assistant – e.g. 1 34
    35. 35. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 35
    36. 36. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 36
    37. 37. IV. Performance Support Tools & Checklists 37
    38. 38. Coach/Support Learning Behaviors tiny.cc/kalearningcoach 38
    39. 39. A Lesson: Concept + Digital Tool to apply Concept in Future 39
    40. 40. Tool for Difficult Conversations 40
    41. 41. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 41
    42. 42. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 42
    43. 43. Acknowledgements • Tania Lombrozo & Tom Griffiths • John Mitchell & Candace Thille • Jascha Sohl-Dickstein • Jace Kohlmeier & Khan Academy • Sam Maldonado • Lytics Lab (lytics.stanford.edu) • VPOL (Vice Provost for Online Learning, online.stanford.edu) 43

    ×