Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Learning Engineering of MOOClets: Simultaneously
benefiting Professional Learning, Financial Success,
and Cognitive & Lear...
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
I. Motivation
• Beliefs about Intelligence
• To what extent do you agree?
How smart you are can be significantly
changed b...
MOOClet 1: Exercise (Khan Academy)
5
1. Number of
Problems
Practiced
2. Accuracy
[TEXT SHOWN HERE – Randomize]
Embed motivational messages
Practice-as-usual
Remember, the more you practice the smarter you become!
Growth Mindset Messa...
Growth Mindset Message helpful?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "M...
Results: More motivated?
• Growth Mindset Message > Practice-as-
Usual
• reduction in dropout rate
• increase in Number of...
Interpreting results
• Obvious?
• Skeptical? Anything’s statistically
significant with large N…
• Second experiment (run s...
Practice-as-usualGrowth Mindset MessagePositive Message
Different motivational messages
Some of these problems are hard. D...
Does any positive message work?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "M...
Effects of Positive Messages?
• Positive Messages not better than Practice-as-
Usual!
• Growth Mindset > Positive Message
...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Constraint Satisfaction
Practical Product
Research
Financial Value 14
“MOOClets” or Modular Online Resources
• E.g. MOOClet: Math exercise, 5 minute
interactive video, 30 minute lesson, Study
...
Integrate Diverse Expertise & Innovations
16
Where is relevant research, best practice, technology, funding,
market? tiny.cc/mapofspace, tiny.cc/stanfordmap
MOOCs
(Cou...
Relevant Resources
• www.josephjaywilliams.com/home/useful-
software
• www.josephjaywilliams.com/education
• www.josephjay...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Messages in Coursera?
20
Week 6 Quiz
Whether or not you’re correct on the first try, each attempt at a tough problem can
m...
Motivational Introductions
21
Practice-as-usual
Have fun in the course!
Anne Friedlander Growth MindsetText version of Min...
Instructor Guide to foster Growth Mindset
• tiny.cc/idstrategymindset
22
Where do you think you will apply this?
Do you ha...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
II. Learning through Questions/Assessments
• Discovery of patterns
(Williams & Lombrozo, 2010, Cognitive Science)
• Use of...
Adding questions/activities
• tiny.cc/augmentedvideo
• tiny.cc/augmentedexercise
26
III. General Learning & Problem-Solving Skills
27
Learning/Self-Questioning Strategies
• Specific Prompt VS.
• General Strategy of Self-Questioning?
• Teach strategies
(McN...
Embedded “On-Request” Strategy Instruction
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]
29
Self-questioning strategy: What? Why? How?
30
Cues – Reflect & Remind
31
Preliminary Results
32N = ~ 800 per condition
• Prompted Problem-solving better than
Practice-as-usual.
• Increased Accura...
Learning Assistant
• tiny.cc/learningassistant
33
Learning Assistant – e.g. 1
34
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
IV. Performance Support Tools & Checklists
37
Coach/Support Learning Behaviors
tiny.cc/kalearningcoach
38
A Lesson: Concept + Digital Tool to apply
Concept in Future
39
Tool for Difficult Conversations
40
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV...
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• F...
Acknowledgements
• Tania Lombrozo & Tom Griffiths
• John Mitchell & Candace Thille
• Jascha Sohl-Dickstein
• Jace Kohlmeie...
Upcoming SlideShare
Loading in …5
×

Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research

487 views

Published on

Invited talk given at Declara

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research

  1. 1. Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research Joseph Jay Williams Lytics Lab, Graduate School of Education Office of the Vice Provost for Online Learning, Stanford josephjaywilliams@stanford.edu www.josephjaywilliams.com 1 [Originally from Trinidad & Tobago – apologies for difficulty in following accent.]
  2. 2. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 2
  3. 3. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 3
  4. 4. I. Motivation • Beliefs about Intelligence • To what extent do you agree? How smart you are can be significantly changed by your effort and hard work. You have a certain amount of intelligence, and what you do doesn’t really change that much. • Growth Mindset vs. Fixed Mindset • Dweck, 2008; Yeager & Walton, 2012 • Williams, Paunesku, Haley & Sohl-Dickstein, 2013 4
  5. 5. MOOClet 1: Exercise (Khan Academy) 5 1. Number of Problems Practiced 2. Accuracy [TEXT SHOWN HERE – Randomize]
  6. 6. Embed motivational messages Practice-as-usual Remember, the more you practice the smarter you become! Growth Mindset Message 6
  7. 7. Growth Mindset Message helpful? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Practice-as-usual Jascha Sohl-Dickstein 7
  8. 8. Results: More motivated? • Growth Mindset Message > Practice-as- Usual • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy: Problems correct/Problems attempted • Accuracy improvement 8
  9. 9. Interpreting results • Obvious? • Skeptical? Anything’s statistically significant with large N… • Second experiment (run simultaneously) 9
  10. 10. Practice-as-usualGrowth Mindset MessagePositive Message Different motivational messages Some of these problems are hard. Do your best! Remember, the more you practice the smarter you become! 10
  11. 11. Does any positive message work? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Positive Message • "Some of these problems are hard. Just do your best." • "This might be a tough problem, but we know you can do it.” • Practice-as-usual 11
  12. 12. Effects of Positive Messages? • Positive Messages not better than Practice-as- Usual! • Growth Mindset > Positive Message • How intuitive is this result? • Teachers? • Ongoing: After incorrect answer • www.assistments.org Neil Heffernan • Gates Grant for Modeling Research 12Rapid Experimentation: tiny.cc/mechanicalturk
  13. 13. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 13
  14. 14. Constraint Satisfaction Practical Product Research Financial Value 14
  15. 15. “MOOClets” or Modular Online Resources • E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment • www.josephjaywilliams.com/mooclets • MOOClets Group Tue 12-1 PST (remote) • tiny.cc/joinmooclets mailing list & updates • Williams, J.J. & Williams, B. A. (2013). Randomized Experiments as a Methodological and Conceptual Tool for Improving Design of Online Learning. Data Driven Education, NIPS Workshop. tiny.cc/experimentsonlinelearning 15
  16. 16. Integrate Diverse Expertise & Innovations 16
  17. 17. Where is relevant research, best practice, technology, funding, market? tiny.cc/mapofspace, tiny.cc/stanfordmap MOOCs (Coursera, EdX, Udacity) www.KhanAcademy.org www.Assistments.org K-12 Ed-Tech space (Bay Area, Boston...) (www.edsurge.com) (www.edudemic.com ) Higher-Ed Technology/LMS E-Learning & Training for Corporate, State, Federal, Army Psychological Scientists Computer Scientists Education Corporate Training Funding Agencies: IES, Dept. of Education, Dept. of Labor, NIH, NSF Practical Reports: What Works Clearinghouse IES Practice Guides University Teaching & Learning Centers Department of Education reports Think Tanks: SRI International WestEd Pittsburgh Science of Learning Center Learning Sciences Mooc Research Initiative Open Learning Initiative Distance Learning Athabasca, Western Governor’s University Economics of Education Habit & Behavior Change Medical Education Centers/Scholarship of Teaching & Learning Assessment Pearson ETS Open Education Resources Creative Commons Foundations: Gates, Spencer, Hewlett, Kresge Venture Capital Private VCs New Schools Impact Investing Accelerators: Imagine K-12, HCI, Design Thinking Workforce Development 17
  18. 18. Relevant Resources • www.josephjaywilliams.com/home/useful- software • www.josephjaywilliams.com/education • www.josephjaywilliams.com/online-education- experiments • tiny.cc/mapofspace • Applying Cognitive Science to Online Learning. Paper presented at NIPS Workshop on Data Driven Education (tiny.cc/cognitivescienceinmoocs) • Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science. 18
  19. 19. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 19
  20. 20. Messages in Coursera? 20 Week 6 Quiz Whether or not you’re correct on the first try, each attempt at a tough problem can make you better at statistics.
  21. 21. Motivational Introductions 21 Practice-as-usual Have fun in the course! Anne Friedlander Growth MindsetText version of Mindset MessageMindset Message – Another InstructorPositive EncouragementBelonging
  22. 22. Instructor Guide to foster Growth Mindset • tiny.cc/idstrategymindset 22 Where do you think you will apply this? Do you have any questions or want clarification? Click to see any of the Case Studies below Would you like to?
  23. 23. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 23
  24. 24. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 24
  25. 25. II. Learning through Questions/Assessments • Discovery of patterns (Williams & Lombrozo, 2010, Cognitive Science) • Use of prior knowledge (Williams & Lombrozo, 2013, Cognitive Psychology) • Children’s causal learning (Walker, Williams, Lombrozo, Gopnik, 2012) • Overgeneralize at expense of specific facts (Williams et al, 2013, Journal of Experimental Psychology: General) • Contradictions & belief revision (Williams, et al 2013; 2014; Cog Sci, CHI WiP) • Comparison of examples (Edwards, Williams, Lombrozo, Gentner, 2013) GLORP DRENT Online/Blended Educational Resources
  26. 26. Adding questions/activities • tiny.cc/augmentedvideo • tiny.cc/augmentedexercise 26
  27. 27. III. General Learning & Problem-Solving Skills 27
  28. 28. Learning/Self-Questioning Strategies • Specific Prompt VS. • General Strategy of Self-Questioning? • Teach strategies (McNamara, 2004, Discourse Processes; Palinscar & Brown, 1984, Cognition and instruction; Williams & Lombrozo, 2010, Cognitive Science) • Habit & Behavior Change • Theory & Technology: Embed & Remind • Methodology: “longitudinal”, large N 28e.g. Weinstein & Mayer, 1986 Rothman, Baldwin, & Hertel, 2004
  29. 29. Embedded “On-Request” Strategy Instruction Clickable link. [Click here to learn about the “What? Why? How?” strategy] 29
  30. 30. Self-questioning strategy: What? Why? How? 30
  31. 31. Cues – Reflect & Remind 31
  32. 32. Preliminary Results 32N = ~ 800 per condition • Prompted Problem-solving better than Practice-as-usual. • Increased Accuracy on intervention and post-intervention problems.
  33. 33. Learning Assistant • tiny.cc/learningassistant 33
  34. 34. Learning Assistant – e.g. 1 34
  35. 35. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 35
  36. 36. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 36
  37. 37. IV. Performance Support Tools & Checklists 37
  38. 38. Coach/Support Learning Behaviors tiny.cc/kalearningcoach 38
  39. 39. A Lesson: Concept + Digital Tool to apply Concept in Future 39
  40. 40. Tool for Difficult Conversations 40
  41. 41. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 41
  42. 42. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 42
  43. 43. Acknowledgements • Tania Lombrozo & Tom Griffiths • John Mitchell & Candace Thille • Jascha Sohl-Dickstein • Jace Kohlmeier & Khan Academy • Sam Maldonado • Lytics Lab (lytics.stanford.edu) • VPOL (Vice Provost for Online Learning, online.stanford.edu) 43

×