SlideShare a Scribd company logo
1 of 43
Download to read offline
Learning Engineering of MOOClets: Simultaneously
benefiting Professional Learning, Financial Success,
and Cognitive & Learning Sciences Research
Joseph Jay Williams
Lytics Lab, Graduate School of Education
Office of the Vice Provost for Online Learning, Stanford
josephjaywilliams@stanford.edu
www.josephjaywilliams.com
1
[Originally from Trinidad & Tobago – apologies for difficulty in following accent.]
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
2
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
3
I. Motivation
• Beliefs about Intelligence
• To what extent do you agree?
How smart you are can be significantly
changed by your effort and hard work.
You have a certain amount of intelligence,
and what you do doesn’t really change
that much.
• Growth Mindset vs. Fixed Mindset
• Dweck, 2008; Yeager & Walton, 2012
• Williams, Paunesku, Haley & Sohl-Dickstein, 2013
4
MOOClet 1: Exercise (Khan Academy)
5
1. Number of
Problems
Practiced
2. Accuracy
[TEXT SHOWN HERE – Randomize]
Embed motivational messages
Practice-as-usual
Remember, the more you practice the smarter you become!
Growth Mindset Message
6
Growth Mindset Message helpful?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "Mistakes help you learn. Think hard
to learn from them.”
• Practice-as-usual
Jascha Sohl-Dickstein
7
Results: More motivated?
• Growth Mindset Message > Practice-as-
Usual
• reduction in dropout rate
• increase in Number of problems
practiced
• increase in Number Correct
• Accuracy: Problems correct/Problems
attempted
• Accuracy improvement
8
Interpreting results
• Obvious?
• Skeptical? Anything’s statistically
significant with large N…
• Second experiment (run simultaneously)
9
Practice-as-usualGrowth Mindset MessagePositive Message
Different motivational messages
Some of these problems are hard. Do your
best!
Remember, the more you practice the smarter you become!
10
Does any positive message work?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "Mistakes help you learn. Think hard
to learn from them.”
• Positive Message
• "Some of these problems are
hard. Just do your best."
• "This might be a tough problem,
but we know you can do it.”
• Practice-as-usual
11
Effects of Positive Messages?
• Positive Messages not better than Practice-as-
Usual!
• Growth Mindset > Positive Message
• How intuitive is this result?
• Teachers?
• Ongoing: After incorrect answer
• www.assistments.org Neil Heffernan
• Gates Grant for Modeling Research
12Rapid Experimentation: tiny.cc/mechanicalturk
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
13
Constraint Satisfaction
Practical Product
Research
Financial Value 14
“MOOClets” or Modular Online Resources
• E.g. MOOClet: Math exercise, 5 minute
interactive video, 30 minute lesson, Study
Strategy coach, Quiz/Assessment
• www.josephjaywilliams.com/mooclets
• MOOClets Group Tue 12-1 PST (remote)
• tiny.cc/joinmooclets mailing list & updates
• Williams, J.J. & Williams, B. A. (2013). Randomized Experiments
as a Methodological and Conceptual Tool for Improving
Design of Online Learning. Data Driven Education, NIPS
Workshop. tiny.cc/experimentsonlinelearning
15
Integrate Diverse Expertise & Innovations
16
Where is relevant research, best practice, technology, funding,
market? tiny.cc/mapofspace, tiny.cc/stanfordmap
MOOCs
(Coursera, EdX, Udacity)
www.KhanAcademy.org
www.Assistments.org
K-12 Ed-Tech space
(Bay Area, Boston...)
(www.edsurge.com)
(www.edudemic.com )
Higher-Ed Technology/LMS
E-Learning & Training for
Corporate, State, Federal, Army
Psychological Scientists
Computer Scientists
Education
Corporate Training
Funding Agencies:
IES, Dept. of Education, Dept. of Labor,
NIH, NSF
Practical Reports:
What Works Clearinghouse
IES Practice Guides
University Teaching & Learning Centers
Department of Education reports
Think Tanks:
SRI International
WestEd
Pittsburgh Science of Learning Center
Learning Sciences
Mooc Research Initiative
Open Learning Initiative
Distance Learning
Athabasca, Western Governor’s University
Economics of Education
Habit & Behavior Change
Medical Education
Centers/Scholarship of
Teaching & Learning
Assessment
Pearson
ETS
Open Education Resources
Creative Commons
Foundations:
Gates, Spencer, Hewlett, Kresge
Venture Capital
Private VCs
New Schools
Impact Investing
Accelerators:
Imagine K-12,
HCI, Design Thinking
Workforce Development
17
Relevant Resources
• www.josephjaywilliams.com/home/useful-
software
• www.josephjaywilliams.com/education
• www.josephjaywilliams.com/online-education-
experiments
• tiny.cc/mapofspace
• Applying Cognitive Science to Online Learning.
Paper presented at NIPS Workshop on Data
Driven Education
(tiny.cc/cognitivescienceinmoocs)
• Improving Outcome of Psychosocial Treatments
by Enhancing Memory and Learning.
Perspectives in Psychological Science.
18
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
19
Messages in Coursera?
20
Week 6 Quiz
Whether or not you’re correct on the first try, each attempt at a tough problem can
make you better at statistics.
Motivational Introductions
21
Practice-as-usual
Have fun in the course!
Anne Friedlander Growth MindsetText version of Mindset MessageMindset Message – Another InstructorPositive EncouragementBelonging
Instructor Guide to foster Growth Mindset
• tiny.cc/idstrategymindset
22
Where do you think you will apply this?
Do you have any questions or want
clarification?
Click to see any of the Case Studies below
Would you like to?
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
23
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
24
II. Learning through Questions/Assessments
• Discovery of patterns
(Williams & Lombrozo, 2010, Cognitive Science)
• Use of prior knowledge
(Williams & Lombrozo, 2013, Cognitive Psychology)
• Children’s causal learning
(Walker, Williams, Lombrozo, Gopnik, 2012)
• Overgeneralize at expense of specific facts
(Williams et al, 2013, Journal of Experimental Psychology: General)
• Contradictions & belief revision
(Williams, et al 2013; 2014; Cog Sci, CHI WiP)
• Comparison of examples
(Edwards, Williams, Lombrozo, Gentner, 2013)
GLORP
DRENT
Online/Blended
Educational
Resources
Adding questions/activities
• tiny.cc/augmentedvideo
• tiny.cc/augmentedexercise
26
III. General Learning & Problem-Solving Skills
27
Learning/Self-Questioning Strategies
• Specific Prompt VS.
• General Strategy of Self-Questioning?
• Teach strategies
(McNamara, 2004, Discourse Processes; Palinscar & Brown, 1984, Cognition and instruction;
Williams & Lombrozo, 2010, Cognitive Science)
• Habit & Behavior Change
• Theory & Technology: Embed & Remind
• Methodology: “longitudinal”, large N
28e.g. Weinstein & Mayer, 1986 Rothman, Baldwin, & Hertel, 2004
Embedded “On-Request” Strategy Instruction
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]
29
Self-questioning strategy: What? Why? How?
30
Cues – Reflect & Remind
31
Preliminary Results
32N = ~ 800 per condition
• Prompted Problem-solving better than
Practice-as-usual.
• Increased Accuracy on intervention
and post-intervention problems.
Learning Assistant
• tiny.cc/learningassistant
33
Learning Assistant – e.g. 1
34
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
35
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
36
IV. Performance Support Tools & Checklists
37
Coach/Support Learning Behaviors
tiny.cc/kalearningcoach
38
A Lesson: Concept + Digital Tool to apply
Concept in Future
39
Tool for Difficult Conversations
40
Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
41
Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
42
Acknowledgements
• Tania Lombrozo & Tom Griffiths
• John Mitchell & Candace Thille
• Jascha Sohl-Dickstein
• Jace Kohlmeier & Khan Academy
• Sam Maldonado
• Lytics Lab (lytics.stanford.edu)
• VPOL (Vice Provost for Online
Learning, online.stanford.edu)
43

More Related Content

What's hot

The Use of Serious Games in the Corporate Sector
The Use of Serious Games in the Corporate SectorThe Use of Serious Games in the Corporate Sector
The Use of Serious Games in the Corporate SectorLearnovate Centre
 
Bryk may 2014 using NICs to tackle practical problems in education
Bryk may 2014 using NICs to tackle practical problems in educationBryk may 2014 using NICs to tackle practical problems in education
Bryk may 2014 using NICs to tackle practical problems in educationlearningemerg
 
There are no bad ideas: 7 steps to ideation
There are no bad ideas: 7 steps to ideationThere are no bad ideas: 7 steps to ideation
There are no bad ideas: 7 steps to ideationHarish
 
5 ie 2015 collaboration cafe
5    ie 2015 collaboration cafe5    ie 2015 collaboration cafe
5 ie 2015 collaboration cafeMaggie Nichols
 
Fostering innovation to achieve mission critical goals
Fostering innovation to achieve mission critical goalsFostering innovation to achieve mission critical goals
Fostering innovation to achieve mission critical goalsLinkedIn Learning Solutions
 
Fetc 2022 10 Questions Workshop
Fetc 2022 10 Questions WorkshopFetc 2022 10 Questions Workshop
Fetc 2022 10 Questions WorkshopJulie Evans
 
Fetc 2022 Leadership Workshop
Fetc 2022 Leadership WorkshopFetc 2022 Leadership Workshop
Fetc 2022 Leadership WorkshopJulie Evans
 
2017 massey ite_lecture
2017 massey ite_lecture2017 massey ite_lecture
2017 massey ite_lectureDavid Kinane
 
Fetc 2022 Building Teacher Capacity
Fetc 2022 Building Teacher CapacityFetc 2022 Building Teacher Capacity
Fetc 2022 Building Teacher CapacityJulie Evans
 
2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...
2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...
2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...Daniela Gachago
 
To the man with a hammer, EVERYTHING looks like a nail
To the man with a hammer, EVERYTHING looks like a nailTo the man with a hammer, EVERYTHING looks like a nail
To the man with a hammer, EVERYTHING looks like a nailSascha Cohen
 
Testing Conventional Wisdom
Testing Conventional Wisdom Testing Conventional Wisdom
Testing Conventional Wisdom braaksma
 
How do we set standards and assess 21st century skills
How do we set standards and assess 21st century skillsHow do we set standards and assess 21st century skills
How do we set standards and assess 21st century skillsAggie14
 
New Models of Purpose-Driven Exploration in Knowledge Work
New Models of Purpose-Driven Exploration in Knowledge WorkNew Models of Purpose-Driven Exploration in Knowledge Work
New Models of Purpose-Driven Exploration in Knowledge WorkWilliam Evans
 
LearningCafe Year End Omnibus Webinar 2018
LearningCafe Year End Omnibus Webinar 2018LearningCafe Year End Omnibus Webinar 2018
LearningCafe Year End Omnibus Webinar 2018LearningCafe
 
Lecture 6 Digital Literacies
Lecture 6 Digital LiteraciesLecture 6 Digital Literacies
Lecture 6 Digital LiteraciesJames Stanfield
 
Not Your Father's e-Learning: New Paths to Learning in the Digital Age
Not Your Father's e-Learning: New Paths to Learning in the Digital AgeNot Your Father's e-Learning: New Paths to Learning in the Digital Age
Not Your Father's e-Learning: New Paths to Learning in the Digital AgeTom Gram
 
Expanding Education to Catapult the Successful Application of Creativity in E...
Expanding Education to Catapult the Successful Application of Creativity in E...Expanding Education to Catapult the Successful Application of Creativity in E...
Expanding Education to Catapult the Successful Application of Creativity in E...Courtney Huntzinger
 

What's hot (19)

The Use of Serious Games in the Corporate Sector
The Use of Serious Games in the Corporate SectorThe Use of Serious Games in the Corporate Sector
The Use of Serious Games in the Corporate Sector
 
Bryk may 2014 using NICs to tackle practical problems in education
Bryk may 2014 using NICs to tackle practical problems in educationBryk may 2014 using NICs to tackle practical problems in education
Bryk may 2014 using NICs to tackle practical problems in education
 
There are no bad ideas: 7 steps to ideation
There are no bad ideas: 7 steps to ideationThere are no bad ideas: 7 steps to ideation
There are no bad ideas: 7 steps to ideation
 
5 ie 2015 collaboration cafe
5    ie 2015 collaboration cafe5    ie 2015 collaboration cafe
5 ie 2015 collaboration cafe
 
Fostering innovation to achieve mission critical goals
Fostering innovation to achieve mission critical goalsFostering innovation to achieve mission critical goals
Fostering innovation to achieve mission critical goals
 
Fetc 2022 10 Questions Workshop
Fetc 2022 10 Questions WorkshopFetc 2022 10 Questions Workshop
Fetc 2022 10 Questions Workshop
 
Fetc 2022 Leadership Workshop
Fetc 2022 Leadership WorkshopFetc 2022 Leadership Workshop
Fetc 2022 Leadership Workshop
 
2017 massey ite_lecture
2017 massey ite_lecture2017 massey ite_lecture
2017 massey ite_lecture
 
Fetc 2022 Building Teacher Capacity
Fetc 2022 Building Teacher CapacityFetc 2022 Building Teacher Capacity
Fetc 2022 Building Teacher Capacity
 
2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...
2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...
2018 07 06 Designing for Design Thinking: Fostering an e-Learning Champion Mi...
 
To the man with a hammer, EVERYTHING looks like a nail
To the man with a hammer, EVERYTHING looks like a nailTo the man with a hammer, EVERYTHING looks like a nail
To the man with a hammer, EVERYTHING looks like a nail
 
Testing Conventional Wisdom
Testing Conventional Wisdom Testing Conventional Wisdom
Testing Conventional Wisdom
 
How do we set standards and assess 21st century skills
How do we set standards and assess 21st century skillsHow do we set standards and assess 21st century skills
How do we set standards and assess 21st century skills
 
New Models of Purpose-Driven Exploration in Knowledge Work
New Models of Purpose-Driven Exploration in Knowledge WorkNew Models of Purpose-Driven Exploration in Knowledge Work
New Models of Purpose-Driven Exploration in Knowledge Work
 
LearningCafe Year End Omnibus Webinar 2018
LearningCafe Year End Omnibus Webinar 2018LearningCafe Year End Omnibus Webinar 2018
LearningCafe Year End Omnibus Webinar 2018
 
Lecture 6 Digital Literacies
Lecture 6 Digital LiteraciesLecture 6 Digital Literacies
Lecture 6 Digital Literacies
 
Not Your Father's e-Learning: New Paths to Learning in the Digital Age
Not Your Father's e-Learning: New Paths to Learning in the Digital AgeNot Your Father's e-Learning: New Paths to Learning in the Digital Age
Not Your Father's e-Learning: New Paths to Learning in the Digital Age
 
Gamification learning
Gamification learningGamification learning
Gamification learning
 
Expanding Education to Catapult the Successful Application of Creativity in E...
Expanding Education to Catapult the Successful Application of Creativity in E...Expanding Education to Catapult the Successful Application of Creativity in E...
Expanding Education to Catapult the Successful Application of Creativity in E...
 

Viewers also liked

Fruit veg food
Fruit veg foodFruit veg food
Fruit veg foodihsan
 
Intro to Dungeons for World of Warcraft
Intro to Dungeons for World of WarcraftIntro to Dungeons for World of Warcraft
Intro to Dungeons for World of Warcraftcenter4edupunx
 
Games and assessment week 6
Games and assessment week 6Games and assessment week 6
Games and assessment week 6center4edupunx
 
Fall Games MOOC Week 4
Fall Games MOOC Week 4Fall Games MOOC Week 4
Fall Games MOOC Week 4center4edupunx
 

Viewers also liked (6)

2
22
2
 
Fruit veg food
Fruit veg foodFruit veg food
Fruit veg food
 
Intro to Dungeons for World of Warcraft
Intro to Dungeons for World of WarcraftIntro to Dungeons for World of Warcraft
Intro to Dungeons for World of Warcraft
 
Games and assessment week 6
Games and assessment week 6Games and assessment week 6
Games and assessment week 6
 
WoW addon essentials
WoW addon essentialsWoW addon essentials
WoW addon essentials
 
Fall Games MOOC Week 4
Fall Games MOOC Week 4Fall Games MOOC Week 4
Fall Games MOOC Week 4
 

Similar to Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research

Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Joseph Jay Williams
 
Research Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive ScienceResearch Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive ScienceiNACOL
 
How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...Joseph Jay Williams
 
UC Berkeley Leadership for Educational Equity Program (LEEP)
UC Berkeley Leadership for Educational Equity Program (LEEP)UC Berkeley Leadership for Educational Equity Program (LEEP)
UC Berkeley Leadership for Educational Equity Program (LEEP)CPEDInitiative
 
Leuphana Conference on Entrepreneurship 2015
Leuphana Conference on Entrepreneurship 2015Leuphana Conference on Entrepreneurship 2015
Leuphana Conference on Entrepreneurship 2015Norris Krueger
 
Design thinking STLinSTL 2016
Design thinking STLinSTL 2016Design thinking STLinSTL 2016
Design thinking STLinSTL 2016lmittler
 
Entrepreneurship education: How would we know if we're moving the needle?
Entrepreneurship education: How would we know if we're moving the needle?Entrepreneurship education: How would we know if we're moving the needle?
Entrepreneurship education: How would we know if we're moving the needle?Norris Krueger
 
Design Thinking Workshop STLinSTL
Design Thinking Workshop STLinSTLDesign Thinking Workshop STLinSTL
Design Thinking Workshop STLinSTLlmittler
 
Design Thinking
Design ThinkingDesign Thinking
Design Thinkinglmittler
 
10 take aways from learn now
10 take aways from learn now10 take aways from learn now
10 take aways from learn nowCrystal Taggart
 
Monaghan DET-CHE 2013
Monaghan DET-CHE 2013Monaghan DET-CHE 2013
Monaghan DET-CHE 2013Jim Monaghan
 
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Jay Williams
 
Low fidelity prototype v1.1 tech ent oct 2013
Low fidelity prototype v1.1 tech ent oct 2013Low fidelity prototype v1.1 tech ent oct 2013
Low fidelity prototype v1.1 tech ent oct 2013Marc-André de Launière
 
Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8Julie Coiro
 
Informal Learning: Broadening the Spectrum of Corporate Learning
Informal Learning: Broadening the Spectrum of Corporate LearningInformal Learning: Broadening the Spectrum of Corporate Learning
Informal Learning: Broadening the Spectrum of Corporate LearningHans de Zwart
 
Empowering self-directed learners: Practical strategies and tools for L&D
Empowering self-directed learners: Practical strategies and tools for L&DEmpowering self-directed learners: Practical strategies and tools for L&D
Empowering self-directed learners: Practical strategies and tools for L&DBrightwave Group
 
Final examining research on learning and its implications for
Final examining research on learning and its implications forFinal examining research on learning and its implications for
Final examining research on learning and its implications forehelfant
 
AISA Leadership Retreat Ghana - Leading complex change 2013
AISA Leadership Retreat Ghana - Leading complex change 2013AISA Leadership Retreat Ghana - Leading complex change 2013
AISA Leadership Retreat Ghana - Leading complex change 2013Chris Jansen
 
2019 New Trends in Education -Teaching Innovation
2019 New Trends in Education -Teaching Innovation 2019 New Trends in Education -Teaching Innovation
2019 New Trends in Education -Teaching Innovation Timothy Wooi
 

Similar to Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research (20)

Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...
 
Research Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive ScienceResearch Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive Science
 
How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...
 
UC Berkeley Leadership for Educational Equity Program (LEEP)
UC Berkeley Leadership for Educational Equity Program (LEEP)UC Berkeley Leadership for Educational Equity Program (LEEP)
UC Berkeley Leadership for Educational Equity Program (LEEP)
 
PBL: Why, What and How
PBL: Why, What and HowPBL: Why, What and How
PBL: Why, What and How
 
Leuphana Conference on Entrepreneurship 2015
Leuphana Conference on Entrepreneurship 2015Leuphana Conference on Entrepreneurship 2015
Leuphana Conference on Entrepreneurship 2015
 
Design thinking STLinSTL 2016
Design thinking STLinSTL 2016Design thinking STLinSTL 2016
Design thinking STLinSTL 2016
 
Entrepreneurship education: How would we know if we're moving the needle?
Entrepreneurship education: How would we know if we're moving the needle?Entrepreneurship education: How would we know if we're moving the needle?
Entrepreneurship education: How would we know if we're moving the needle?
 
Design Thinking Workshop STLinSTL
Design Thinking Workshop STLinSTLDesign Thinking Workshop STLinSTL
Design Thinking Workshop STLinSTL
 
Design Thinking
Design ThinkingDesign Thinking
Design Thinking
 
10 take aways from learn now
10 take aways from learn now10 take aways from learn now
10 take aways from learn now
 
Monaghan DET-CHE 2013
Monaghan DET-CHE 2013Monaghan DET-CHE 2013
Monaghan DET-CHE 2013
 
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
 
Low fidelity prototype v1.1 tech ent oct 2013
Low fidelity prototype v1.1 tech ent oct 2013Low fidelity prototype v1.1 tech ent oct 2013
Low fidelity prototype v1.1 tech ent oct 2013
 
Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8
 
Informal Learning: Broadening the Spectrum of Corporate Learning
Informal Learning: Broadening the Spectrum of Corporate LearningInformal Learning: Broadening the Spectrum of Corporate Learning
Informal Learning: Broadening the Spectrum of Corporate Learning
 
Empowering self-directed learners: Practical strategies and tools for L&D
Empowering self-directed learners: Practical strategies and tools for L&DEmpowering self-directed learners: Practical strategies and tools for L&D
Empowering self-directed learners: Practical strategies and tools for L&D
 
Final examining research on learning and its implications for
Final examining research on learning and its implications forFinal examining research on learning and its implications for
Final examining research on learning and its implications for
 
AISA Leadership Retreat Ghana - Leading complex change 2013
AISA Leadership Retreat Ghana - Leading complex change 2013AISA Leadership Retreat Ghana - Leading complex change 2013
AISA Leadership Retreat Ghana - Leading complex change 2013
 
2019 New Trends in Education -Teaching Innovation
2019 New Trends in Education -Teaching Innovation 2019 New Trends in Education -Teaching Innovation
2019 New Trends in Education -Teaching Innovation
 

More from Joseph Jay Williams

CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...Joseph Jay Williams
 
Experiments in Educational Research and Practice
Experiments in Educational Research and PracticeExperiments in Educational Research and Practice
Experiments in Educational Research and PracticeJoseph Jay Williams
 
Learning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractJoseph Jay Williams
 
Experiments in Educational Research & Practice
Experiments in Educational Research & PracticeExperiments in Educational Research & Practice
Experiments in Educational Research & PracticeJoseph Jay Williams
 
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Joseph Jay Williams
 
Doing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueDoing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueJoseph Jay Williams
 
How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?Joseph Jay Williams
 
Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Joseph Jay Williams
 

More from Joseph Jay Williams (9)

CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
 
Experiments in Educational Research and Practice
Experiments in Educational Research and PracticeExperiments in Educational Research and Practice
Experiments in Educational Research and Practice
 
Learning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstract
 
Experiments in Educational Research & Practice
Experiments in Educational Research & PracticeExperiments in Educational Research & Practice
Experiments in Educational Research & Practice
 
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
 
Doing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueDoing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial value
 
How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?
 
Williams lombrozo2010
Williams lombrozo2010Williams lombrozo2010
Williams lombrozo2010
 
Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)
 

Recently uploaded

Indexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfIndexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfChristalin Nelson
 
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxBIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxSayali Powar
 
Shark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristicsShark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristicsArubSultan
 
Objectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptxObjectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptxMadhavi Dharankar
 
Narcotic and Non Narcotic Analgesic..pdf
Narcotic and Non Narcotic Analgesic..pdfNarcotic and Non Narcotic Analgesic..pdf
Narcotic and Non Narcotic Analgesic..pdfPrerana Jadhav
 
Q-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITWQ-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITWQuiz Club NITW
 
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...Nguyen Thanh Tu Collection
 
Team Lead Succeed – Helping you and your team achieve high-performance teamwo...
Team Lead Succeed – Helping you and your team achieve high-performance teamwo...Team Lead Succeed – Helping you and your team achieve high-performance teamwo...
Team Lead Succeed – Helping you and your team achieve high-performance teamwo...Association for Project Management
 
Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...
Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...
Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...DhatriParmar
 
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...HetalPathak10
 
4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptxmary850239
 
Scientific Writing :Research Discourse
Scientific  Writing :Research  DiscourseScientific  Writing :Research  Discourse
Scientific Writing :Research DiscourseAnita GoswamiGiri
 
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Osopher
 
Congestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentationCongestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentationdeepaannamalai16
 
How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17Celine George
 

Recently uploaded (20)

Indexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfIndexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdf
 
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxBIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
 
Shark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristicsShark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristics
 
Objectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptxObjectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptx
 
Narcotic and Non Narcotic Analgesic..pdf
Narcotic and Non Narcotic Analgesic..pdfNarcotic and Non Narcotic Analgesic..pdf
Narcotic and Non Narcotic Analgesic..pdf
 
Q-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITWQ-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITW
 
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
 
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
 
Paradigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTAParadigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTA
 
Team Lead Succeed – Helping you and your team achieve high-performance teamwo...
Team Lead Succeed – Helping you and your team achieve high-performance teamwo...Team Lead Succeed – Helping you and your team achieve high-performance teamwo...
Team Lead Succeed – Helping you and your team achieve high-performance teamwo...
 
Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...
Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...
Blowin' in the Wind of Caste_ Bob Dylan's Song as a Catalyst for Social Justi...
 
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
Satirical Depths - A Study of Gabriel Okara's Poem - 'You Laughed and Laughed...
 
4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx
 
Faculty Profile prashantha K EEE dept Sri Sairam college of Engineering
Faculty Profile prashantha K EEE dept Sri Sairam college of EngineeringFaculty Profile prashantha K EEE dept Sri Sairam college of Engineering
Faculty Profile prashantha K EEE dept Sri Sairam college of Engineering
 
Scientific Writing :Research Discourse
Scientific  Writing :Research  DiscourseScientific  Writing :Research  Discourse
Scientific Writing :Research Discourse
 
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
 
Chi-Square Test Non Parametric Test Categorical Variable
Chi-Square Test Non Parametric Test Categorical VariableChi-Square Test Non Parametric Test Categorical Variable
Chi-Square Test Non Parametric Test Categorical Variable
 
CARNAVAL COM MAGIA E EUFORIA _
CARNAVAL COM MAGIA E EUFORIA            _CARNAVAL COM MAGIA E EUFORIA            _
CARNAVAL COM MAGIA E EUFORIA _
 
Congestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentationCongestive Cardiac Failure..presentation
Congestive Cardiac Failure..presentation
 
How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17
 

Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research

  • 1. Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research Joseph Jay Williams Lytics Lab, Graduate School of Education Office of the Vice Provost for Online Learning, Stanford josephjaywilliams@stanford.edu www.josephjaywilliams.com 1 [Originally from Trinidad & Tobago – apologies for difficulty in following accent.]
  • 2. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 2
  • 3. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 3
  • 4. I. Motivation • Beliefs about Intelligence • To what extent do you agree? How smart you are can be significantly changed by your effort and hard work. You have a certain amount of intelligence, and what you do doesn’t really change that much. • Growth Mindset vs. Fixed Mindset • Dweck, 2008; Yeager & Walton, 2012 • Williams, Paunesku, Haley & Sohl-Dickstein, 2013 4
  • 5. MOOClet 1: Exercise (Khan Academy) 5 1. Number of Problems Practiced 2. Accuracy [TEXT SHOWN HERE – Randomize]
  • 6. Embed motivational messages Practice-as-usual Remember, the more you practice the smarter you become! Growth Mindset Message 6
  • 7. Growth Mindset Message helpful? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Practice-as-usual Jascha Sohl-Dickstein 7
  • 8. Results: More motivated? • Growth Mindset Message > Practice-as- Usual • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy: Problems correct/Problems attempted • Accuracy improvement 8
  • 9. Interpreting results • Obvious? • Skeptical? Anything’s statistically significant with large N… • Second experiment (run simultaneously) 9
  • 10. Practice-as-usualGrowth Mindset MessagePositive Message Different motivational messages Some of these problems are hard. Do your best! Remember, the more you practice the smarter you become! 10
  • 11. Does any positive message work? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Positive Message • "Some of these problems are hard. Just do your best." • "This might be a tough problem, but we know you can do it.” • Practice-as-usual 11
  • 12. Effects of Positive Messages? • Positive Messages not better than Practice-as- Usual! • Growth Mindset > Positive Message • How intuitive is this result? • Teachers? • Ongoing: After incorrect answer • www.assistments.org Neil Heffernan • Gates Grant for Modeling Research 12Rapid Experimentation: tiny.cc/mechanicalturk
  • 13. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 13
  • 15. “MOOClets” or Modular Online Resources • E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment • www.josephjaywilliams.com/mooclets • MOOClets Group Tue 12-1 PST (remote) • tiny.cc/joinmooclets mailing list & updates • Williams, J.J. & Williams, B. A. (2013). Randomized Experiments as a Methodological and Conceptual Tool for Improving Design of Online Learning. Data Driven Education, NIPS Workshop. tiny.cc/experimentsonlinelearning 15
  • 16. Integrate Diverse Expertise & Innovations 16
  • 17. Where is relevant research, best practice, technology, funding, market? tiny.cc/mapofspace, tiny.cc/stanfordmap MOOCs (Coursera, EdX, Udacity) www.KhanAcademy.org www.Assistments.org K-12 Ed-Tech space (Bay Area, Boston...) (www.edsurge.com) (www.edudemic.com ) Higher-Ed Technology/LMS E-Learning & Training for Corporate, State, Federal, Army Psychological Scientists Computer Scientists Education Corporate Training Funding Agencies: IES, Dept. of Education, Dept. of Labor, NIH, NSF Practical Reports: What Works Clearinghouse IES Practice Guides University Teaching & Learning Centers Department of Education reports Think Tanks: SRI International WestEd Pittsburgh Science of Learning Center Learning Sciences Mooc Research Initiative Open Learning Initiative Distance Learning Athabasca, Western Governor’s University Economics of Education Habit & Behavior Change Medical Education Centers/Scholarship of Teaching & Learning Assessment Pearson ETS Open Education Resources Creative Commons Foundations: Gates, Spencer, Hewlett, Kresge Venture Capital Private VCs New Schools Impact Investing Accelerators: Imagine K-12, HCI, Design Thinking Workforce Development 17
  • 18. Relevant Resources • www.josephjaywilliams.com/home/useful- software • www.josephjaywilliams.com/education • www.josephjaywilliams.com/online-education- experiments • tiny.cc/mapofspace • Applying Cognitive Science to Online Learning. Paper presented at NIPS Workshop on Data Driven Education (tiny.cc/cognitivescienceinmoocs) • Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science. 18
  • 19. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 19
  • 20. Messages in Coursera? 20 Week 6 Quiz Whether or not you’re correct on the first try, each attempt at a tough problem can make you better at statistics.
  • 21. Motivational Introductions 21 Practice-as-usual Have fun in the course! Anne Friedlander Growth MindsetText version of Mindset MessageMindset Message – Another InstructorPositive EncouragementBelonging
  • 22. Instructor Guide to foster Growth Mindset • tiny.cc/idstrategymindset 22 Where do you think you will apply this? Do you have any questions or want clarification? Click to see any of the Case Studies below Would you like to?
  • 23. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 23
  • 24. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 24
  • 25. II. Learning through Questions/Assessments • Discovery of patterns (Williams & Lombrozo, 2010, Cognitive Science) • Use of prior knowledge (Williams & Lombrozo, 2013, Cognitive Psychology) • Children’s causal learning (Walker, Williams, Lombrozo, Gopnik, 2012) • Overgeneralize at expense of specific facts (Williams et al, 2013, Journal of Experimental Psychology: General) • Contradictions & belief revision (Williams, et al 2013; 2014; Cog Sci, CHI WiP) • Comparison of examples (Edwards, Williams, Lombrozo, Gentner, 2013) GLORP DRENT Online/Blended Educational Resources
  • 27. III. General Learning & Problem-Solving Skills 27
  • 28. Learning/Self-Questioning Strategies • Specific Prompt VS. • General Strategy of Self-Questioning? • Teach strategies (McNamara, 2004, Discourse Processes; Palinscar & Brown, 1984, Cognition and instruction; Williams & Lombrozo, 2010, Cognitive Science) • Habit & Behavior Change • Theory & Technology: Embed & Remind • Methodology: “longitudinal”, large N 28e.g. Weinstein & Mayer, 1986 Rothman, Baldwin, & Hertel, 2004
  • 29. Embedded “On-Request” Strategy Instruction Clickable link. [Click here to learn about the “What? Why? How?” strategy] 29
  • 31. Cues – Reflect & Remind 31
  • 32. Preliminary Results 32N = ~ 800 per condition • Prompted Problem-solving better than Practice-as-usual. • Increased Accuracy on intervention and post-intervention problems.
  • 35. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 35
  • 36. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 36
  • 37. IV. Performance Support Tools & Checklists 37
  • 39. A Lesson: Concept + Digital Tool to apply Concept in Future 39
  • 40. Tool for Difficult Conversations 40
  • 41. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 41
  • 42. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 42
  • 43. Acknowledgements • Tania Lombrozo & Tom Griffiths • John Mitchell & Candace Thille • Jascha Sohl-Dickstein • Jace Kohlmeier & Khan Academy • Sam Maldonado • Lytics Lab (lytics.stanford.edu) • VPOL (Vice Provost for Online Learning, online.stanford.edu) 43

Editor's Notes

  1. Originally from Trinidad. Thank you for having me here, it’s a pleasure because of how this entire movement has impacted me. Two years ago was finishing my dissertation work in cognitive science – psychology experiments on learning, & computational psychology, or using Bayesian statistics & machine learning to model human thinking, but I had been doing work in the education school, and wanted to have a real-world impact, and actually build products. Seemed impossible to do both, but all of a sudden – online education exploded! Tremendously excited, started research in collaboration with Khan Academy, and have been trying to add value by bringing to bear scientific knowledge about learning in a way that’s actually practically useful, and linking researchers to these practical platforms, like writing grants with Khan Academy & Stanford Online. So I will present a subset of that work. But it will be different from a typical research talk, because I took such an amazing group as a chance to share explicitly on the approach I take to developing online educational materials, point to resources people here might also find useful, and emphasize some broad themes in online learning where I think real-world impact and science converge. Interested in discussion & feedback from you on these – you have thought about it a tremendous deal here at Coursera, and much more familiar with your and others’ platforms. Don’t hesitate to make suggestions or state different opinion, I’d like to learn more about your thinking on these issues. Start with practical or “product-first”. Not necessarily a typical focus for researchers, except Candace Thille, Ken Koedinger, ? 50% is primary research, which I’ll present. 50% synthesizing existing work, having broad view of technology, entrepreneurship. I like collecting these list, and think there’s a lot of value to be tapped. Take a business perspective – where is there unrealized value, because of lack of information? So I’m going to discuss ongoing work with Khan Academy & MOOCs and residential education. Atypical Trajectory, which means the talk has many moving pieces: Use digital online educational resources to do use-inspired basic research. Extremely tough to link research and practice, so the last few years I’ve taken positions that allowed me to invest a lot of time into having a broad view of the emerging trends – understanding technology and goals of the MOOC platforms, Khan Academy, hundreds of Startups in Bay Area, as well as e-learning in the corporate training & workforce development world. Also been trying to get a broad view of which research topics and themes are particularly well suited to these contexts, from cognitive psychology to educational topics, social & clinical, health behavior change. Methodologies from HCI, education policy.
  2. Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  3. o.32 0.27
  4. Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  5. Have an unusual background, so I thought it might help to provide some context. Approach: Use digital educational resources to do inspired basic research (e.g. math exercises), understand how cognitive processes like explaining can guide learning & how people’s beliefs influence motivation. (Pasteur’s Quadrant). Principles & Demonstrations for doing this kind of work laid by PSLC. ust to provide some context about Use digital online An interdisciplinary approach to linking research and practice,PhD training is a mixture of Experimental Psychology, Computational Psychology (using methods from machine learning & Bayesian statistics to model cognition),
  6. Exhausting… but exciting! Criticized for not using literature… but don’t have time to. Researchers haven’t looked at & thought enough about platforms, to synthesize the key ideas. Also rare to have the time to take a broad view.
  7. Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  8. Add textbox below first two questions for people to type into. Add some hyperlink looking things below “Click … case studies” with something like these fields: Encouraging students to see literary skills as malleable through class emails Improving persistence in Database Course programming assignments Enhancing receptiveness to peer feedback in management courses Under would you like to, same kind of thing, but with: Learn more about the research behind this concept Ask a question or share thoughts with the Teach-Net mailing list Request a future reminder for applying this guide
  9. Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  10. Not traditional, but ubiquitous A challenging choice… Take the next theoretical step in this line of research, with a new set of materials. On the other hand, this is venturing into new territory. PSLC & others been successful, they have full control. Also, since there’s little direct past research in many of these contexts, don’t even know yet what form the phenomenon takes. Start with a practical challenge first, seek to improve it by bringing to bear research. Establishing whether there is an effect, sets the stage for using this online context to probe its underlying mechanisms. Jump to the other side, use-inspired basic research.
  11. Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
  12. Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?