SlideShare a Scribd company logo

Experiments in Educational Research and Practice

Slides of a talk given at the Pittsburgh Science of Learning Center

1 of 19
Download to read offline
Experiments in Educational Research
& Practice
Joseph Jay Williams
josephjaywilliams@gmail.com
www.josephjaywilliams.com
Lytics Lab, Office of the Vice Provost of Online
Education, Graduate School of Education, Stanford
Experiments in Educational Research & Practice
•

Kinds of Experiments
–
–
–
–

•

Laboratory
Design experiment (Brown, 1992)
In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)
Randomized Controlled Trial/Intervention

Experiments & Internet, Online Education
–

Internet-facilitated Laboratory studies
•
•
•

–
–
–
–

•
•

Mechanical Turk (tiny.cc/mechanicalturk )
Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )
“A/B testing” more widespread

Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)
In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)
Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)
Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)

REPEAT Approach to Experimental Research on Online/Blended Resources
Identifying a Practical Online Context for your Research Question
–
–
–
–

Partner with platform *early* in research
“Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)
Examples of Blended Learning Resources & MOOCs
Examples of resources that support Randomized Trials

–
–

Interventions extended over time
Bridging studies

•

•

Engaging with Practitioners
–
–

How to communicate value of experiments
Consulting
•
•
•
•

–
–

•

Educational companies & publishers (marcy.baughman@pearson.com )
Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer)
Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange)
Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education

Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)
Virtual collaboration on practical resources (www.learnnetwork.net )

Experiments & …
–
–
–

•

Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,

Computer Supported Collaborative Learning (Rose)
Cognitive Tutors (Aleven, Ritter, Koedinger)
EDM & AIED (Stamper, Gordon, Pardos)
Conducting Experiments (tiny.cc/conductingexperiments)
Kinds of Experiments
–
–
–
–

Laboratory
Design experiment (Brown, 1992)
In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)
Randomized Controlled Trial/Intervention
Experiments & Internet, Online Education
– Online Educational Resources
– Internet-facilitated Laboratory studies
• Mechanical Turk (tiny.cc/mechanicalturk )
• Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )
• “A/B testing” more widespread

– Iterative Design & Improvement (Amy Collier, Stanford Office of Online
Learning)
– In vivo motivation experiment on Khan Academy (Williams et
al, 2013, MOOCshop)
– Conducting Randomized Trials over the Internet
(Paunesku, Greene, Williams)
– Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
Online Education

Send questions to tiny.cc/apsonlinelearning
A Novel Opportunity for Research
Ad

Recommended

Experiments in Educational Research & Practice
Experiments in Educational Research & PracticeExperiments in Educational Research & Practice
Experiments in Educational Research & PracticeJoseph Jay Williams
 
What is Learning Analytics?
What is Learning Analytics?What is Learning Analytics?
What is Learning Analytics?Doug Clow
 
Evidence Hub Activity
Evidence Hub ActivityEvidence Hub Activity
Evidence Hub ActivityDoug Clow
 
Big Data in Learning Analytics - Analytics for Everyday Learning
Big Data in Learning Analytics - Analytics for Everyday LearningBig Data in Learning Analytics - Analytics for Everyday Learning
Big Data in Learning Analytics - Analytics for Everyday LearningStefan Dietze
 
The OERu from the inside out and the outside in
The OERu from the inside out and the outside inThe OERu from the inside out and the outside in
The OERu from the inside out and the outside inwitthaus
 
Love for science or 'Academic Prostitution' - IAA version
Love for science or 'Academic Prostitution' - IAA versionLove for science or 'Academic Prostitution' - IAA version
Love for science or 'Academic Prostitution' - IAA versionLourdes Verdes-Montenegro
 
Anonymous peer review062016 shared
Anonymous peer review062016 sharedAnonymous peer review062016 shared
Anonymous peer review062016 sharedGjoa Andrichuk
 

More Related Content

What's hot

Nitle 2013 presentation
Nitle 2013 presentationNitle 2013 presentation
Nitle 2013 presentationNITLE
 
Early Career Reseachers and Open Healthcare
Early Career Reseachers and Open HealthcareEarly Career Reseachers and Open Healthcare
Early Career Reseachers and Open Healthcarepetermurrayrust
 
Data, Responsibly: The Next Decade of Data Science
Data, Responsibly: The Next Decade of Data ScienceData, Responsibly: The Next Decade of Data Science
Data, Responsibly: The Next Decade of Data ScienceUniversity of Washington
 
Measured Steps: An Exploratory Study in Mulitmodal Literacy and Learning
Measured  Steps: An Exploratory Study in Mulitmodal Literacy and LearningMeasured  Steps: An Exploratory Study in Mulitmodal Literacy and Learning
Measured Steps: An Exploratory Study in Mulitmodal Literacy and LearningWestern Illinois University
 
Altmetrics 2015 jan
Altmetrics 2015 janAltmetrics 2015 jan
Altmetrics 2015 janCMHSL
 
Altmetrics: The Movement, The Tools, and the Implications
Altmetrics: The Movement, The Tools, and the ImplicationsAltmetrics: The Movement, The Tools, and the Implications
Altmetrics: The Movement, The Tools, and the ImplicationsCMHSL
 
Open Science & Altmetrics
Open Science & AltmetricsOpen Science & Altmetrics
Open Science & AltmetricsIsabella Peters
 
The Reputation Economy
The Reputation EconomyThe Reputation Economy
The Reputation EconomyCMHSL
 

What's hot (10)

Nitle 2013 presentation
Nitle 2013 presentationNitle 2013 presentation
Nitle 2013 presentation
 
Early Career Reseachers and Open Healthcare
Early Career Reseachers and Open HealthcareEarly Career Reseachers and Open Healthcare
Early Career Reseachers and Open Healthcare
 
Data, Responsibly: The Next Decade of Data Science
Data, Responsibly: The Next Decade of Data ScienceData, Responsibly: The Next Decade of Data Science
Data, Responsibly: The Next Decade of Data Science
 
Measured Steps: An Exploratory Study in Mulitmodal Literacy and Learning
Measured  Steps: An Exploratory Study in Mulitmodal Literacy and LearningMeasured  Steps: An Exploratory Study in Mulitmodal Literacy and Learning
Measured Steps: An Exploratory Study in Mulitmodal Literacy and Learning
 
Altmetrics 2015 jan
Altmetrics 2015 janAltmetrics 2015 jan
Altmetrics 2015 jan
 
Science Data, Responsibly
Science Data, ResponsiblyScience Data, Responsibly
Science Data, Responsibly
 
Altmetrics: The Movement, The Tools, and the Implications
Altmetrics: The Movement, The Tools, and the ImplicationsAltmetrics: The Movement, The Tools, and the Implications
Altmetrics: The Movement, The Tools, and the Implications
 
1330 mon dochart2 brock
1330 mon dochart2 brock1330 mon dochart2 brock
1330 mon dochart2 brock
 
Open Science & Altmetrics
Open Science & AltmetricsOpen Science & Altmetrics
Open Science & Altmetrics
 
The Reputation Economy
The Reputation EconomyThe Reputation Economy
The Reputation Economy
 

Viewers also liked

Teacher Quality Power Point
Teacher Quality Power PointTeacher Quality Power Point
Teacher Quality Power PointWSU Cougars
 
Passport to class room English
Passport to class room   EnglishPassport to class room   English
Passport to class room EnglishImran Zakir
 
social problems faced by women
social problems faced by women social problems faced by women
social problems faced by women immanuel66
 
Teacher quality power point
Teacher quality power pointTeacher quality power point
Teacher quality power pointhpetrich
 
Quality Education
Quality EducationQuality Education
Quality EducationSaba Ali
 
The elements of teaching and learning
The elements of teaching and learningThe elements of teaching and learning
The elements of teaching and learningJanette Balagot
 
Teaching and Learning Process
Teaching and Learning ProcessTeaching and Learning Process
Teaching and Learning ProcessJaser Daher
 
HOW can we have Quality Education for All?
HOW can we have Quality Education for All?HOW can we have Quality Education for All?
HOW can we have Quality Education for All?Yogesh Upadhyaya
 
Education today ppt
Education today pptEducation today ppt
Education today pptAnwar Pasha
 
Principles of education and teaching learning process
Principles of education and teaching learning processPrinciples of education and teaching learning process
Principles of education and teaching learning processNursing Path
 

Viewers also liked (15)

Thinking Skills.pptx
Thinking Skills.pptxThinking Skills.pptx
Thinking Skills.pptx
 
Teacher Quality Power Point
Teacher Quality Power PointTeacher Quality Power Point
Teacher Quality Power Point
 
Passport to class room English
Passport to class room   EnglishPassport to class room   English
Passport to class room English
 
Quality Education
Quality EducationQuality Education
Quality Education
 
social problems faced by women
social problems faced by women social problems faced by women
social problems faced by women
 
Quality education
Quality educationQuality education
Quality education
 
Teacher quality power point
Teacher quality power pointTeacher quality power point
Teacher quality power point
 
Women working
Women workingWomen working
Women working
 
Quality Education
Quality EducationQuality Education
Quality Education
 
Presentation on effective teaching
Presentation on effective teachingPresentation on effective teaching
Presentation on effective teaching
 
The elements of teaching and learning
The elements of teaching and learningThe elements of teaching and learning
The elements of teaching and learning
 
Teaching and Learning Process
Teaching and Learning ProcessTeaching and Learning Process
Teaching and Learning Process
 
HOW can we have Quality Education for All?
HOW can we have Quality Education for All?HOW can we have Quality Education for All?
HOW can we have Quality Education for All?
 
Education today ppt
Education today pptEducation today ppt
Education today ppt
 
Principles of education and teaching learning process
Principles of education and teaching learning processPrinciples of education and teaching learning process
Principles of education and teaching learning process
 

Similar to Experiments in Educational Research and Practice

NGSS Assessment: Identifying and Integrating Effective Technologies
NGSS Assessment: Identifying and Integrating Effective TechnologiesNGSS Assessment: Identifying and Integrating Effective Technologies
NGSS Assessment: Identifying and Integrating Effective TechnologiesDermot Donnelly
 
How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...Joseph Jay Williams
 
Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Joseph Jay Williams
 
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...Simon Bates
 
Research Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive ScienceResearch Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive ScienceiNACOL
 
Big Data for Student Learning
Big Data for Student LearningBig Data for Student Learning
Big Data for Student LearningMarie Bienkowski
 
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous TransformationalKeynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformationalthe Hartsook Letter
 
Vijay Kumar Presentation - Sakai 09
Vijay Kumar Presentation - Sakai 09Vijay Kumar Presentation - Sakai 09
Vijay Kumar Presentation - Sakai 09mollywoggly
 
Don’t leave me alone: effectiveness of a framed wiki-based learning activity
Don’t leave me alone: effectiveness of a framed wiki-based learning activityDon’t leave me alone: effectiveness of a framed wiki-based learning activity
Don’t leave me alone: effectiveness of a framed wiki-based learning activityNikolaos Tselios
 
Educ 526 presentation
Educ 526 presentationEduc 526 presentation
Educ 526 presentationchervitz
 
Institute online inquiry 7 12
Institute online inquiry 7 12Institute online inquiry 7 12
Institute online inquiry 7 12jillcastek
 
Doha College Mobile Learning Conference 2014: Learning from Research
Doha College Mobile Learning Conference 2014: Learning from ResearchDoha College Mobile Learning Conference 2014: Learning from Research
Doha College Mobile Learning Conference 2014: Learning from ResearchKevin Burden
 
Transformingtechnologies session7
Transformingtechnologies session7Transformingtechnologies session7
Transformingtechnologies session7sarahattersley
 
Learning Analytics: Seeking new insights from educational data
Learning Analytics: Seeking new insights from educational dataLearning Analytics: Seeking new insights from educational data
Learning Analytics: Seeking new insights from educational dataAndrew Deacon
 
Computer Science Education: Tools and Data
Computer Science Education: Tools and DataComputer Science Education: Tools and Data
Computer Science Education: Tools and DataPeter Brusilovsky
 
Personalized Learning: Expanding the Social Impact of AI
Personalized Learning: Expanding the Social Impact of AIPersonalized Learning: Expanding the Social Impact of AI
Personalized Learning: Expanding the Social Impact of AIPeter Brusilovsky
 
Data driveneducationicwl2016
Data driveneducationicwl2016Data driveneducationicwl2016
Data driveneducationicwl2016Peter Brusilovsky
 
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...Barry Ryan
 

Similar to Experiments in Educational Research and Practice (20)

NGSS Assessment: Identifying and Integrating Effective Technologies
NGSS Assessment: Identifying and Integrating Effective TechnologiesNGSS Assessment: Identifying and Integrating Effective Technologies
NGSS Assessment: Identifying and Integrating Effective Technologies
 
How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...How online educational resources provide novel affordances for conducting pra...
How online educational resources provide novel affordances for conducting pra...
 
Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Using Experiments and Cognitive Science Research to Improve the Design of Onl...
Using Experiments and Cognitive Science Research to Improve the Design of Onl...
 
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...
 
Research Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive ScienceResearch Webinar: OERS and Cognitive Science
Research Webinar: OERS and Cognitive Science
 
Big Data for Student Learning
Big Data for Student LearningBig Data for Student Learning
Big Data for Student Learning
 
Berlin_presentation _30.09.2022.pptx
Berlin_presentation _30.09.2022.pptxBerlin_presentation _30.09.2022.pptx
Berlin_presentation _30.09.2022.pptx
 
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous TransformationalKeynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
 
Vijay Kumar Presentation - Sakai 09
Vijay Kumar Presentation - Sakai 09Vijay Kumar Presentation - Sakai 09
Vijay Kumar Presentation - Sakai 09
 
Don’t leave me alone: effectiveness of a framed wiki-based learning activity
Don’t leave me alone: effectiveness of a framed wiki-based learning activityDon’t leave me alone: effectiveness of a framed wiki-based learning activity
Don’t leave me alone: effectiveness of a framed wiki-based learning activity
 
Bcuenr310
Bcuenr310Bcuenr310
Bcuenr310
 
Educ 526 presentation
Educ 526 presentationEduc 526 presentation
Educ 526 presentation
 
Institute online inquiry 7 12
Institute online inquiry 7 12Institute online inquiry 7 12
Institute online inquiry 7 12
 
Doha College Mobile Learning Conference 2014: Learning from Research
Doha College Mobile Learning Conference 2014: Learning from ResearchDoha College Mobile Learning Conference 2014: Learning from Research
Doha College Mobile Learning Conference 2014: Learning from Research
 
Transformingtechnologies session7
Transformingtechnologies session7Transformingtechnologies session7
Transformingtechnologies session7
 
Learning Analytics: Seeking new insights from educational data
Learning Analytics: Seeking new insights from educational dataLearning Analytics: Seeking new insights from educational data
Learning Analytics: Seeking new insights from educational data
 
Computer Science Education: Tools and Data
Computer Science Education: Tools and DataComputer Science Education: Tools and Data
Computer Science Education: Tools and Data
 
Personalized Learning: Expanding the Social Impact of AI
Personalized Learning: Expanding the Social Impact of AIPersonalized Learning: Expanding the Social Impact of AI
Personalized Learning: Expanding the Social Impact of AI
 
Data driveneducationicwl2016
Data driveneducationicwl2016Data driveneducationicwl2016
Data driveneducationicwl2016
 
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
 

More from Joseph Jay Williams

CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...Joseph Jay Williams
 
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Joseph Jay Williams
 
Learning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractJoseph Jay Williams
 
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Joseph Jay Williams
 
Doing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueDoing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueJoseph Jay Williams
 
How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?Joseph Jay Williams
 
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Jay Williams
 
Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Joseph Jay Williams
 

More from Joseph Jay Williams (9)

CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
 
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
 
Learning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstract
 
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
 
Doing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueDoing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial value
 
How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?
 
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
 
Williams lombrozo2010
Williams lombrozo2010Williams lombrozo2010
Williams lombrozo2010
 
Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)
 

Recently uploaded

mean stack mean stack mean stack mean stack
mean stack mean stack  mean stack  mean stackmean stack mean stack  mean stack  mean stack
mean stack mean stack mean stack mean stackNuttavutThongjor1
 
UniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tourUniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tourUSC_Library
 
2.15.24 Making Whiteness -- Baldwin.pptx
2.15.24 Making Whiteness -- Baldwin.pptx2.15.24 Making Whiteness -- Baldwin.pptx
2.15.24 Making Whiteness -- Baldwin.pptxMaryPotorti1
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...Nguyen Thanh Tu Collection
 
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptxCONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptxAnupkumar Sharma
 
IR introduction Introduction, Principle & Theory
IR introduction Introduction, Principle & TheoryIR introduction Introduction, Principle & Theory
IR introduction Introduction, Principle & Theorynivedithag131
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaFaheema Hasan
 
The Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyThe Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyTrushali Dodiya
 
LIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATIONLIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATIONSayali Powar
 
Data Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdfData Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdfChristalin Nelson
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptxMaryPotorti1
 
ACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docxACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docxMaria Lucia Céspedes
 
Intuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov ChainsIntuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov ChainsTushar Tank
 
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapitolTechU
 
Practical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxPractical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxKatherine Villaluna
 
ICSE English Literature Class X Handwritten Notes
ICSE English Literature Class X Handwritten NotesICSE English Literature Class X Handwritten Notes
ICSE English Literature Class X Handwritten NotesGauri S
 
Appendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptxAppendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptxRenuka N Sunagad
 
Grades 7 to 8 Anti- OSAEC and CSAEM session.pptx
Grades 7 to 8 Anti- OSAEC and CSAEM session.pptxGrades 7 to 8 Anti- OSAEC and CSAEM session.pptx
Grades 7 to 8 Anti- OSAEC and CSAEM session.pptxGladysValencia13
 
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...Rabiya Husain
 

Recently uploaded (20)

mean stack mean stack mean stack mean stack
mean stack mean stack  mean stack  mean stackmean stack mean stack  mean stack  mean stack
mean stack mean stack mean stack mean stack
 
UniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tourUniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tour
 
2.15.24 Making Whiteness -- Baldwin.pptx
2.15.24 Making Whiteness -- Baldwin.pptx2.15.24 Making Whiteness -- Baldwin.pptx
2.15.24 Making Whiteness -- Baldwin.pptx
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
 
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptxCONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
 
IR introduction Introduction, Principle & Theory
IR introduction Introduction, Principle & TheoryIR introduction Introduction, Principle & Theory
IR introduction Introduction, Principle & Theory
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic Leukemia
 
The Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyThe Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati Roy
 
LIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATIONLIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATION
 
Data Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdfData Modeling - Entity Relationship Diagrams-1.pdf
Data Modeling - Entity Relationship Diagrams-1.pdf
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
 
ACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docxACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docx
 
Intuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov ChainsIntuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov Chains
 
Time-Honored Wisdom: African Teachings for VUCA Leaders
Time-Honored Wisdom: African Teachings for VUCA LeadersTime-Honored Wisdom: African Teachings for VUCA Leaders
Time-Honored Wisdom: African Teachings for VUCA Leaders
 
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
 
Practical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxPractical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptx
 
ICSE English Literature Class X Handwritten Notes
ICSE English Literature Class X Handwritten NotesICSE English Literature Class X Handwritten Notes
ICSE English Literature Class X Handwritten Notes
 
Appendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptxAppendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptx
 
Grades 7 to 8 Anti- OSAEC and CSAEM session.pptx
Grades 7 to 8 Anti- OSAEC and CSAEM session.pptxGrades 7 to 8 Anti- OSAEC and CSAEM session.pptx
Grades 7 to 8 Anti- OSAEC and CSAEM session.pptx
 
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
 

Experiments in Educational Research and Practice

  • 1. Experiments in Educational Research & Practice Joseph Jay Williams josephjaywilliams@gmail.com www.josephjaywilliams.com Lytics Lab, Office of the Vice Provost of Online Education, Graduate School of Education, Stanford
  • 2. Experiments in Educational Research & Practice • Kinds of Experiments – – – – • Laboratory Design experiment (Brown, 1992) In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010) Randomized Controlled Trial/Intervention Experiments & Internet, Online Education – Internet-facilitated Laboratory studies • • • – – – – • • Mechanical Turk (tiny.cc/mechanicalturk ) Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics ) “A/B testing” more widespread Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning) In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop) Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams) Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013) REPEAT Approach to Experimental Research on Online/Blended Resources Identifying a Practical Online Context for your Research Question – – – – Partner with platform *early* in research “Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010) Examples of Blended Learning Resources & MOOCs Examples of resources that support Randomized Trials – – Interventions extended over time Bridging studies • • Engaging with Practitioners – – How to communicate value of experiments Consulting • • • • – – • Educational companies & publishers (marcy.baughman@pearson.com ) Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer) Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange) Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education Grant funding (IES, NSF, Gates, Spencer, moocresearch.com) Virtual collaboration on practical resources (www.learnnetwork.net ) Experiments & … – – – • Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia, Computer Supported Collaborative Learning (Rose) Cognitive Tutors (Aleven, Ritter, Koedinger) EDM & AIED (Stamper, Gordon, Pardos) Conducting Experiments (tiny.cc/conductingexperiments)
  • 3. Kinds of Experiments – – – – Laboratory Design experiment (Brown, 1992) In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010) Randomized Controlled Trial/Intervention
  • 4. Experiments & Internet, Online Education – Online Educational Resources – Internet-facilitated Laboratory studies • Mechanical Turk (tiny.cc/mechanicalturk ) • Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics ) • “A/B testing” more widespread – Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning) – In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop) – Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams) – Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
  • 5. Online Education Send questions to tiny.cc/apsonlinelearning
  • 6. A Novel Opportunity for Research
  • 7. Integrate Research & Practice • Randomized assignment • Experimental Control • Rich data • Real-world environment • Authentic activities • Practical Challenges • Generalizable theories • “in vivo” experiments • Diverse populations • Practical improvements • Disseminate research • Generate Funding
  • 8. Experiments & Internet, Online Education – Internet-facilitated Laboratory studies • Mechanical Turk (tiny.cc/mechanicalturk ) • Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics ) • “A/B testing” more widespread – Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning) – In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop) – Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams) – Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
  • 9. In-vivo motivation experiment on Khan Academy Jascha Paunesku, • Williams,Sohl-Dickstein Haley, & SohlDickstein, MOOCshop, AIED • Embed messages in online Khan Academy exercises • Effect of Growth Mindset beyond encouragement?
  • 10. Implicit beliefs about Intelligence • On a scale from 1 to 10, how much do you agree that? – Your intelligence is something very basic about you that you can’t change very much. – No matter how much intelligence you have, you can always change it quite a bit. • Growth vs. Fixed Mindset (Dweck, 2007)
  • 11. Experimentally manipulate added messages Practice-as-usual Message Growth Mindset Message Positive Some of these problems are hard. Do your best! Remember, the more you practice the smarter you become!
  • 12. Design Practice-as-usual • Growth Mindset Message • Positive Message • • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” • "Some of these problems are hard. Just do your best." "This might be a tough problem, but we know you can do it.” • 50 000+ students per condition • Dependent measures: – Number of problems completed – Number correct on first attempt – Accuracy
  • 13. R.E.P.E.A.T. Framework for Online Education Research • • • • • • Realistic Experimental Product Evaluated Accessible Theoretically motivated • REPEAT – iteratively improve through revision & collaboration 13
  • 14. Research & Practice in Online Education • APS symposium: Williams, Saxberg, Means, Mitros. (2013). Online Learning and Psychological Science: Opportunities to integrate research and practice. • Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society.
  • 15. Experiments in Educational Research & Practice • Kinds of Experiments – – – – • Laboratory Design experiment (Brown, 1992) In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010) Randomized Controlled Trial/Intervention Experiments & Internet, Online Education – Internet-facilitated Laboratory studies • • • – – – – • • Mechanical Turk (tiny.cc/mechanicalturk ) Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics ) “A/B testing” more widespread Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning) In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop) Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams) Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013) REPEAT Approach to Experimental Research on Online/Blended Resources Identifying a Practical Online Context for your Research Question – – – – Partner with platform *early* in research “Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010) Examples of Blended Learning Resources & MOOCs Examples of resources that support Randomized Trials – – Interventions extended over time Bridging studies • • Engaging with Practitioners – – How to communicate value of experiments Consulting • • • • – – • Educational companies & publishers (marcy.baughman@pearson.com ) Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer) Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange) Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education Grant funding (IES, NSF, Gates, Spencer, moocresearch.com) Virtual collaboration on practical resources (www.learnnetwork.net ) Experiments & … – – – • Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia, Computer Supported Collaborative Learning (Rose) Cognitive Tutors (Aleven, Ritter, Koedinger) EDM & AIED (Stamper, Gordon, Pardos) Conducting Experiments (tiny.cc/conductingexperiments)
  • 16. Identifying a Practical Online Context for your Research Question – Partner with platform *early* in research – “Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010) – Examples of Blended Learning Resources & MOOCs – Examples of resources that support Randomized Trials • Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia, – Interventions extended over time – Bridging studies
  • 17. Engaging with Practitioners – How to communicate value of experiments – Consulting • Educational companies & publishers (marcy.baughman@pearson.com ) • Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer) • Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange) • Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education – Grant funding (IES, NSF, Gates, Spencer, moocresearch.com) – Virtual collaboration on practical resources (www.learnnetwork.net )
  • 18. Experiments & … – Computer Supported Collaborative Learning (Rose) – Cognitive Tutors (Aleven, Ritter, Koedinger) – EDM & AIED (Stamper, Gordon, Pardos)
  • 19. Experiments in Educational Research & Practice • Kinds of Experiments – – – – • Laboratory Design experiment (Brown, 1992) In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010) Randomized Controlled Trial/Intervention Experiments & Internet, Online Education – Internet-facilitated Laboratory studies • • • – – – – • • Mechanical Turk (tiny.cc/mechanicalturk ) Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics ) “A/B testing” more widespread Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning) In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop) Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams) Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013) REPEAT Approach to Experimental Research on Online/Blended Resources Identifying a Practical Online Context for your Research Question – – – – Partner with platform *early* in research “Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010) Examples of Blended Learning Resources & MOOCs Examples of resources that support Randomized Trials – – Interventions extended over time Bridging studies • • Engaging with Practitioners – – How to communicate value of experiments Consulting • • • • – – • Educational companies & publishers (marcy.baughman@pearson.com ) Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer) Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange) Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education Grant funding (IES, NSF, Gates, Spencer, moocresearch.com) Virtual collaboration on practical resources (www.learnnetwork.net ) Experiments & … – – – • Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia, Computer Supported Collaborative Learning (Rose) Cognitive Tutors (Aleven, Ritter, Koedinger) EDM & AIED (Stamper, Gordon, Pardos) Conducting Experiments (tiny.cc/conductingexperiments)

Editor's Notes

  1. Reframing Education to teachers – it’s about INVESTIGATION. Evidence-based practice.*Teacher buy-in, teacher CONTRIBUTION. Teacher modification.Unnatural things researchers are doing…Vs. Teachers doing things anyway, trying it out. Researchers are being involved.“Experiment”. It’s a comparison of instruction… (less the theory, more about its effectiveness).Personalization.Bridging studies.It’s on a case-by-case basis.What’s similar, what’s different.Still pretty diverse.Population demographics.Motivational issues – being paid.Are your assumptions correct? Just having to IMPLEMENT it concretely is valuable, as a DESIGN exercise.Cons:I AM CHOOSING just the right contexts. Most people start with others.> Can you adapt your research question to make it easier to run in a Bridging Study?Interventions:Short & Fat.Long & Skinny.
  2. Features. Heuristics. Guiding principles.Context of how this fits in the larger context.
  3. Reframing Education to teachers – it’s about INVESTIGATION. Evidence-based practice.*Teacher buy-in, teacher CONTRIBUTION. Teacher modification.Unnatural things researchers are doing…Vs. Teachers doing things anyway, trying it out. Researchers are being involved.“Experiment”. It’s a comparison of instruction… (less the theory, more about its effectiveness).Personalization.Bridging studies.It’s on a case-by-case basis.What’s similar, what’s different.Still pretty diverse.Population demographics.Motivational issues – being paid.Are your assumptions correct? Just having to IMPLEMENT it concretely is valuable, as a DESIGN exercise.Cons:I AM CHOOSING just the right contexts. Most people start with others.> Can you adapt your research question to make it easier to run in a Bridging Study?Short & Fat.Long & Skinny.
  4. Reframing Education to teachers – it’s about INVESTIGATION. Evidence-based practice.*Teacher buy-in, teacher CONTRIBUTION. Teacher modification.Unnatural things researchers are doing…Vs. Teachers doing things anyway, trying it out. Researchers are being involved.“Experiment”. It’s a comparison of instruction… (less the theory, more about its effectiveness).Personalization.Bridging studies.It’s on a case-by-case basis.What’s similar, what’s different.Still pretty diverse.Population demographics.Motivational issues – being paid.Are your assumptions correct? Just having to IMPLEMENT it concretely is valuable, as a DESIGN exercise.Cons:I AM CHOOSING just the right contexts. Most people start with others.> Can you adapt your research question to make it easier to run in a Bridging Study?Short & Fat.Long & Skinny.