Tranformational Model of Translational Research that Leverages Educational Technology for Fast Data-Discovery Feedback Loops

Transformational Model of TranslationalResearch that Leverages EducationalTechnology for Fast Data-DiscoveryFeedback LoopsJohn StamperPittsburgh Science of Learning CenterHuman-Computer InteractionCarnegie Mellon UniversityConnecting How we Learn to EducationalPractice and Policy: Research Evidence andImplications International Conference23-24 January 2012 1

rigorous, sustained scientific research in education (NRC, 2002)Vision for PSLC•  Why? Chasm between science & practice –  Low success rate (<10%) of randomized field trials•  LearnLab = a socio-technical bridge between lab psychology & schools –  E-science of learning & education –  Social processes for research-practice engagement•  Purpose: Leverage cognitive theory and computational modeling to identify the conditions that cause robust student learning 2

PSLC is multidisciplinary170+ multidisciplinary researchers from California toGermanyKen Koedinger - Carnegie Mellon Co-DirectorCharles Perfetti - University of Pittsburgh Co-DirectorExecutive Committee:Vincent Aleven (HCI), Maxine Eskenazi (LTI; Diversity Director),Julie Fiez (Psych), Geoff Gordon (ML), David Klahr (Psych; EducationDirector), Marsha Lovett (Psych), Tim Nokes (Psych), Lauren Resnick(Psych), Carolyn Rose (LTI), John Stamper (HCI) 3

The Setting & Inspiration•  Rich tradition of research on Learning and Instruction at CMU & University of Pittsburgh –  Basic Cognitive Science –  Research in schools –  Intelligent tutors•  PSLC inspiration: Educational technology as research platform to launch new learning science Built in generalization to practice, dissemination. 4

Translational ResearchFeedback Loop DesignDiscover Deploy Data 5

Real World Impact ofCognitive Science Algebra Cognitive Tutor•  Based on computational models of student thinking & learning•  Course used nation wide – Over 2600 schools, 500K students use for ~80 minutes per week•  Spin-off: Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

Scaling UpLearning & EducationalEducational PracticeScience 7

Translational Research 1:Bringing Cognitive Science to School Research base Practice base Cognitive Psychology Math Educators Artificial Intelligence Standards Design Cognitive Tutor courses: Tech, Text, Training Deploy Address social context 8

Which kind of problem is mostdifficult for Algebra students? Story Problem As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work? Word Problem Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with? Equation x * 6 + 66 = 81.90 9

Data contradicts common beliefs of researchers and teachers Expert Blind Spot! High School Algebra Students 100! 100% 90! % Correctly ranking equations as 80! hardest ! 70!Percent Correct 80% 70% 61% 60! 60% 50! 42% 40! 40% 30! 20! 20% 10! 0! 0% Elementary! Middle! High School! Story Word Equation Teachers! School! Teachers! Teachers! Problem Representation Koedinger & Nathan (2004). The real story behind story Nathan & Koedinger (2000). An investigation of problems: Effects of representations on quantitative teachers beliefs of students algebra development. reasoning. The Journal of the Learning Sciences. Cognition and Instruction.

Cognitive Tutor TechnologyUse ACT-R theory to individualize instruction•  Cognitive Model: A system that can solve problems in the various ways students can 3(2x - 5) = 9 If goal is solve a(bx+c) = d If goal is solve a(bx+c) = d Then rewrite as abx + ac = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a 6x - 15 = 9 2x - 5 = 3 6x - 5 = 9•  Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction 11

Cognitive Tutor TechnologyUse ACT-R theory to individualize instruction•  Cognitive Model: A system that can solve problems in the various ways students can 3(2x - 5) = 9 If goal is solve a(bx+c) = d If goal is solve a(bx+c) = d Then rewrite as abx + ac = d Then rewrite as abx + c = d Hint message: Distribute a Bug message: You need to across the parentheses. multiply c by a also. Known? = 85% chance Known? = 45% 6x - 15 = 9 2x - 5 = 3 6x - 5 = 9•  Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction•  Knowledge Tracing: Assesses students knowledge growth -> individualized activity selection and pacing 12

Translational Research 1:Bringing Cognitive Science to School Research base Practice base Cognitive Psychology Math Educators Artificial Intelligence Standards Design Cognitive Tutor courses: Tech, Text, Training Deploy Address social context 13

Cognitive Tutor Algebra: Problemsthat engage intuition & interestHealth CareExtinctionLocal FactsSmoking RisksImportance of Math Education 14

Cognitive Tutor Algebra: Rich Interactions 15

Cognitive Tutor Algebra: Rich Interactions 16

Cognitive Tutor Algebra course yields significantly better learning 60Course includes text, Traditional Algebra Coursetutor, teacher 50 Cognitive Tutor Algebraprofessionaldevelopment 40 308 of 10 full-yearcontrolled studies 20demonstratesignificantly better 10student learning 0 Iowa SAT subset Problem Represent- Solving ations Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

Scaling success? YesDone? No!Why not?•  Final performance particularly in urban schools is still far from desirable•  Weaknesses in field study results –  Not all studies are random assignment –  Two null results•  Many design decisions not guided by science•  We can use the deployed technology to collect data, make discoveries, and continually improve the instructional design 18

Scaling DownLearning & EducationalEducational PracticeScience 19

Translational Research 2: Fielded Systems Provide Data for New Discoveries Research base Practice base Cognitive Psychology Math Educators Artificial Intelligence Standards Design Cognitive Tutor courses: Tech, Text, TrainingDiscover DeployCognition, learning, Address social contextinstruction, context Data Qual, quant; process, product 20

How are cognitive models developed?Cognitive Task AnalysisTraditional methods•  Structured interviews & think alouds of experts & novices=> Create symbolic modelNewer methods•  Data-Driven•  Educational Data Mining=> Create statistical model => symbolic modelMeta-analysis: CTA produces 1.7 effect size (Lee, 2004) 21

Good Cognitive Model =>Good Learning Curve•  An empirical basis for determining when a cognitive model is good•  Accurate predictions of student task performance & learning transfer –  Repeated practice on tasks involving the same skill should reduce the error rate on those tasks => A declining learning curve should emerge 22

A good cognitive model produces a learning curve Without decomposition, using just a single Geometry skill, no smooth learning curve.But with decomposition, (Rise in error rate because12 skills for area, poorer students get assigned more problems)a smooth learning curve. Is this the correct or best cognitive model? 23

Inspect curves for individualknowledge components (KCs) Many curves show a Some do not => reasonable decline Opportunity to improve model!

Can a data-driven process beautomated & brought toscale?Yes!•  Combine Cognitive Science, Psychometrics, Machine Learning …•  Collect a rich body of data•  Develop new model discovery algorithms, visualizations, & on-line collaboration support 25

Automating the CognitiveModel Discovery ProcessLearning Factors Analysis•  Input –  Factors that may differentiate tasks –  Student performance across tasks & over time•  Output: Best cognitive model Cen, H., Koedinger, K., Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. 26

Discovery of new cognitivemodels: Strategy & Results•  Mixed initiative human & machine discovery –  Visualizations to aid human discovery –  AI search for statistically better models Stamper, J., Koedinger, K.R. (2011) Human-machine Student Model Discovery and Improvement Using DataShop. •  Better models discovered in Geometry, Statistics, English, Physics 27

LFA –Model Search Process •  Fully automated machine learning guided search •  Input: Existing proposed models •  Output: Best cognitive model based on splitting and merging existing models Original Model BIC = 4328 Split by Embed Split by Backward Split by Initial Automates the process of 50+ 4301 4322 4312 4320 hypothesizing alternative cognitive4320 4322 4313 4322 4325 4324 models & testing them against data 15 expansions later 4248 28

Summary•  Most ed field trials yield null results –  Need better data & cumulative theory•  Optimal instructional design requires discoveries –  The student is not like me•  Scale up success: Cognitive Tutor Algebra•  LearnLab: E-science infrastructure to support science of learning 29

Opportunities Ahead•  Better models => better instruction•  Combine cog sci & machine learning –  Machine learning competitions –  PSLC s DataShop has 300+ datasets –  SimStudent learns new models 30

Thank you!Acknowledgements•  Cognitive Tutors John R. Anderson (Psych), Albert Corbett (HCI), Steve Ritter (Carnegie Learning), …•  Cognitive Task Analysis Mitchell Nathan (UW Ed Psych), Mimi McLaughlin (HCI), Neil Heffernan (WPI CS), Marsha Lovett (Psych) …•  Cognitive Model Discovery Brian Junker (Stats), Hao Cen (Machine Learning), Geoff Gordon (ML) …•  Pittsburgh Science of Learning Center –  Kurt VanLehn (ASU CS) -- original PSLC co-director –  Ken Koedinger (HCI/Pysch), Charles Perfetti (Upitt Psych), David Klahr (Psych), Lauren Resnick (Upitt Psych), Vincent Aleven (HCI), Maxine Eskenazi (LTI), Carolyn Rose (LTI/HCI) –  All 200+ past & current members! 31

Tranformational Model of Translational Research that Leverages Educational Technology for Fast Data-Discovery Feedback Loops

by OECD Education

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 64

Statistics

Tranformational Model of Translational Research that Leverages Educational Technology for Fast Data-Discovery Feedback Loops Presentation Transcript