SlideShare a Scribd company logo

Guerrilla (or Agile) Evaluation for Learning

Workplace Learning & Development professionals have a problem -- too often they don't get enough (or any) feedback on the efficacy of their designs. What can we do to fix that?

1 of 28
Guerilla 
Evaluation 
Closing the Feedback Loop 
Julie Dirksen – Usable Learning 
(c) Usable Learning 2014
Houston, we have a problem… 
Learning and development has a problem, but, 
in particular, elearning REALLY has a 
problem.
10,000 Hour Rule 
Deliberate Practice 
requires frequent, 
often expert, 
feedback.
My favorite instance...
It’s not just the time…
How many of you regularly get to 
see people use your learning? 
So how do you know if it works?

Recommended

Interface Design for Elearning - Tips and Tricks
Interface Design for Elearning - Tips and TricksInterface Design for Elearning - Tips and Tricks
Interface Design for Elearning - Tips and TricksJulie Dirksen
 
Designing for Habit Formation
Designing for Habit FormationDesigning for Habit Formation
Designing for Habit FormationJulie Dirksen
 
UXWeek 2015 - Designing for Behavior Change
UXWeek 2015 - Designing for Behavior ChangeUXWeek 2015 - Designing for Behavior Change
UXWeek 2015 - Designing for Behavior ChangeStephen Wendel
 
The Science of Behavior Change
The Science of Behavior ChangeThe Science of Behavior Change
The Science of Behavior ChangeJulie Dirksen
 
Strategies for Complex Skill Development
Strategies for Complex Skill DevelopmentStrategies for Complex Skill Development
Strategies for Complex Skill DevelopmentJulie Dirksen
 
Retrospective Anti-Patterns by Aino Corry at #AgileIndia2019
Retrospective Anti-Patterns by Aino Corry at #AgileIndia2019Retrospective Anti-Patterns by Aino Corry at #AgileIndia2019
Retrospective Anti-Patterns by Aino Corry at #AgileIndia2019Agile Software Community of India
 
Attention, Willpower and Decision-making for Design of Learning
Attention, Willpower and Decision-making for Design of Learning Attention, Willpower and Decision-making for Design of Learning
Attention, Willpower and Decision-making for Design of Learning Julie Dirksen
 
Designing for behaviour change
Designing for behaviour changeDesigning for behaviour change
Designing for behaviour changePhil Barrett
 

More Related Content

What's hot

Design for Behavior Change
Design for Behavior ChangeDesign for Behavior Change
Design for Behavior ChangeJulie Dirksen
 
Sense & Respond: Book Review & Panel Discussion
Sense & Respond: Book Review & Panel DiscussionSense & Respond: Book Review & Panel Discussion
Sense & Respond: Book Review & Panel DiscussionTremis Skeete
 
Creating a Virtuous Cycle - The Research and Design Feedback Loop
Creating a Virtuous Cycle - The Research and Design Feedback LoopCreating a Virtuous Cycle - The Research and Design Feedback Loop
Creating a Virtuous Cycle - The Research and Design Feedback LoopJulie Stanford
 
Building innovative products
Building innovative productsBuilding innovative products
Building innovative productsShishir Choudhary
 
Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...
Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...
Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...Laz Allen
 
Barcamp Conway: A Design Mindset
Barcamp Conway: A Design MindsetBarcamp Conway: A Design Mindset
Barcamp Conway: A Design Mindsetjxrichardso1
 
Graham Thomas - The Testers Toolbox - EuroSTAR 2010
Graham Thomas - The Testers Toolbox - EuroSTAR 2010Graham Thomas - The Testers Toolbox - EuroSTAR 2010
Graham Thomas - The Testers Toolbox - EuroSTAR 2010TEST Huddle
 
Everybody Lies: Rapid Cadence Research & Usability Testing
Everybody Lies: Rapid Cadence Research & Usability TestingEverybody Lies: Rapid Cadence Research & Usability Testing
Everybody Lies: Rapid Cadence Research & Usability TestingWilliam Evans
 
Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019
Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019
Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019agileoutsideIT
 
Electronic Performance Support Workshop
Electronic Performance Support WorkshopElectronic Performance Support Workshop
Electronic Performance Support Workshopguest76aa795
 
An Introduction to Applied Behavioral Science, for Project Managers
An Introduction to Applied Behavioral Science, for Project ManagersAn Introduction to Applied Behavioral Science, for Project Managers
An Introduction to Applied Behavioral Science, for Project ManagersStephen Wendel
 
CYCLES course (5): Systems and System Thinking
CYCLES course (5): Systems and System ThinkingCYCLES course (5): Systems and System Thinking
CYCLES course (5): Systems and System ThinkingBryan Cassady
 
Behavioral Science for Data Scientists
Behavioral Science for Data ScientistsBehavioral Science for Data Scientists
Behavioral Science for Data ScientistsStephen Wendel
 
You want me to what a practical guide to diary studies
You want me to what  a practical guide to diary studiesYou want me to what  a practical guide to diary studies
You want me to what a practical guide to diary studiesMichael Kennedy
 
EVOLVE & DISRUPT (Agileee 2015)
EVOLVE & DISRUPT (Agileee 2015) EVOLVE & DISRUPT (Agileee 2015)
EVOLVE & DISRUPT (Agileee 2015) Arthur Lokaychuk
 
decision making and problem solving
decision making and problem solvingdecision making and problem solving
decision making and problem solvingAntony Britto
 
Design for Behavior Change
Design for Behavior ChangeDesign for Behavior Change
Design for Behavior ChangeJulie Dirksen
 
Problem Solving 101
Problem Solving 101Problem Solving 101
Problem Solving 101Aaron Medina
 

What's hot (20)

Design for Behavior Change
Design for Behavior ChangeDesign for Behavior Change
Design for Behavior Change
 
Sense & Respond: Book Review & Panel Discussion
Sense & Respond: Book Review & Panel DiscussionSense & Respond: Book Review & Panel Discussion
Sense & Respond: Book Review & Panel Discussion
 
Creating a Virtuous Cycle - The Research and Design Feedback Loop
Creating a Virtuous Cycle - The Research and Design Feedback LoopCreating a Virtuous Cycle - The Research and Design Feedback Loop
Creating a Virtuous Cycle - The Research and Design Feedback Loop
 
Building innovative products
Building innovative productsBuilding innovative products
Building innovative products
 
Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...
Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...
Solving Problems with Theory of Constraints Current Reality Trees @ Lean Agil...
 
Barcamp Conway: A Design Mindset
Barcamp Conway: A Design MindsetBarcamp Conway: A Design Mindset
Barcamp Conway: A Design Mindset
 
Graham Thomas - The Testers Toolbox - EuroSTAR 2010
Graham Thomas - The Testers Toolbox - EuroSTAR 2010Graham Thomas - The Testers Toolbox - EuroSTAR 2010
Graham Thomas - The Testers Toolbox - EuroSTAR 2010
 
Everybody Lies: Rapid Cadence Research & Usability Testing
Everybody Lies: Rapid Cadence Research & Usability TestingEverybody Lies: Rapid Cadence Research & Usability Testing
Everybody Lies: Rapid Cadence Research & Usability Testing
 
Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019
Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019
Ross Chapman Etch Design Sprints Agile Outside IT presentation 9 January 2019
 
Electronic Performance Support Workshop
Electronic Performance Support WorkshopElectronic Performance Support Workshop
Electronic Performance Support Workshop
 
An Introduction to Applied Behavioral Science, for Project Managers
An Introduction to Applied Behavioral Science, for Project ManagersAn Introduction to Applied Behavioral Science, for Project Managers
An Introduction to Applied Behavioral Science, for Project Managers
 
CYCLES course (5): Systems and System Thinking
CYCLES course (5): Systems and System ThinkingCYCLES course (5): Systems and System Thinking
CYCLES course (5): Systems and System Thinking
 
Behavioral Science for Data Scientists
Behavioral Science for Data ScientistsBehavioral Science for Data Scientists
Behavioral Science for Data Scientists
 
Adopting innovation
Adopting innovationAdopting innovation
Adopting innovation
 
You want me to what a practical guide to diary studies
You want me to what  a practical guide to diary studiesYou want me to what  a practical guide to diary studies
You want me to what a practical guide to diary studies
 
Finding Leverage with System Dynamics
Finding Leverage with System DynamicsFinding Leverage with System Dynamics
Finding Leverage with System Dynamics
 
EVOLVE & DISRUPT (Agileee 2015)
EVOLVE & DISRUPT (Agileee 2015) EVOLVE & DISRUPT (Agileee 2015)
EVOLVE & DISRUPT (Agileee 2015)
 
decision making and problem solving
decision making and problem solvingdecision making and problem solving
decision making and problem solving
 
Design for Behavior Change
Design for Behavior ChangeDesign for Behavior Change
Design for Behavior Change
 
Problem Solving 101
Problem Solving 101Problem Solving 101
Problem Solving 101
 

Viewers also liked

L&D_Metrics-Improvement_20150409
L&D_Metrics-Improvement_20150409L&D_Metrics-Improvement_20150409
L&D_Metrics-Improvement_20150409Singydan
 
Narrative Techniques for Learning
Narrative Techniques for LearningNarrative Techniques for Learning
Narrative Techniques for LearningJulie Dirksen
 
Guided Discovery for Language Instruction
Guided Discovery for Language InstructionGuided Discovery for Language Instruction
Guided Discovery for Language InstructionVicky Saumell
 
Designing for Flow: Creating Compelling User Experiences for Learning
Designing for Flow: Creating Compelling User Experiences for Learning Designing for Flow: Creating Compelling User Experiences for Learning
Designing for Flow: Creating Compelling User Experiences for Learning Julie Dirksen
 
Discovery method Introduction
Discovery method IntroductionDiscovery method Introduction
Discovery method IntroductionMelanie Wildman
 
Employee Training and Development: How to Measure Effectiveness and Impact - ...
Employee Training and Development: How to Measure Effectiveness and Impact - ...Employee Training and Development: How to Measure Effectiveness and Impact - ...
Employee Training and Development: How to Measure Effectiveness and Impact - ...BizLibrary
 
Employee Training and Development: How to Measure Effectiveness and Impact | ...
Employee Training and Development: How to Measure Effectiveness and Impact | ...Employee Training and Development: How to Measure Effectiveness and Impact | ...
Employee Training and Development: How to Measure Effectiveness and Impact | ...BizLibrary
 
Experience the Discovery Learning Approach – Paradigm Learning
Experience the Discovery Learning Approach – Paradigm LearningExperience the Discovery Learning Approach – Paradigm Learning
Experience the Discovery Learning Approach – Paradigm LearningParadigm Learning
 
Discovery learning presentation
Discovery learning presentation Discovery learning presentation
Discovery learning presentation holzl
 
Guided Discovery
Guided DiscoveryGuided Discovery
Guided Discoveryglennvs
 
Jerome bruner learning theory
Jerome bruner learning theoryJerome bruner learning theory
Jerome bruner learning theoryRN Yogendra Mehta
 

Viewers also liked (14)

L&D_Metrics-Improvement_20150409
L&D_Metrics-Improvement_20150409L&D_Metrics-Improvement_20150409
L&D_Metrics-Improvement_20150409
 
Narrative Techniques for Learning
Narrative Techniques for LearningNarrative Techniques for Learning
Narrative Techniques for Learning
 
Guided Discovery for Language Instruction
Guided Discovery for Language InstructionGuided Discovery for Language Instruction
Guided Discovery for Language Instruction
 
Designing for Flow: Creating Compelling User Experiences for Learning
Designing for Flow: Creating Compelling User Experiences for Learning Designing for Flow: Creating Compelling User Experiences for Learning
Designing for Flow: Creating Compelling User Experiences for Learning
 
Discovery method Introduction
Discovery method IntroductionDiscovery method Introduction
Discovery method Introduction
 
Employee Training and Development: How to Measure Effectiveness and Impact - ...
Employee Training and Development: How to Measure Effectiveness and Impact - ...Employee Training and Development: How to Measure Effectiveness and Impact - ...
Employee Training and Development: How to Measure Effectiveness and Impact - ...
 
Discovery learning
Discovery learningDiscovery learning
Discovery learning
 
Employee Training and Development: How to Measure Effectiveness and Impact | ...
Employee Training and Development: How to Measure Effectiveness and Impact | ...Employee Training and Development: How to Measure Effectiveness and Impact | ...
Employee Training and Development: How to Measure Effectiveness and Impact | ...
 
Guided Discovery
Guided DiscoveryGuided Discovery
Guided Discovery
 
Experience the Discovery Learning Approach – Paradigm Learning
Experience the Discovery Learning Approach – Paradigm LearningExperience the Discovery Learning Approach – Paradigm Learning
Experience the Discovery Learning Approach – Paradigm Learning
 
Discovery learning presentation
Discovery learning presentation Discovery learning presentation
Discovery learning presentation
 
Guided Discovery
Guided DiscoveryGuided Discovery
Guided Discovery
 
Narrative
NarrativeNarrative
Narrative
 
Jerome bruner learning theory
Jerome bruner learning theoryJerome bruner learning theory
Jerome bruner learning theory
 

Similar to Guerrilla (or Agile) Evaluation for Learning

Take one step
Take one step Take one step
Take one step SEDA
 
Needs Assessment
Needs AssessmentNeeds Assessment
Needs AssessmentLeila Zaim
 
Black, Adam Dr - Efficacy and how to improve learner outcomes
Black, Adam Dr - Efficacy and how to improve learner outcomesBlack, Adam Dr - Efficacy and how to improve learner outcomes
Black, Adam Dr - Efficacy and how to improve learner outcomeseaquals
 
Training needs analysis, skills auditing and training
Training needs analysis, skills auditing and trainingTraining needs analysis, skills auditing and training
Training needs analysis, skills auditing and trainingCharles Cotter, PhD
 
Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...BASPCAN
 
Training needs analysis, skills auditing and training roi presentation 31 aug...
Training needs analysis, skills auditing and training roi presentation 31 aug...Training needs analysis, skills auditing and training roi presentation 31 aug...
Training needs analysis, skills auditing and training roi presentation 31 aug...Charles Cotter, PhD
 
Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...
Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...
Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...Michael M Grant
 
Training Program Evaluation
Training Program EvaluationTraining Program Evaluation
Training Program EvaluationLaura Pasquini
 
1 complete research
1 complete research1 complete research
1 complete researchHemant Kumar
 
Evaluation of Training & Development
Evaluation of Training & DevelopmentEvaluation of Training & Development
Evaluation of Training & DevelopmentMegha0000
 
Evaluation design
Evaluation designEvaluation design
Evaluation designMina Badiei
 
Evaluation design
Evaluation designEvaluation design
Evaluation designMina Badiei
 
Measuring Learning Impact
Measuring Learning ImpactMeasuring Learning Impact
Measuring Learning ImpactIskandar Noor
 
Vicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact EvaluationVicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact EvaluationJosh Chandler
 
Alcazar methods of evaluation
Alcazar  methods of evaluationAlcazar  methods of evaluation
Alcazar methods of evaluationYouise Saculo
 
SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...
SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...
SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...Susan Hanley
 
Next Generation Open Courseware
Next Generation Open CoursewareNext Generation Open Courseware
Next Generation Open CoursewareLumen Learning
 
Quality Assurance_Final
Quality Assurance_FinalQuality Assurance_Final
Quality Assurance_Finalkristin kipp
 

Similar to Guerrilla (or Agile) Evaluation for Learning (20)

Take one step
Take one step Take one step
Take one step
 
Needs Assessment
Needs AssessmentNeeds Assessment
Needs Assessment
 
Black, Adam Dr - Efficacy and how to improve learner outcomes
Black, Adam Dr - Efficacy and how to improve learner outcomesBlack, Adam Dr - Efficacy and how to improve learner outcomes
Black, Adam Dr - Efficacy and how to improve learner outcomes
 
Training needs analysis, skills auditing and training
Training needs analysis, skills auditing and trainingTraining needs analysis, skills auditing and training
Training needs analysis, skills auditing and training
 
Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...Using realist evaluation with vulnerable young people and the services that s...
Using realist evaluation with vulnerable young people and the services that s...
 
Developing Direct Reports
Developing Direct ReportsDeveloping Direct Reports
Developing Direct Reports
 
Training needs analysis, skills auditing and training roi presentation 31 aug...
Training needs analysis, skills auditing and training roi presentation 31 aug...Training needs analysis, skills auditing and training roi presentation 31 aug...
Training needs analysis, skills auditing and training roi presentation 31 aug...
 
Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...
Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...
Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging S...
 
Training Program Evaluation
Training Program EvaluationTraining Program Evaluation
Training Program Evaluation
 
1 complete research
1 complete research1 complete research
1 complete research
 
Evaluation of Training & Development
Evaluation of Training & DevelopmentEvaluation of Training & Development
Evaluation of Training & Development
 
Evaluation design
Evaluation designEvaluation design
Evaluation design
 
Evaluation design
Evaluation designEvaluation design
Evaluation design
 
Measuring Learning Impact
Measuring Learning ImpactMeasuring Learning Impact
Measuring Learning Impact
 
Vicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact EvaluationVicky Pelka's Training Session On Impact Evaluation
Vicky Pelka's Training Session On Impact Evaluation
 
Alcazar methods of evaluation
Alcazar  methods of evaluationAlcazar  methods of evaluation
Alcazar methods of evaluation
 
SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...
SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...
SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metric...
 
Next Generation Open Courseware
Next Generation Open CoursewareNext Generation Open Courseware
Next Generation Open Courseware
 
Trg evaluation
Trg evaluationTrg evaluation
Trg evaluation
 
Quality Assurance_Final
Quality Assurance_FinalQuality Assurance_Final
Quality Assurance_Final
 

More from Julie Dirksen

Preserving Attention for Learning in the New Normal
Preserving Attention for Learning in the New NormalPreserving Attention for Learning in the New Normal
Preserving Attention for Learning in the New NormalJulie Dirksen
 
Science of Attention for Learning
Science of Attention for LearningScience of Attention for Learning
Science of Attention for LearningJulie Dirksen
 
The Science of Attention and Engagement for Learning
The Science of Attention and Engagement for LearningThe Science of Attention and Engagement for Learning
The Science of Attention and Engagement for LearningJulie Dirksen
 
UX for Learning Design
UX for Learning DesignUX for Learning Design
UX for Learning DesignJulie Dirksen
 
Ten principles of game design for learning
Ten principles of game design for learningTen principles of game design for learning
Ten principles of game design for learningJulie Dirksen
 
Instructional Design Web Comic #4 - Addendum
Instructional Design Web Comic #4 - AddendumInstructional Design Web Comic #4 - Addendum
Instructional Design Web Comic #4 - AddendumJulie Dirksen
 
LxD - Learner Experience Design
LxD - Learner Experience DesignLxD - Learner Experience Design
LxD - Learner Experience DesignJulie Dirksen
 
Why Your Brain Loves Video Games & The Implications for e-Learning
Why Your Brain Loves Video Games & The Implications for e-LearningWhy Your Brain Loves Video Games & The Implications for e-Learning
Why Your Brain Loves Video Games & The Implications for e-LearningJulie Dirksen
 
Creating Game-like Engagement for Learning
Creating Game-like Engagement for LearningCreating Game-like Engagement for Learning
Creating Game-like Engagement for LearningJulie Dirksen
 

More from Julie Dirksen (9)

Preserving Attention for Learning in the New Normal
Preserving Attention for Learning in the New NormalPreserving Attention for Learning in the New Normal
Preserving Attention for Learning in the New Normal
 
Science of Attention for Learning
Science of Attention for LearningScience of Attention for Learning
Science of Attention for Learning
 
The Science of Attention and Engagement for Learning
The Science of Attention and Engagement for LearningThe Science of Attention and Engagement for Learning
The Science of Attention and Engagement for Learning
 
UX for Learning Design
UX for Learning DesignUX for Learning Design
UX for Learning Design
 
Ten principles of game design for learning
Ten principles of game design for learningTen principles of game design for learning
Ten principles of game design for learning
 
Instructional Design Web Comic #4 - Addendum
Instructional Design Web Comic #4 - AddendumInstructional Design Web Comic #4 - Addendum
Instructional Design Web Comic #4 - Addendum
 
LxD - Learner Experience Design
LxD - Learner Experience DesignLxD - Learner Experience Design
LxD - Learner Experience Design
 
Why Your Brain Loves Video Games & The Implications for e-Learning
Why Your Brain Loves Video Games & The Implications for e-LearningWhy Your Brain Loves Video Games & The Implications for e-Learning
Why Your Brain Loves Video Games & The Implications for e-Learning
 
Creating Game-like Engagement for Learning
Creating Game-like Engagement for LearningCreating Game-like Engagement for Learning
Creating Game-like Engagement for Learning
 

Recently uploaded

Understanding the New PCHF Analysis Guidance
Understanding the New PCHF Analysis GuidanceUnderstanding the New PCHF Analysis Guidance
Understanding the New PCHF Analysis GuidanceSafetyChain Software
 
Decision on Curriculum Change Path: Towards Standards-Based Curriculum in Ghana
Decision on Curriculum Change Path: Towards Standards-Based Curriculum in GhanaDecision on Curriculum Change Path: Towards Standards-Based Curriculum in Ghana
Decision on Curriculum Change Path: Towards Standards-Based Curriculum in GhanaPrince Armah, PhD
 
Kartik Nair In Media Res Media Component
Kartik Nair In Media Res Media ComponentKartik Nair In Media Res Media Component
Kartik Nair In Media Res Media ComponentInMediaRes1
 
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...Katherine Villaluna
 
New Features in the Odoo 17 Sales Module
New Features in  the Odoo 17 Sales ModuleNew Features in  the Odoo 17 Sales Module
New Features in the Odoo 17 Sales ModuleCeline George
 
Dr.M.Florence Dayana-Cloud Computing-Unit - 1.pdf
Dr.M.Florence Dayana-Cloud Computing-Unit - 1.pdfDr.M.Florence Dayana-Cloud Computing-Unit - 1.pdf
Dr.M.Florence Dayana-Cloud Computing-Unit - 1.pdfDr.Florence Dayana
 
Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Sue Beckingham
 
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdfA LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdfDr.M.Geethavani
 
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Aggregage
 
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...EduSkills OECD
 
Genetic deterioration Seed ageing of improved variety seed, Maintenance of G...
Genetic deterioration  Seed ageing of improved variety seed, Maintenance of G...Genetic deterioration  Seed ageing of improved variety seed, Maintenance of G...
Genetic deterioration Seed ageing of improved variety seed, Maintenance of G...AKSHAYMAGAR17
 
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...AKSHAYMAGAR17
 
EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...
EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...
EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...liera silvan
 
2.22.24 Black Nationalism and the Nation of Islam.pptx
2.22.24 Black Nationalism and the Nation of Islam.pptx2.22.24 Black Nationalism and the Nation of Islam.pptx
2.22.24 Black Nationalism and the Nation of Islam.pptxMaryPotorti1
 
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfDiploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfSUMIT TIWARI
 
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptxBBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptxProf. Kanchan Kumari
 

Recently uploaded (20)

Understanding the New PCHF Analysis Guidance
Understanding the New PCHF Analysis GuidanceUnderstanding the New PCHF Analysis Guidance
Understanding the New PCHF Analysis Guidance
 
Decision on Curriculum Change Path: Towards Standards-Based Curriculum in Ghana
Decision on Curriculum Change Path: Towards Standards-Based Curriculum in GhanaDecision on Curriculum Change Path: Towards Standards-Based Curriculum in Ghana
Decision on Curriculum Change Path: Towards Standards-Based Curriculum in Ghana
 
Kartik Nair In Media Res Media Component
Kartik Nair In Media Res Media ComponentKartik Nair In Media Res Media Component
Kartik Nair In Media Res Media Component
 
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
 
New Features in the Odoo 17 Sales Module
New Features in  the Odoo 17 Sales ModuleNew Features in  the Odoo 17 Sales Module
New Features in the Odoo 17 Sales Module
 
Dr.M.Florence Dayana-Cloud Computing-Unit - 1.pdf
Dr.M.Florence Dayana-Cloud Computing-Unit - 1.pdfDr.M.Florence Dayana-Cloud Computing-Unit - 1.pdf
Dr.M.Florence Dayana-Cloud Computing-Unit - 1.pdf
 
Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...
 
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdfA LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
A LABORATORY MANUAL FOR ORGANIC CHEMISTRY.pdf
 
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
 
CLUBE PERLINGUAS .
CLUBE PERLINGUAS                        .CLUBE PERLINGUAS                        .
CLUBE PERLINGUAS .
 
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
 
Genetic deterioration Seed ageing of improved variety seed, Maintenance of G...
Genetic deterioration  Seed ageing of improved variety seed, Maintenance of G...Genetic deterioration  Seed ageing of improved variety seed, Maintenance of G...
Genetic deterioration Seed ageing of improved variety seed, Maintenance of G...
 
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
Ideotype concept and climate resilient crop varieties for future- Wheat, Rice...
 
Lipids as Biopolymer
Lipids as Biopolymer Lipids as Biopolymer
Lipids as Biopolymer
 
Advance Mobile Application Development class 04
Advance Mobile Application Development class 04Advance Mobile Application Development class 04
Advance Mobile Application Development class 04
 
EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...
EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...
EmpTech Lesson 7 - Online Creation Tools, Platforms, and Applications for ICT...
 
2.22.24 Black Nationalism and the Nation of Islam.pptx
2.22.24 Black Nationalism and the Nation of Islam.pptx2.22.24 Black Nationalism and the Nation of Islam.pptx
2.22.24 Black Nationalism and the Nation of Islam.pptx
 
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfDiploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
 
ThinkTech 2024 Prelims- U25 BizTech Quiz by Pragya
ThinkTech 2024 Prelims- U25 BizTech Quiz by PragyaThinkTech 2024 Prelims- U25 BizTech Quiz by Pragya
ThinkTech 2024 Prelims- U25 BizTech Quiz by Pragya
 
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptxBBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
BBA 603 FUNDAMENTAL OF E- COMMERCE UNIT 1.pptx
 

Guerrilla (or Agile) Evaluation for Learning

  • 1. Guerilla Evaluation Closing the Feedback Loop Julie Dirksen – Usable Learning (c) Usable Learning 2014
  • 2. Houston, we have a problem… Learning and development has a problem, but, in particular, elearning REALLY has a problem.
  • 3. 10,000 Hour Rule Deliberate Practice requires frequent, often expert, feedback.
  • 5. It’s not just the time…
  • 6. How many of you regularly get to see people use your learning? So how do you know if it works?
  • 7. Typical Evaluation Measures Kirkpatrick's Levels 1. Reaction (participants' opinions) 2. Learning (pre/post test) 3. Behavior (measurable behavior change) 4. Results/ROI (return on investment)
  • 8. Issues with Typical Measures • Levels 1 & 2 are not meaningful • Levels 3 & 4 are difficult and costly o Require access to the full target audience o Measuring behaviors requires extensive and costly observation o Difficult to implement without pre-existing organizational performance metrics in place o Difficult to attribute due to confounding variables
  • 9. The Evaluation Venn Diagram Enough budget or resources or resources to measure Enough control over the environment Good methods to evaluate All too often, they don’t overlap at all.
  • 11. We are measuring what we can control • Seat Time • # of learning objects • # of people trained • Completion Status • Pre/post scores Why don’t we just weigh them? The Inestimable Gloria Gery
  • 12. Streetlamp Effect So, there’s this story…
  • 13. It’s kind of like this…
  • 14. So, what can we do about this? visible desirable feasible What is in this intersection?
  • 15. Guerilla Evaluation • A quicker and less expensive method to ensure a feedback loop that can be used to assess and improve the training intervention • Not intended to be a full measure of efficacy • Qualitative measures of: o Retention of information o Attitude o Anecdotal or Observable behavior change for a small sample size
  • 16. Based on Nielsen's Guerilla HCI In 1994, Jakob Nielsen wrote a highly influential article called Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier The article addressed the reasons software development teams rarely did usability research to improve the design of software interfaces. Studies showed that qualitative feedback quickly became repetitive after 5-6 users, and that working with a small sample could provide meaningful design feedback. http://www.nngroup.com/articles/guerrilla-hci/.
  • 17. Right vs. Better https://twitter.com/karlfast/status/223825451079057408
  • 18. It’s like Traditional PM vs Agile VS To Do Doing Done “By putting the most serious planning at the beginning, with subsequent work derived from the plan, the waterfall method amounts to a pledge by all parties not to learn anything while doing the actual work.” - Clay Shirky
  • 19. Keep the cycles short Why feedback is like weather prediction
  • 20. Formative - User Testing Standard Usability Testing The first part of the evaluation process is standard usability testing that involves watching end users interact with the software, followed by a short interview. Typical evaluation measures such as a pre/post test could be incorporated here.
  • 21. Summative - Follow up Interview • Can be used in conjunction with other evaluation measures • 30-45 minute follow-up interviews that occur 4-6 weeks after the training intervention • Small sample group (~6 users per audience) • Structured interview questions
  • 22. Structured Interview Format Structured interview questions relating to: ● Learner impressions/feedback ● Most memorable elements ● Small number of retention questions related to key learning objectives ● Anecdotal usage of the material (How have they applied the ideas from the training?)
  • 23. Brinkerhoff Success Case “Performance results can’t be achieved by training alone; therefore training should not be the object of evaluation” • Part 1: Survey to determine who was successful and who was not • Part 2: In depth interviews with a selection of successful and not successful users Find Out Quickly What’s Working and What’s Not
  • 24. Cohort Analysis • Follow smaller groups through level 3 analysis 100 90 80 70 60 50 40 30 20 10 0 Week 1 Week 2 Week 3 Week 4 Cohort 1 Cohort 2
  • 25. Signaling Ask the magic question: If you woke up tomorrow and it was all perfect, how would you know?
  • 26. Look for data • xAPI • Google Analytics
  • 27. What do you think? • With 2-3 people around you, make a list of quick and dirty evaluation options. • As soon as you think of one, come up with another one as quickly as possible.
  • 28. Questions? • Thanks for coming • Contact: o Julie Dirksen o julie@usablelearning.com o http://usablelearning.com o Twitter: usablelearning