New Research Shows How to Use               Quizzes and Tests to Slow the                           Forgetting Curve      ...
Would you like to                                                  reduce learner                                         ...
Retrieval helps you learn         Francis Bacon, 1620                                                                    ...
Learning        Receive      Working                    Long term      information    memory                      memory  ...
Learning useful if can retrieve                • Question                                                      Research s...
Experimental methodology                                                                      Quiz / retrieval practice   ...
Roediger & Karpicke experiment 2006         120 students divided into groups randomly         Given reading comprehensio...
How much learners retained     90%     80%     70%     60%     50%                                                        ...
Marsh & others 2007. Experiments    with multiple choice questions     60%     50%     40%     30%                        ...
Roediger & Butler. Feedback on    multiple choice questions                           Retention after one week    60%    5...
Karpicke & Blunt experiment 2011          80 students in 4 groups, studying a science text              Study           ...
How much learners retained                                                                                               A...
Larson experiment 2009Quizzes reduce forgetting curve   In real-life, not in the laboratory   Medical residents (doctors...
Part of the results                                            Quiz group                        Re-study group           ...
Retrieval practice key findings          Retrieval practice helps retention of learning              Applies to facts   ...
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Computer quizzes are effective    retrieval practice because …          Learners do not realize they will forget         ...
Indirect benefits of post-learning    assessments          Direct to further learning          Correct misconceptions   ...
If you are a practitioner …          Give quizzes after learning sessions          Focus questions on important things t...
If you are a manager …          How easy is it for learners to take quizzes?          How easy is it for your instructor...
Potential objections          “No time”          “We teach creativity not facts”          “How sure are you of the rese...
In a Perfect World…            Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known...
In a Perfect World…    we would remember everything we learn!           Copyright © 1995-2011 Questionmark Corporation and...
In the Actual World…            Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, know...
In the Actual World…    we forget some of it!            Copyright © 1995-2011 Questionmark Corporation and/or Questionmar...
Questions through Learning Process                                                                                        ...
ReferencesWho           Title                                        Journal                         Date             Desc...
Some questions for you:          What is the best way to remember this session for           the future?          What q...
Your questions          Contact me at john at questionmark.com or           Twitter: @johnkleeman                  Copyri...
Upcoming SlideShare
Loading in...5
×

ONLINE EDUCA 2011

2,623

Published on

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,623
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
24
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • In the actual world…
  • ONLINE EDUCA 2011

    1. 1. New Research Shows How to Use Quizzes and Tests to Slow the Forgetting Curve Educa Online, December 1st 2011 John Kleeman, Questionmark Chairman www.questionmark.comCopyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    2. 2. Would you like to reduce learner forgetting curves? Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 2 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    3. 3. Retrieval helps you learn  Francis Bacon, 1620 If you read anything over twenty times, you will not learn it by heart so easily as if you were to read it only ten, trying to repeat it between whiles, and when memory failed, looking at the book. Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 3 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    4. 4. Learning Receive Working Long term information memory memory Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 4 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    5. 5. Learning useful if can retrieve • Question  Research shows Cue • Other stimulus retrieval practice helps future retrieval • Retrieve from long term memory  Successful retrieval Search modifies memory making it more • Act or respond retrievable in future Act Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 5 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    6. 6. Experimental methodology Quiz / retrieval practice Re-study group group Study Study Re-study Quiz Measure Measure Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 6 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    7. 7. Roediger & Karpicke experiment 2006  120 students divided into groups randomly  Given reading comprehension passages used in TOEFL  Some students  Study for 14 mins  Some students  Study for 7 mins  Recall material for 7 mins  Compared results after 5 mins and 1 week Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 7 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    8. 8. How much learners retained 90% 80% 70% 60% 50% Restudy 40% Quiz 30% 20% 10% 0% After 5 mins After 1 week Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 8 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    9. 9. Marsh & others 2007. Experiments with multiple choice questions 60% 50% 40% 30% No quiz Quiz 20% 10% 0% SAT II Bloom Definition Bloom Application Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 9 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    10. 10. Roediger & Butler. Feedback on multiple choice questions Retention after one week 60% 50% 40% 30% 20% 10% 0% No quiz Quiz, no feedback Quiz, immediate Quiz, delayed feedback feedback Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 10 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    11. 11. Karpicke & Blunt experiment 2011  80 students in 4 groups, studying a science text  Study  Repeated Study  Concept Mapping (mind maps)  Retrieval Practice  Surveyed learners on expectation of retention  Tested a week later to see how much retained Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 11 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    12. 12. How much learners retained Actual retention Learner expectation of 80% retention 70% 80% 70% 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% 0% Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 12 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    13. 13. Larson experiment 2009Quizzes reduce forgetting curve In real-life, not in the laboratory Medical residents (doctors in training) 20 doctors in each group Quiz group Restudy group Day 1 Study Study Day 1 Quiz Re-study 2 weeks later Quiz Re-study 2 weeks later Quiz Re-study 6 months later Final exam Final exam Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    14. 14. Part of the results Quiz group Re-study group 62% 48% 44% 36% 19% Initial Quiz Quiz 6 months Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 14 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    15. 15. Retrieval practice key findings  Retrieval practice helps retention of learning  Applies to facts  And also concepts  Retrieval practice includes  Recall  Formative quizzes/tests  Self-assessment  Explain or read back of material  Timing of retrieval  Works best when spaced out  Strongest effect when retrieval is difficult  Feedback very helpful  Repetition of retrieval  Research suggests 5-7 spaced repetitions helpful Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 15 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    16. 16. Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 16 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    17. 17. Computer quizzes are effective retrieval practice because …  Learners do not realize they will forget  Quizzes encourage/force retrieval practice  Quizzes give corrective feedback  And also:  Randomize so easy to repeat  Mobile devices  Accessible  Measure that practice has happened  Computer assessments also have many “indirect“ benefits Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 17 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    18. 18. Indirect benefits of post-learning assessments  Direct to further learning  Correct misconceptions  Feed back to improve the course  Compare to pre-test  Measure achievement  Encourage study  Engage students Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 18 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    19. 19. If you are a practitioner …  Give quizzes after learning sessions  Focus questions on important things to remember  Make quizzes effortful (delay & difficulty)  Question types:  Recall (e.g. short answer) best  Multiple choice work too  Include feedback  Usually the correct answer  Slight delay if possible  Encourage retrieval (don’t allow peek ahead)  Spacing helps significantly Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 19 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    20. 20. If you are a manager …  How easy is it for learners to take quizzes?  How easy is it for your instructors to quiz learners?  Could you make it easier? Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 20 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    21. 21. Potential objections  “No time”  “We teach creativity not facts”  “How sure are you of the research?”  “Our learners don’t like to be tested”  “What about privacy?” Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 21 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    22. 22. In a Perfect World… Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 22 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    23. 23. In a Perfect World… we would remember everything we learn! Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 23 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    24. 24. In the Actual World… Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 24 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    25. 25. In the Actual World… we forget some of it! Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 25 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    26. 26. Questions through Learning Process Reduce Forgetting Create Intrigue Knowledge/Skills Strengthen Memory Questions Questions Questions Questions Questions Time Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 26 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    27. 27. ReferencesWho Title Journal Date Description URLRoegiger & Test-Enhanced Learning Psychological 2006 Widely cited paper HereKarpicke Science showing benefits of testing vs studyingMarsh, The memorial consequences Psychonomic 2007 Shows benefits of multiple HereRoediger, of multiple-choice testing Bulletin & Review choice questionsBjork&BjorkLarsen, Test-enhanced learning in Medical Education 2009 Shows experiment on HereButler & medical education. medical education andRoediger retrieval practiceFazio, Right/wrong feedback: Memory 2010 Research showing just HereHuelser, Consequences for learning telling students right orJohnson & wrong without correctMarsh answer unhelpfulRoediger & The critical role of retrieval Trends in Cognitive 2010 8 page review article on HereButler practice in long-term retention Sciences retrieval and retentionKarpicke & Retrieval Practice Produces Science Magazine 2011 Clear paper comparing Select from hereBlunt More Learning than Elaborative retrieval with study and (non- Studying with Concept Mapping mind maps commercial)Kleeman Recent cognitive psychology International CAA 2011 Some more detail from Here research shows strongly that Conference me quizzes help retain learning. What does this mean1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Copyright © for CAA? Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    28. 28. Some questions for you:  What is the best way to remember this session for the future?  What question types seem to give best effect?  What is the best timing for feedback? Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 28 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    29. 29. Your questions  Contact me at john at questionmark.com or Twitter: @johnkleeman Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Slide 29 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×