The document summarizes research on the benefits of assessment and feedback for learning. It finds that retrieval practice through regular quizzing and testing improves long-term retention of information more than just studying. Providing feedback, especially on incorrect answers, further enhances learning by allowing students to correct their misunderstandings. The author advocates incorporating retrieval practice and feedback into the learning process to promote deeper and longer-lasting learning.

1. Learning from US Research on
Assessment and Feedback
John Kleeman, Questionmark
CETIS Assessment and Feedback : Feb 2nd 2011
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2. Introduction
 John Kleeman
 Founder and Chairman of Questionmark
 www.questionmark.com
 blog.questionmark.com
 Twitter: @johnkleeman
 Experienced e-assessment practitioner
 Not an academic nor a psychologist
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 2
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

3. Which learning method will retain
information best a week later?
A. Study a subject, no test
B. Study a subject, take a multiple choice test, no
feedback
C. Study a subject, take a multiple choice test,
feedback after each question
D. Study a subject, take a multiple choice test,
feedback at the end of the test
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 3
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

4. Graph from Roediger & Butler : The critical role of retrieval
practice in long-term retention. Trends in Cognitive
Sciences 2010.
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 4
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

5. Which learning method will retain
information best a week later?
A. Study a subject, no test
B. Study a subject, take a multiple choice test, no
feedback
C. Study a subject, take a multiple choice test,
feedback after each question
D. Study a subject, take a multiple choice
test, feedback at the end of the quiz
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 5
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

6. Retrieval helps you learn
 Francis Bacon, 1620
 If you read anything
over twenty times, you
will not learn it by
heart so easily as if
you were to read it
only ten, trying to
repeat it between
whiles, and when
memory failed, looking
at the book.
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 6
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

7. Learning
 Learning is only
useful if you can use
or retrieve the
learning
Receive Working Long term
information memory memory
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 7
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

8. Retrieval
• Question  Research evidence
Cue • Other stimulus shows that retrieval
practice helps future
retrieval
• Retrieve from long
term memory
Search
• Act or respond
Act
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 8
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

9. Two randomly selected groups study a
reading passage. Group 1 studies it for 14
minutes. Group 2 studies it for 7 minutes
and takes a quiz/test
 Who learns most?
A. Group 1
B. Group 2
C. Group 1 learn more immediately but retain less after a
week
D. Group 2 learn more immediately but retain less after a
week
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 9
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

10. Graph from Test-Enhanced Learning : Roediger
and Karpicke 2006. Psychological Science.
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 10
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

11. Two randomly selected groups study a
reading passage. Group 1 studies it for 14
minutes. Group 2 studies it for 7 minutes
and takes a quiz/test
 Who learns most?
A. Group 1
B. Group 2
C. Group 1 learn more immediately but retain less
after a week
D. Group 2 learn more immediately but retain less after a
week
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 11
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

12. Retrieval practice key findings
 Retrieval practice helps retention of learning
 Questions/quizzes are a good way of giving
retrieval practice, other routes also possible
 Unprompted questions (e.g. short answer) best
 Multiple choice questions still useful
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 12
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

13. Learning in a Perfect World (a stylized view)
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 13
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

14. Learning in the Actual World
They don’t get it All!
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 14
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

15. Learning Curve
Purpose:
• Content repetition
• Memory retrieval practice
• Strengthens memory recall
Ouch! The
Forgetting
Curve
Knowledge/Skills
Experience
Experience
Experience
Questions
Questions
Learning
Learning
Time Learning
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 15
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

16. In the Actual World
They Forget It!
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 16
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

17. Slow the Forgetting Curve
Purpose:
• Memory retrieval practice
• Strengthens memory recall
• Reduce Forgetting Curve
Knowledge/Skills
Experience
Experience
Experience
Questions
Questions
Questions
Questions
Learning
Learning
Time Learning
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 17
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

18. Where does feedback fit in?
• Question  Feedback most useful
• Other stimulus
Cue
on incorrect answers
• Retrieve from long
 Some value on
Search term memory correct answers
• Act or respond
Act
• Correct misconceptions
Feedback • Re-encode
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 18
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

19. Feedback tip #1 – give feedback
 Students only know they are wrong if you tell
them
 Short answer questions give retrieval practice
without feedback, help learning more with
feedback
 Multiple choice questions can be dangerous for
learning without feedback
 Do include the correct answer, in most contexts,
just telling people they are wrong doesn’t help
learning
 Applies also in summative exams
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 19
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

20. Feedback tip #2 – do retrieval first
 Usually force people do the retrieval before you
give them feedback
 For instance, don’t allow them to peek ahead to see the
right answer
 Otherwise you are depriving them of the retrieval
practice
 Exception when building understanding
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 20
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

21. Feedback tip #3 – give retrieval
practice later
 If someone gets a question wrong, they are
deprived the positive learning effects of retrieval
 So if you can, after giving feedback, give another
opportunity for retrieval practice (e.g. retake
questions on the topic)
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 21
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

22. Feedback tip #4 – get people to pay
attention
 Feedback is only useful if learners pay attention
 Give feedback in a way that people will attend
 Consider attention span
 Vary feedback methods
 Review whether to give feedback on correct answers
 Monitor effect of feedback
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 22
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

23. Feedback tip #5 – delay feedback if
you can
 A slight delay on feedback is usually better than
immediate feedback
 Probably due to the spacing effect in learning
 Research not fully conclusive
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 23
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

24. A reminder of the impact of retrieval
and feedback
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 24
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

25. Some questions for you
 Do people retain more by studying or by studying
and answering questions?
 Is feedback more important for short answer
questions or multiple choice questions?
 Should you allow people to peek ahead at
feedback before answering a question?
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 25
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

26. Where to learn more
 Three prominent researchers in the field
 Professor Henry Roediger
 University of Washington in St. Louis, Memory Lab
 http://psych.wustl.edu/memory/publications/
 Assistant Professor Jeffrey Karpicke
 Purdue University, Indiana
 http://memory.psych.purdue.edu/publications/
 Assistant Professor Elizabeth Marsh
 Duke University, North Carolina
 http://pn.aas.duke.edu/sites/marsh/publications.html
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 26
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

27. Dr Will Thalheimer
 Expert at translating research into practice
 Two free white papers on Questionmark website
http://www.questionmark.com/whitepapers
 The Learning Benefit of Questions
 Providing Feedback to Learners
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 27
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

28. Learn more from me
 Follow my learning in this area
 http://blog.questionmark.com
 Twitter : @johnkleeman
 Any questions or comments or suggestions feel
free to email me at john@questionmark.com
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 28
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

29. Can you give some feedback to me?
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 29
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.