1. “There are known knowns. These are
things we know that we know. There
are known unknowns. That is to say,
there are things that we know we
don't know. But there are also
unknown unknowns. There are things
we don't know we don't know.”
Donald Rumsfeld, February 2002

2. The use of regular multiple choice
quizzes to enhance the development
of students self-assessment and self-
feedback skills
Pecha Kucha Presentation
Alan Brown
CHEP Annual Conference
January 2013

3. 4 types of student?
A thirteenth-century Persian poet, Ibn
Yamin said there are four types of men:
1. One who knows and knows that
he knows... His horse of wisdom
will reach the skies.
2. One who knows, but doesn't
know that he knows... He is fast
asleep, so you should wake him
up!
Source: Wikipedia, 2013

4. 4 types of student?
3. One who doesn't know, but knows that
he doesn't know... His limping mule will
eventually get him home.
4. One who doesn't know and doesn't
know that he doesn't know... He will be
eternally lost in his hopeless oblivion!
• Our first job is to get students away from 4
Source: Wikipedia, 2013

5. Fluid Mechanics in Year 1
• Brand new subject
• Not on ‘A’ level physics or maths syllabus
• Several new concepts
– Pressure
– Viscosity
– Density

6. Fluid Mechanics in Year 1
• Several ‘Old’ concepts
– Forces
– Velocity and Acceleration
– Momentum
• But these don’t look the
same as in physics
Assessments historically
based on applied
mathematical problems

7. The problem
• Enable students to find out what they know
and do not know
• This is the role of Feedback
• Several new concepts requires lots of rapid
feedback
• I taught this module to 88 students

8. The solution
• Multiple choice quiz questions each week on
Blackboard
• Based on the previous week’s material
• The adaptive release system is used to enforce
the quiz questions

10. • Any attempt at the questions allows access to
the next set of resources
• Answers and feedback are given instantly
• The questions are designed to test conceptual
knowledge, not learned definitions
• Class test uses similar types of multiple choice
questions

11. Result 1: Class test marks
• Scores in the class test were similar to the
previous year (applied maths problems)
• Quiz designed to test the understanding of the
new and old concepts
– misunderstandings much easier to spot and correct
than using applied mathematics type questions

12. Result 1: Class test marks
Feedback to correct conceptual misunderstandings given
1 week after the test

13. Result 2: Predicting performance
• Are students more aware of their own
knowledge?
• Students were asked to predict their grade:
– Before they sat the test
– After they sat the test
– After they had seen all the answers

14. Result 2: Predicting performance
• With MCQs the students were much better at
predicting their marks
With MCQs Without MCQs

15. Result 3: Engagement
• Out of 88 students Mean overall
Test attempts mark
– 4 attempted none 0 24
– 7 attempted 1 or 2 1 12
2 25
– 15 attempted 3 or 4 3 42
– 62 attempted 5 or 4 52
5 48
more 6 53
7 54

16. Result 3: Engagement
• Student feedback on the MCQ system :
– 87 % of respondents found the MCQs
helpful or very helpful,
– 5 % said they were unhelpful
• Many students answered multiple times

17. What have I learned
• New concepts are difficult to learn
• Students need to start from a position of self-
knowledge
• Applied mathematics is not an effective way of
assessing conceptual knowledge
• Most students will engage with feedback
opportunities

18. What I will do next
• Check MCQ answers to try and determine
misunderstandings earlier
– Don’t wait until the class test
• Refine teaching based on misunderstandings
from last year
• Enhance the automated feedback in the MCQs
• Use MCQs to monitor engagement

19. Conclusions
• Rapid feedback should enhance students self-
knowledge
• MCQs used to give rapid automated feedback
to 88 students on a weekly basis
• Adaptive release gives an incentive without
marking and assessment

20. Conclusions
• Students are better able to predict their marks
• Students were overwhelmingly positive about
the quiz questions
• Engagement easy to assess
• Hopefully fewer “Unknown Unknowns”