PEER INSTRUCTION 1:
WRITING GOOD PEER
INSTRUCTION QUESTIONS
Peter Newbury
Center for Teaching Development,
University of California, San Diego
pnewbury@ucsd.edu @polarisdotca
ctd.ucsd.edu #ctducsd
resources: ctd.ucsd.edu/programs/weekly-workshops-spring-2014/
Wednesday, May 7, 2014
12:00 – 12:50 pm Center Hall, Rm 316
please
sign in
2
Writing good peer instruction questions
Clicker Question (Economics)
Writing good peer instruction questions3
For which of the following professionals is driving an
expensive car a sign of success compared to others in the
same profession?
A) a carpenter
B) a realtor
C) a politician
D) a major league baseball player
(adapted from Steve Morris, UCSD)
Typical episode of peer instruction
Writing good peer instruction questions4
1. Instructor poses a conceptually-challenging
multiple-choice question.
2. Students think about question on their own
and vote using clickers, colored ABCD cards,
smartphones,…
3. The instructor prompts students, “Turn to your
neighbors and convince them you’re right.”
4. After the peer-to-peer discussion, [the students vote
again and] the instructor leads a class-wide
discussion concluding with why the right answer(s) is
right and the wrong answers are wrong.
In effective peer instruction
Writing good peer instruction questions5
 students teach each other while
they may still hold or remember
their novice preconceptions
 students discuss the concepts in their
own (novice) language
 each student finds out what s/he does(n’t) know
 the instructor finds out what the students (don’t) know
and reacts, building on their initial understanding
and preconceptions.
students learn
and practice
how to think,
communicate
like experts
Effective peer instruction requires
Writing good peer instruction questions6
1. identifying key concepts, misconceptions
2. creating multiple-choice questions that
require deeper thinking and learning
3. facilitating peer instruction episodes that
spark and support student discussion
4. leading a class-wide discussion to clarify
the concept, resolve the misconception
5. reflecting on the question: note curious
things you overheard, how they voted, etc.
before
class
during
class
after
class
today
next week
What makes a good clicker question?
Writing good peer instruction questions7
clarity Students should waste no effort trying to figure
out what’s being asked.
context Is this topic currently being covered
in class?
connection to
learning goals
Does the question make students do the right
thing to demonstrate they grasp the concept.
distractors What do the “wrong” answers tell you about
students’ thinking?
difficulty Is the question too trivial? too hard?
stimulates
thoughtful
discussion
Will the question engage the students and
spark thoughtful discussions?
Is there potential for you to be “agile”?
(Adapted from Stephanie Chasteen, CU Boulder)
t h e l e a r n i n g c y c l e
Peer instruction helps students learn...
Writing good peer instruction questions8
BEFORE DURING AFTER
setting up
instruction
developing
knowledge
assessing
learning
Adapted from Rosie Piller, Ian Beatty, Stephanie Chasteen
Clicker question
Writing good peer instruction questions9
Melt chocolate over low heat. Remove the chocolate
from the heat. What will happen to the chocolate?
A) It will condense.
B) It will evaporate.
C) It will freeze.
(Question: Sujatha Raghu from Braincandy via LearningCatalytics)
(Image: CIM9926 by number657 on flickr CC)
assess prior knowledge
Clicker question
Writing good peer instruction questions10
In your opinion, which had the most positive impact on
the modern world? Be prepared to defend your choice
with evidence from the readings.
A) coffee
B) tea
C) chocolate
D) spice
E) sugar
(Herbst, UCSD)
provoke thinking
Clicker question
Writing good peer instruction questions11
A ball is rolling around
the inside of a circular
track. The ball
leaves the track
at point P.
Which path
does the ball
follow?
P
A
B
C
E
D
(adapted from Mazur)
predict
t h e l e a r n i n g c y c l e
Peer instruction helps students learn...
Writing good peer instruction questions12
BEFORE DURING AFTER
setting up
instruction
developing
knowledge
assessing
learning
Adapted from Rosie Piller, Ian Beatty, Stephanie Chasteen
The students have not
solved concept X.
But they’re know X exists
and why X is interesting.
Clicker question
Writing good peer instruction questions13
Which of these are reasons for the seasons?
i. the height of the Sun in the sky during the day
ii. Earth’s distance from the Sun
iii. how many hours the Sun is up each day
A) ii only
B) iii only
C) i and ii
D) i and iii
E) i, ii and iii
clarity
context
learning goals
distractors
difficulty
discussion
probe misconception
Clicker question
Writing good peer instruction questions14
How many of these are reasons for the seasons?
height: the height of the Sun in the sky during the day
distance: Earth’s distance from the Sun
hours: how many hours the Sun is up each day
A) none of them
B) one
C) two
D) all three
probe misconception
Clicker question
Writing good peer instruction questions15
Select the line that
you feel has the
strongest imagery in
“Fast rode the
knight” by Stephen
Crane (1905).
analysis
Fast rode the knight
With spurs, hot and reeking,
Ever waving an eager sword,
"To save my lady!"
Fast rode the knight,
And leaped from saddle to war.
Men of steel flickered and gleamed
Like riot of silver lights,
And the gold of the knight's good banner
Still waved on a castle wall.
. . . . .
A horse,
Blowing, staggering, bloody thing,
Forgotten at foot of castle wall.
A horse
Dead at foot of castle wall.
A
B
C
D
E
(David Kurtz, via LearningCatalytics)
Clicker question
Writing good peer instruction questions16
Evaluate:
A)
B)
(adapted from Bruff (2009))
 
4
0
32
1 dxxx
23
)65(16
9
16
C.
D.
)165(
9
2 23

3
1022
clarity
context
learning goals
distractors
difficulty
discussion
exercise skill
Clicker question
Writing good peer instruction questions17
Which of the following is an incorrect step when using
the substitution method to evaluate the definite integral
A)
B)
 
4
0
32
1 dxxx
3
1 xu 
dxx
du 2
3

C.
D. none of the above

4
03
1
duu
(adapted from Bruff (2009))
exercise skill
Clicker question
Writing good peer instruction questions18
According to Augustine, which do you think was the most
significant event in his conversion to Christianity? Be
prepared to defend your choice with evidence from the
readings.
A) Corporal punishments as a school-boy
B) Student years of follies (studies and promiscuity)
C) Stealing the pears from the neighbor’s tree
D) Death of his classmate/friend
exercise skill analysis
Heidi Keller-Lapp (UCSD)
Clicker question
Writing good peer instruction questions19
Susan throws a ball straight up into the air. It goes up
and then falls back into her hand 2 seconds later.
Draw a graph showing the velocity of the ball from the
moment it leaves her hand until she catches it again.
time
velocity
2 sec0
exercise skill
(CWSEI UBC)
time
velocity
2 sec0
A
time
velocity
2 sec0
B
time
velocity
2 sec0
C
time
velocity
2 sec0
D
E) some other graph
Which one is the closest match to your graph? exercise skill
(CWSEI UBC)Writing good peer instruction questions20
t h e l e a r n i n g c y c l e
Peer instruction helps students learn...
Writing good peer instruction questions21
BEFORE DURING AFTER
setting up
instruction
developing
knowledge
assessing
learning
Adapted from Rosie Piller, Ian Beatty, Stephanie Chasteen
Students have had
opportunities to
try, fail, receive feedback
and try again without facing
a summative evaluation. [3]
Writing good peer instruction questions22
Clicker question
Are features X and Y
ridges or valleys?
A) X=ridge, Y=valley
B) X=valley, Y=ridge
C) both are ridges
D) both are valleys
X
Y
(EOSC / CWSEI, UBC)
demonstrate success
Clicker question
Writing good peer instruction questions23
What are the mean and median of these numbers?
74, 32, 35, 87, 28, 36, 11, 26, 93, 56, 34, 52, 8
A) 13, 36
B) 44, 35
C) 572, 93
D) 48,13
(terribly adapted from Peck, mathquest.carroll.edu/resources.html)
review / recap
clarity
context
learning goals
distractors
difficulty
discussion
Clicker question
Writing good peer instruction questions24
For the data given below, which is larger, the mean or
the median?
74, 32, 35, 87, 28, 36, 11, 26, 93, 56, 34, 52, 8
A) mean
B) median
(poorly adapted from Peck, mathquest.carroll.edu/resources.html)
review / recap
clarity
context
learning goals
distractors
difficulty
discussion
Clicker question
Writing good peer instruction questions25
For the data set displayed in the following histogram,
which would be larger, the mean or the median?
A) mean
B) median
C) can’t tell from the given histogram
(adapted from Peck, mathquest.carroll.edu/resources.html)
review / recap
Clicker question
Writing good peer instruction questions26
In your opinion, which had the most positive impact on
the modern world?
A) coffee
B) tea
C) chocolate
D) spice
E) sugar
“big picture”
(Herbst, UCSD)
Clicker question
Writing good peer instruction questions27
Traxoline is a new form of zionter. It is monotilled in
Ceristanna. The Ceristannians gristerlate large amounts
of fevon and then bracter it to quasel traxoline.
Traxoline may well be one of our most lukised snezlaus
in the future because of our zionter lescelidge.
Why is traxoline important?
A) important gristerlation for Ceristannians
B) new quasel for excess fevon and bracter
C) addresses the zionter lescelidge
(adapted from Judy Lanier)
Clicker question
Writing good peer instruction questions28
In your opinion, how many of these statements are true
about the traxoline peer instruction question?
o It builds on pre-existing knowledge.
o It nicely assesses the students’ current understanding.
o Difficult to discuss one choice over another.
o Superficial feedback to students about their current
understanding.
A) none B) one C) two D) three E) all four are true
Peer instruction helps teachers teach
Writing good peer instruction questions29
BEFORE DURING AFTER
setting up
instruction
developing
knowledge
assessing
learning
t h e l e a r n i n g c y c l e
t h e l e a r n i n g c y c l e
Peer instruction helps teachers teach
Writing good peer instruction questions30
BEFORE DURING AFTER
setting up
instruction
developing
knowledge
assessing
learning
Do they care about this?
Are they ready for the next topic?
What DO they care about, anyway?
What do they already know?
t h e l e a r n i n g c y c l eDid they notice key idea X?
Where are they in the activity?
Peer instruction helps teachers teach
Writing good peer instruction questions31
BEFORE DURING AFTER
setting up
instruction
developing
knowledge
assessing
learning
Are they getting it?
Do I need to intervene?
t h e l e a r n i n g c y c l eHow did I do?
Did they get it?
Peer instruction helps teachers teach
Writing good peer instruction questions32
BEFORE DURING AFTER
setting up
instruction
developing
knowledge
assessing
learning
Can I move to the next topic?
Did that activity work?
Writing good PI questions
Writing good peer instruction questions33
It’s critical to have
 content knowledge (the concepts)
 pedagogical content knowledge (how people learn
the concepts in your discipline and how to teach them)
Running effective PI
Writing good peer instruction questions34
It’s critical to
 teach the students how to engage in peer instruction
 choreograph each episode so students waste no
precious cognitive load wondering what to do
(call us for another workshop!)
You might not write the perfect question the first time so
 listen to the students’ conversations
 write your self some notes immediately after class
 revise and try it again next year
References
Writing good peer instruction questions35
1. National Research Council (2000). How People Learn: Brain,
Mind, Experience, and School: Expanded Edition. J.D.
Bransford, A.L Brown & R.R. Cocking (Eds.),Washington, DC:
The National Academies Press.
2. Bruff, D. (2009). Teaching with Classroom Response Systems.
San Francisco: Jossey-Bass.
3. Bain, K. (2004). What the best college teachers do.
Cambridge, MA: Harvard University Press.
What makes a good clicker question?
Writing good peer instruction questions
clarity Students should waste no effort trying to figure
out what’s being asked.
context Is this topic currently being covered
in class?
connection to
learning goals
Does the question make students do the right
thing to demonstrate they grasp the concept.
distractors What do the “wrong” answers tell you about
students’ thinking?
difficulty Is the question too trivial? too hard?
stimulates
thoughtful
discussion
Will the question engage the students and
spark thoughtful discussions?
Is there potential for you to be “agile”?
(Adapted from Stephanie Chasteen, CU Boulder)36

CTD Sp14 Weekly Workshop: Writing Good Peer Instruction ("Clicker") Questions

  • 1.
    PEER INSTRUCTION 1: WRITINGGOOD PEER INSTRUCTION QUESTIONS Peter Newbury Center for Teaching Development, University of California, San Diego pnewbury@ucsd.edu @polarisdotca ctd.ucsd.edu #ctducsd resources: ctd.ucsd.edu/programs/weekly-workshops-spring-2014/ Wednesday, May 7, 2014 12:00 – 12:50 pm Center Hall, Rm 316 please sign in
  • 2.
    2 Writing good peerinstruction questions
  • 3.
    Clicker Question (Economics) Writinggood peer instruction questions3 For which of the following professionals is driving an expensive car a sign of success compared to others in the same profession? A) a carpenter B) a realtor C) a politician D) a major league baseball player (adapted from Steve Morris, UCSD)
  • 4.
    Typical episode ofpeer instruction Writing good peer instruction questions4 1. Instructor poses a conceptually-challenging multiple-choice question. 2. Students think about question on their own and vote using clickers, colored ABCD cards, smartphones,… 3. The instructor prompts students, “Turn to your neighbors and convince them you’re right.” 4. After the peer-to-peer discussion, [the students vote again and] the instructor leads a class-wide discussion concluding with why the right answer(s) is right and the wrong answers are wrong.
  • 5.
    In effective peerinstruction Writing good peer instruction questions5  students teach each other while they may still hold or remember their novice preconceptions  students discuss the concepts in their own (novice) language  each student finds out what s/he does(n’t) know  the instructor finds out what the students (don’t) know and reacts, building on their initial understanding and preconceptions. students learn and practice how to think, communicate like experts
  • 6.
    Effective peer instructionrequires Writing good peer instruction questions6 1. identifying key concepts, misconceptions 2. creating multiple-choice questions that require deeper thinking and learning 3. facilitating peer instruction episodes that spark and support student discussion 4. leading a class-wide discussion to clarify the concept, resolve the misconception 5. reflecting on the question: note curious things you overheard, how they voted, etc. before class during class after class today next week
  • 7.
    What makes agood clicker question? Writing good peer instruction questions7 clarity Students should waste no effort trying to figure out what’s being asked. context Is this topic currently being covered in class? connection to learning goals Does the question make students do the right thing to demonstrate they grasp the concept. distractors What do the “wrong” answers tell you about students’ thinking? difficulty Is the question too trivial? too hard? stimulates thoughtful discussion Will the question engage the students and spark thoughtful discussions? Is there potential for you to be “agile”? (Adapted from Stephanie Chasteen, CU Boulder)
  • 8.
    t h el e a r n i n g c y c l e Peer instruction helps students learn... Writing good peer instruction questions8 BEFORE DURING AFTER setting up instruction developing knowledge assessing learning Adapted from Rosie Piller, Ian Beatty, Stephanie Chasteen
  • 9.
    Clicker question Writing goodpeer instruction questions9 Melt chocolate over low heat. Remove the chocolate from the heat. What will happen to the chocolate? A) It will condense. B) It will evaporate. C) It will freeze. (Question: Sujatha Raghu from Braincandy via LearningCatalytics) (Image: CIM9926 by number657 on flickr CC) assess prior knowledge
  • 10.
    Clicker question Writing goodpeer instruction questions10 In your opinion, which had the most positive impact on the modern world? Be prepared to defend your choice with evidence from the readings. A) coffee B) tea C) chocolate D) spice E) sugar (Herbst, UCSD) provoke thinking
  • 11.
    Clicker question Writing goodpeer instruction questions11 A ball is rolling around the inside of a circular track. The ball leaves the track at point P. Which path does the ball follow? P A B C E D (adapted from Mazur) predict
  • 12.
    t h el e a r n i n g c y c l e Peer instruction helps students learn... Writing good peer instruction questions12 BEFORE DURING AFTER setting up instruction developing knowledge assessing learning Adapted from Rosie Piller, Ian Beatty, Stephanie Chasteen The students have not solved concept X. But they’re know X exists and why X is interesting.
  • 13.
    Clicker question Writing goodpeer instruction questions13 Which of these are reasons for the seasons? i. the height of the Sun in the sky during the day ii. Earth’s distance from the Sun iii. how many hours the Sun is up each day A) ii only B) iii only C) i and ii D) i and iii E) i, ii and iii clarity context learning goals distractors difficulty discussion probe misconception
  • 14.
    Clicker question Writing goodpeer instruction questions14 How many of these are reasons for the seasons? height: the height of the Sun in the sky during the day distance: Earth’s distance from the Sun hours: how many hours the Sun is up each day A) none of them B) one C) two D) all three probe misconception
  • 15.
    Clicker question Writing goodpeer instruction questions15 Select the line that you feel has the strongest imagery in “Fast rode the knight” by Stephen Crane (1905). analysis Fast rode the knight With spurs, hot and reeking, Ever waving an eager sword, "To save my lady!" Fast rode the knight, And leaped from saddle to war. Men of steel flickered and gleamed Like riot of silver lights, And the gold of the knight's good banner Still waved on a castle wall. . . . . . A horse, Blowing, staggering, bloody thing, Forgotten at foot of castle wall. A horse Dead at foot of castle wall. A B C D E (David Kurtz, via LearningCatalytics)
  • 16.
    Clicker question Writing goodpeer instruction questions16 Evaluate: A) B) (adapted from Bruff (2009))   4 0 32 1 dxxx 23 )65(16 9 16 C. D. )165( 9 2 23  3 1022 clarity context learning goals distractors difficulty discussion exercise skill
  • 17.
    Clicker question Writing goodpeer instruction questions17 Which of the following is an incorrect step when using the substitution method to evaluate the definite integral A) B)   4 0 32 1 dxxx 3 1 xu  dxx du 2 3  C. D. none of the above  4 03 1 duu (adapted from Bruff (2009)) exercise skill
  • 18.
    Clicker question Writing goodpeer instruction questions18 According to Augustine, which do you think was the most significant event in his conversion to Christianity? Be prepared to defend your choice with evidence from the readings. A) Corporal punishments as a school-boy B) Student years of follies (studies and promiscuity) C) Stealing the pears from the neighbor’s tree D) Death of his classmate/friend exercise skill analysis Heidi Keller-Lapp (UCSD)
  • 19.
    Clicker question Writing goodpeer instruction questions19 Susan throws a ball straight up into the air. It goes up and then falls back into her hand 2 seconds later. Draw a graph showing the velocity of the ball from the moment it leaves her hand until she catches it again. time velocity 2 sec0 exercise skill (CWSEI UBC)
  • 20.
    time velocity 2 sec0 A time velocity 2 sec0 B time velocity 2sec0 C time velocity 2 sec0 D E) some other graph Which one is the closest match to your graph? exercise skill (CWSEI UBC)Writing good peer instruction questions20
  • 21.
    t h el e a r n i n g c y c l e Peer instruction helps students learn... Writing good peer instruction questions21 BEFORE DURING AFTER setting up instruction developing knowledge assessing learning Adapted from Rosie Piller, Ian Beatty, Stephanie Chasteen Students have had opportunities to try, fail, receive feedback and try again without facing a summative evaluation. [3]
  • 22.
    Writing good peerinstruction questions22 Clicker question Are features X and Y ridges or valleys? A) X=ridge, Y=valley B) X=valley, Y=ridge C) both are ridges D) both are valleys X Y (EOSC / CWSEI, UBC) demonstrate success
  • 23.
    Clicker question Writing goodpeer instruction questions23 What are the mean and median of these numbers? 74, 32, 35, 87, 28, 36, 11, 26, 93, 56, 34, 52, 8 A) 13, 36 B) 44, 35 C) 572, 93 D) 48,13 (terribly adapted from Peck, mathquest.carroll.edu/resources.html) review / recap clarity context learning goals distractors difficulty discussion
  • 24.
    Clicker question Writing goodpeer instruction questions24 For the data given below, which is larger, the mean or the median? 74, 32, 35, 87, 28, 36, 11, 26, 93, 56, 34, 52, 8 A) mean B) median (poorly adapted from Peck, mathquest.carroll.edu/resources.html) review / recap clarity context learning goals distractors difficulty discussion
  • 25.
    Clicker question Writing goodpeer instruction questions25 For the data set displayed in the following histogram, which would be larger, the mean or the median? A) mean B) median C) can’t tell from the given histogram (adapted from Peck, mathquest.carroll.edu/resources.html) review / recap
  • 26.
    Clicker question Writing goodpeer instruction questions26 In your opinion, which had the most positive impact on the modern world? A) coffee B) tea C) chocolate D) spice E) sugar “big picture” (Herbst, UCSD)
  • 27.
    Clicker question Writing goodpeer instruction questions27 Traxoline is a new form of zionter. It is monotilled in Ceristanna. The Ceristannians gristerlate large amounts of fevon and then bracter it to quasel traxoline. Traxoline may well be one of our most lukised snezlaus in the future because of our zionter lescelidge. Why is traxoline important? A) important gristerlation for Ceristannians B) new quasel for excess fevon and bracter C) addresses the zionter lescelidge (adapted from Judy Lanier)
  • 28.
    Clicker question Writing goodpeer instruction questions28 In your opinion, how many of these statements are true about the traxoline peer instruction question? o It builds on pre-existing knowledge. o It nicely assesses the students’ current understanding. o Difficult to discuss one choice over another. o Superficial feedback to students about their current understanding. A) none B) one C) two D) three E) all four are true
  • 29.
    Peer instruction helpsteachers teach Writing good peer instruction questions29 BEFORE DURING AFTER setting up instruction developing knowledge assessing learning t h e l e a r n i n g c y c l e
  • 30.
    t h el e a r n i n g c y c l e Peer instruction helps teachers teach Writing good peer instruction questions30 BEFORE DURING AFTER setting up instruction developing knowledge assessing learning Do they care about this? Are they ready for the next topic? What DO they care about, anyway? What do they already know?
  • 31.
    t h el e a r n i n g c y c l eDid they notice key idea X? Where are they in the activity? Peer instruction helps teachers teach Writing good peer instruction questions31 BEFORE DURING AFTER setting up instruction developing knowledge assessing learning Are they getting it? Do I need to intervene?
  • 32.
    t h el e a r n i n g c y c l eHow did I do? Did they get it? Peer instruction helps teachers teach Writing good peer instruction questions32 BEFORE DURING AFTER setting up instruction developing knowledge assessing learning Can I move to the next topic? Did that activity work?
  • 33.
    Writing good PIquestions Writing good peer instruction questions33 It’s critical to have  content knowledge (the concepts)  pedagogical content knowledge (how people learn the concepts in your discipline and how to teach them)
  • 34.
    Running effective PI Writinggood peer instruction questions34 It’s critical to  teach the students how to engage in peer instruction  choreograph each episode so students waste no precious cognitive load wondering what to do (call us for another workshop!) You might not write the perfect question the first time so  listen to the students’ conversations  write your self some notes immediately after class  revise and try it again next year
  • 35.
    References Writing good peerinstruction questions35 1. National Research Council (2000). How People Learn: Brain, Mind, Experience, and School: Expanded Edition. J.D. Bransford, A.L Brown & R.R. Cocking (Eds.),Washington, DC: The National Academies Press. 2. Bruff, D. (2009). Teaching with Classroom Response Systems. San Francisco: Jossey-Bass. 3. Bain, K. (2004). What the best college teachers do. Cambridge, MA: Harvard University Press.
  • 36.
    What makes agood clicker question? Writing good peer instruction questions clarity Students should waste no effort trying to figure out what’s being asked. context Is this topic currently being covered in class? connection to learning goals Does the question make students do the right thing to demonstrate they grasp the concept. distractors What do the “wrong” answers tell you about students’ thinking? difficulty Is the question too trivial? too hard? stimulates thoughtful discussion Will the question engage the students and spark thoughtful discussions? Is there potential for you to be “agile”? (Adapted from Stephanie Chasteen, CU Boulder)36