1. Reading Item Development
Juliette Mendelovits
Australian Council for Educational Research
Marian Sainsbury
N ti l F d ti f Ed ti l R h UKNational Foundation for Educational Research, UK

2. Test development
process for a large scaleprocess for a large‐scale
assessment
Framework
development
Initial item writing Item developer Cognitive laboratoriesInitial item writing p
review
Cog t e abo ato es
Item revision
External review
Item ready for trialItem ready for trial
(or rejected)
Trial testing
Psychometric and
item developer
i
Final
instrument reviewinstrument

3. Reading itemsReading items
Features of
Reading skills
/ /
Features of
the text
/ aspects /
processes
QuestionsQuestions
and
answersanswers

4. The importance of text selection

5. A good textA good text …
i i h d i t ti• is rich and interesting
• is optimally challenging – not too hard
or too easy
• does not pose artificial challengesdoes not pose artificial challenges
• is (more or less) equally accessible and
equitable for different candidatesequitable for different candidates
• offers opportunity to pose searching
iquestions

6. Sample Text from PISA 2000 and 2009
MacondoMacondo
Dazzled by so many and such marvellous inventions, the people of Macondo
did not know where their amazement began They stayed up all night lookingdid not know where their amazement began. They stayed up all night looking
at the pale electric bulbs fed by the plant that Aureliano Triste had brought
back when the train made its second trip, and it took time and effort for them
to grow accustomed to its obsessive toom‐toom. They became indignant over
That’s crazy ... But is there some to grow accustomed to its obsessive toom toom. They became indignant over
the living images that the prosperous merchant Don Bruno Crespi projected
in the theatre with the lion‐head ticket windows, for a character who had
died and was buried in one film, and for whose misfortune tears of affliction
An intriguing opening sentence ...
What inventions???
truth in it?
had been shed, would reappear alive and transformed into an Arab in the
next one. The audience, who paid two centavos apiece to share the
difficulties of the actors, would not tolerate that outlandish fraud and they
broke up the seats. The mayor, at the urging of Don Bruno Crespi, explained
by means of a proclamation that the cinema was a machine of illusions that
did not merit the emotional outburst of the audience. With that discouraging
explanation many felt that they had been the victims of some new and showy
gypsy business and they decided not to return to the movies, considering that
they already had too many troubles of their own to weep over the acted‐out
i f f i i b i
What’s this about? ... A familiar
thing from an unfamiliar
timisfortunes of imaginary beings.perspective

7. A non‐continuous text (original version)
Why are some of the stations
marked with big circles?
How would you
get from x to y?
marked with big circles?
Why does this blue line look
different to all the other lines?

8. A non‐continuous text (modified for PISA)

9. Bad texts: what to avoid
• Giving offence or creating disturbance:
T ( id t i l )– Trauma (car accidents, violence)
– Sex, religion, politics or other emotionally contentious
issues
– Nasty behaviour, violence, racism, immorality or
irresponsibility
Undesirable models drugs alcohol inducements to do– Undesirable models – drugs, alcohol, inducements to do
anything potentially dangerous
– Bad language
• Expecting too much or too little of students
– Texts that rely on esoteric knowledge
– Texts that provide general knowledge that most
students can be expected to have already
– Texts that are boring!!!Texts that are boring!!!

10. Text maps:
analysing the features of the text
Stories (PIRLS literary)
• ThemeTheme
• Main events
• Main characters
• SettingSetting
• Language features

11. Example: Fly Eagle FlyExample: Fly, Eagle, Fly
h f d i f b i• Theme: freedom, importance of being true to
one’s nature, fulfilling one’s potential
• Main events: farmer finds eagle chick; raises it
as a chicken; friend argues for eagle’s true ; f g f g
nature; eagle is finally free to fly
• Main characters: farmer friend eagleMain characters: farmer, friend, eagle
• Setting: Africa
• Language features: ‘you belong not to the
earth but to the sky’; description of dawn

12. Text maps:
analysing the features of the text
Non‐fiction (PIRLS informational)
• PurposePurpose
• Theme
• Content
• StructureStructure
• Presentational features
• Language features

13. Example: The Giant Tooth MysteryExample: The Giant Tooth Mystery
i i f i f i• Purpose: exposition of information
• Theme: the development of ideas in the history of
science
• Content: how people came to understand that p p
fossils were the remains of extinct creatures
• StructureStructure
• Presentational features: continuous prose;
illustrations and diagrams with captionsillustrations and diagrams with captions
• Language features: definition of ‘fossil’

14. StructureStructure
Introduction: definition of fossil
Long ago People thought fossils were the bones of giantsg g p g g
Hundreds of years ago Palissey was imprisoned for suggesting extinct creatures
I th 1820 Wif f Gid M t ll f d i t t thIn the 1820s Wife of Gideon Mantell found a giant tooth
Observation 1 Observation 2 Observation 3
Tooth worn down
A plant eater
Age of rocks
A reptile
Reptiles do not chew food
But iguanas do!
A giant extinct reptile ‐ Iguanodon
Today We call them dinosaurs

15. Reading itemsReading items
Features of
Reading skills
/ /
Features of
the text
/ aspects /
processes
QuestionsQuestions
and
answersanswers

16. The importance of itemThe importance of item
construction

17. What makes a good item?What makes a good item?
fl f h f k• Reflection of the framework
• Centrality to the text• Centrality to the text
• TransparencyTransparency
• Equityq y
• Appropriate item format

18. Multiple choice itemsMultiple choice items
Stem
Key
Distracters
Options

19. Multiple choice itemsMultiple choice items
St h ld b l t d t h ld b bl t• Stem should be clear – students should be able to
answer question without reading options
• Do not use negatively worded stems• Do not use negatively‐worded stems
• Only one of the response options should be the
correct answercorrect answer
• Distracters should be plausible based on likely
student errors or misconceptionsstudent errors or misconceptions
• Make sure each option is about the same length,
and that the grammar of each option “fits” theand that the grammar of each option fits the
stem

20. Constructed response itemsConstructed response items
• Used when:
– The answer is straightforward and the text does g
not offer plausible distracters or
– Students are required to show a developedStudents are required to show a developed
understanding drawing on different parts of the
texttext
• Give clear expectations to students with
i i lik “ l i ” “ i 3 ”instructions like “explain” or “give 3 reasons”

21. Developing coding guidesDeveloping coding guides
• Begin by writing a full‐credit answer to the
questionq
• Define the levels of partial credit (if
applicable) keeping in mind the completenessapplicable), keeping in mind the completeness
and accuracy of the response
• Provide examples of partial credit responses

22. Refining the coding guide for PIRLSRefining the coding guide for PIRLS
• Full‐credit answers and partial credit responses
defined at the time the items are written
• But 4th grade students express their answers in
unpredictable ways which do not always fit theunpredictable ways which do not always fit the
guide
Aft i iti l il t i f l d i• After an initial pilot in a few classes, and again
after the field test, we use many examples of
d ’ fi h id dstudents’ answers to refine the guides and
develop training materials

23. You learn what the farmer’s friend was like from the
things he did.
Describe what the friend was like and give an example of
h h did h h hi
2 – Complete Comprehension
what he did that shows this.
The response describes one plausible character trait (persistent, stubborn, nice,
clever, friendly to animals, etc.). In addition, the response provides one example of
the farmer’s friend’s actions that are evidence of the character trait.
H d t i d H k t t i t t h th l t flHe was determined. He kept trying to teach the eagle to fly.
He was kind to animals. He wanted the eagle to be free.
1 Partial Comprehension1 – Partial Comprehension
The response provides one plausible character trait.
Or, the response provides one example of the friend’s actions that are evidence of
the friend’s characterthe friend s character.
He is kind to animals.
He takes the eagle to see the sun and fly away never to live among the
chickenschickens.

24. Some student responsesSome student responses …
• He want to put things right. He want the eagle to fly as normal
eagles do and he don’t want it to be a chicken.
• He makes the eagle be a chicken
• I have learn that the eagle is cruel• I have learn that the eagle is cruel
• He was a good friend he was always having a laugh with you
• What things belong to the sky belong to the sky. The farmersg g y g y
friend told that the eagle belong to the sky not the earth
• “Let nature choose its path” person. Wanted the eagle to be an
eagleeagle
• I’ve learn that animals should have their own family like us
• He showed confidence he showed strength because he searchedg
after a storm for his calf and he never gave up
• He liked to prove things

25. CodedCoded …
• He was kind. He freed the eagle and let it become what it was
• He want to put things right. He want the eagle to fly as normal
eagles do and he don’t want it to be a chicken.
• “Let nature choose its path” person. Wanted the eagle to be anp p g
eagle
• He liked to prove things
• What things belong to the sky belong to the sky The farmers• What things belong to the sky belong to the sky. The farmers
friend told that the eagle belong to the sky not the earth
• He makes the eagle be a chicken
I h l th t th l i l• I have learn that the eagle is cruel
• He was a good friend he was always having a laugh with you
• I’ve learn that animals should have their own family like usy
• He showed confidence he showed strength because he searched
after a storm for his calf and he never gave up

26. The importance of unitThe importance of unit
construction
A unit is a set of items built around aA unit is a set of items built around a
piece of stimulus (or related pieces of
stimulus)stimulus)

27. What makes a good unit?What makes a good unit?
• Items cover a range of difficulty
• Items cover a range of framework featuresItems cover a range of framework features
• Each item is independent of all the others
• Items as a group deal with the “heart” of the
text

28. THANK YOU!THANK YOU!