1. KRISTIN CONRADI
NOVEMBER 30, 2012
Kristin_conradi@ncsu.edu
Examining Item Validity Using
Cognitive Interviews

2. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
“How many times did you visit a
doctor in the last year?”
What do you mean
by last year?
What does “visit a
doctor” mean?

3. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
“How do you feel about reading
a book before you go to bed?”
Hate it
It’s
okay
Love it

4. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
“How do you feel about reading
graphic novels?”
What do you mean
by graphic novels?

5. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Purpose: Present a rationale for conducting
cognitive interviews in reading research.
Cognitive interviews are a technique
researchers can employ to evaluate how
participants understand questions screening
for potential problems with interpretation
(Beatty & Willis, 2007; Conrad & Blair, 2004; Desimone & Floch, 2004; Presser et al.,
2004; Willis, 2005; Willis, Royston, & Bercini, 1991).

6. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Why do cognitive
interviews matter?

7. (Wiederhold & Block, Gray Silent Reading Test Story 1)
BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
The boy is not big.
But he helps Mother and Father.
He puts his toys up after he plays with them.
He tells the baby a funny story if she begins to cry.
He sings to make her go to sleep.
Why do you think the boy likes to help?
a) It makes him feel good
b) Mother and Father give him money.
c) Father is too sick to work.
d) Mother makes him do it.

8. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

9. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
• Cognitive Interviews
• Reading Motivation Instruments

10. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
What are cognitive interviews?

11. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
The standardized
approach creates
a communication
straightjacket.
(Gordon B. Willis, 2012)

12. Think-
Aloud Probing
BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

13. Types of Probes Examples
Comprehension/Interpretation Probe What does the term “outpatient” mean
to you?
Paraphrasing Can you repeat the question I just asked
in your own words?
Confidence Judgment How sure are you that your health
insurance covers drug and alcohol
treatment?
Recall Probe How do you remember that you went to
the doctor five times in the past 12
months?
Specific Probe Why do you think that cancer is the most
serious health problem?
General Probes How did you arrive at that answer? Was
that easy or hard to answer? I noticed
that you hesitated. Tell me what you
were thinking.
BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

14. Reading Self-
Concept Scale
(1995)
Motivation to
Read Profile
(1996)
Elementary
Reading Attitude
Survey (1990)
BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

15. Sample Questions
Are the books
you read in class
too hard?
My best friends
think reading is. .
.
How do you feel
about reading on
a rainy Saturday?
BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

16. Uses in Literacy
• Survey of Adolescent Reading Attitudes
(McKenna, Conradi, Lawrence, Jang, & Meyer,
2012)
BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

17. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
The Present Study
Are there any terminology/phrasing issues
in reading motivation measures that
threaten instrument validity?

18. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
DATA SOURCES
• Test administrators’ feedback
• Respondents’ notes on surveys
• Histograms

19. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

20. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

21. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
PROCEDURE
Data sources
were analyzed
to identify
candidate
items
Probing
techniques
were decided
upon
One-on-one
interviews
were
conducted

22. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Subsample (N=16) of
larger study (N=491)
 Elementary Reading
Attitude Survey
 Motivation to Read
Profile
 Reading Self-Concept
Scale
PARTICIPANTS & MEASURES

23. ANALYSIS
Item level
response forms
3 reviewers
read them and
coded for issues
Response forms
were read again
for supporting
or refuting
examples
Written
summaries
were created
BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION

24. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
ANALYSIS

25. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
RESULTS
Terminological Situational
Logical

26. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
EXAMPLES OF PROBLEM-TYPES
Do you like playing
word games in
class? (RSCS)
Situational
Logical

27. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
EXAMPLES OF PROBLEM-TYPES
Terminological
When someone
gives me a book
for a present, I feel
(MRP)
Logical

28. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
EXAMPLES OF PROBLEM-TYPES
Terminological Situational
I read
• Not as well as my friends
• About the same as my friends
• A little better than my friends
• A lot better than my friends

29. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
When I grow up, I will
spend. . .
A grown-up can do
whatever they want!
Will probably read a
lot because I read a lot
now
I will do more fun
things instead of
reading
A grown-up will have a
job and responsibilities
Maybe the job won’t
require reading, so I
won’t spend time
reading
Maybe the job will
require reading, so I
will have to read.
Because I will be
busy, I won’t have time
to read
Even if the job keeps
me busy, I will still read
and be sleep deprived

30. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
DISCUSSION & IMPLICATIONS
• Reading motivation measures
• Other reading measures: particularly as what
we mean by literacy evolves (Pitcher et al.
2007)

31. BACKGROUND &
RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
kristin_conradi@ncsu.edu