The study presented in this paper is written from a report maker’s perspective, primarily asking: How can the design of score reports support the utility of achievement data by personnel in the school system? Thus, the focus is on understanding how score reports can be designed optimally to enhance their use by the educational community. Before attempting to address this issue, we as a community need to understand how educators interpret reports. This paper is an initial step in this process of enhancing score reports by investigating the thought process of educators. It proposes a conceptual model, “Data Habit of Mind,” to study educators’ understanding, interpretation and potential applications of results from large-scale assessments.

1. Investigating Educators’
Statistical Literacy and
Score Report Interpretation
Saad Chahine, PhD
April 30, 2013
NCME | San Francisco
13-04-30

2. Direction
• Need for research on how educators read and
interpret score reports
• Current research focuses on the culture of data use in
schools
• This research study asks the question:
How can the design score reports support the utility of
achievement data by personnel in the school system?

3. Research Questions
1. How are educators proficient in
statistical literacy?
2. How do educators interpret and
propose to use test score
reports?
π2

4. Statistical Literacy
Describing Organizing &
Reducing
Representing Analyzing &
Interpreting
Statistical Literacy Skills
Statistical Literacy
ThinkingLevels
Analytical
Quantitative
Transitional
Idiosyncratic

5. Score Report Interpretation
Describing Summarizing Questioning Proposed
Application
Score Report Interpretation Skills
Score Report Interpretation
ThinkingLevels
Analytical
Quantitative
Transitional
Idiosyncratic

6. Conceptual Framework
“habit of mind” as a metaphor
Statistical Literacy
Data
Habit of Mind
ScoreReportInterpretation

7. Methods
• Cognitive interviews with 20
educators
• Statistical literacy tasks
• Two mock score reports
• Six stage protocol analysis
π6

8. π7
Example
Statistical
Literacy Task

9. Coopers Score Report π8

10. Jabberwocky Score Report π9

11. 13-04-30 Mount Saint Vincent University 10
Skill Minimum Maximum Mean Std. D.
Statistical Literacy
Describing 3.00 4.00 3.77 0.24
Organizing and Reducing 2.00 4.00 2.89 0.65
Representing 1.67 4.00 3.33 0.65
Analyzing and Interpreting 2.00 4.00 3.38 0.53
Score Report Interpretation: Cooper’s Fitness Test Score Report
Describing 4.00 4.00 4.00 0.00
Summarizing 3.00 4.00 3.75 0.44
Questioning 1.00 4.00 2.10 0.09
Proposed Application 1.00 4.00 2.45 1.10
Score Report Interpretation: Jabberwocky Test Score Report
Describing 4.00 4.00 4.00 0.00
Summarizing 3.00 4.00 3.75 0.44
Questioning 1.00 4.00 1.95 0.89
Proposed Application 1.00 4.00 2.00 0.97

12. π11

13. Discussion P1
• Data Habit of Mind represents a
starting point
• Educators on average appear to
have well developed statistical
literacy skills
• Educators on average appear to
struggle on score report
interpretation skills: Questioning &
Proposing an Application
π12

14. Discussion P2
• We can begin to identify clusters
and patterns
• Potentially there are four ways in
which we can cluster the
patterns that we see when it
comes to data habit of mind
π13

15. 4-Way Data Habit of Mind
π14
MountSaintVincentUniversity
Statistical Literacy
Idiosyncratic Transitional Quantitative Analytical
ScoreReportInterpretation
Idiosyncratic Quadrant 1:
Lower Statistical Literacy
&
Lower Score Report
Interpretation
Quadrant 3:
Higher Statistical Literacy
&
Lower Score Report
Interpretation
Transitional
Quantitative Quadrant 2:
Lower Statistical Literacy
&
Higher Score Report
Interpretation
Quadrant 4:
Higher Statistical Literacy
&
Higher Score Report
Interpretation
Analytical

16. Thank You
saad.chahine@msvu.ca
in@saadchahine
@saadchahine
π15