This document discusses using data wisely from a superintendent's perspective. It covers three main topics: assessment basics, improving assessment programs, and developing a data culture. The document emphasizes that what is measured gets attended to, so assessments must be properly aligned and designed. It also stresses using multiple years of data to provide context and control for outside factors to fairly evaluate teachers. Developing the right assessment systems and using data thoughtfully can significantly improve student achievement.

1. From a
Superintendent’s
Perspective:
Using data wisely
Andy Hegedus, Ed.D.
September 2014

2. Trying to gauge my audience
and adjust my speed . . .
• How many of you think your literacy with
assessments is “Good” or better?
• How many of you have a fine tuned
assessment program?
• How many of you think your practical
knowledge about using data for systemic
improvement is “Good” or better?

3. The Bottom Line First:
Here’s what you must have
• An adequate depth of knowledge to ask really
good questions and the tenacity to do so
• A person willing to “Speak the Truth, With
Love, To Power”
• A big lever –
“What gets measured (and attended to),
gets done”
The measurement and reinforcement system
is your responsibility

4. My Purpose
• Increase your understanding about
various urgent assessment related topics
– Ask better questions
– Useful for making all types of decisions with
data

5. Three main topics
• Assessment basics +
• Improving your assessment program
• Data culture

6. Go forth thoughtfully
with care
• What we’ve known to be true is now being
shown to be true
– Using data thoughtfully improves student
achievement and growth rates
– 12% mathematics, 13% reading
• There are dangers present however
– Unintended Consequences
Slotnik, W. J. , Smith, M. D., It’s more than money, February 2013, retrieved from
http://www.ctacusa.com/PDFs/MoreThanMoney-report.pdf

7. Remember the old adage?
“What gets measured (and attended to),
gets done”

8. • NCLB
An infamous example
– Cast light on inequities
– Improved performance of “Bubble Kids”
– Narrowed taught curriculum
The same dynamic happens
inside your schools

9. A patient’s health
doesn’t change
because we know
their blood pressure
It’s our response that
makes all the
difference
It’s what we do that counts

10. Be considerate of the continuum
of stakes involved
Support
Compensate
Terminate
Increasing levels of required rigor
Increasing risk

11. Assessment basics
(in a teacher evaluation frame)

12. Three primary conditions
1. Alignment between the content assessed and
the content to be taught
2. Selection of an appropriate assessment
• Used for the purpose for which it was designed
(proficiency vs. growth)
• Can accurately measure the knowledge of all
students
• Adequate sensitivity to growth
3. Adjust for context/control for factors outside a
teacher’s direct control (value-added)

13. Two approaches we like
1. Assessment results used
wisely as part of a
dialogue to help teachers
set and meet challenging
goals
2. Use of tests as a “yellow
light” to identify teachers
who may be in need of
additional support or are
ready for more

14. What question is being answered in support of
using data in evaluating teachers?
Is the progress produced
by this teacher
dramatically different
than teaching peers who
deliver instruction to
comparable students in
comparable situations?

15. Marcus’ growth
College readiness standard
Marcus Normal Growth Needed Growth

16. There are four key steps required to
The Test
answer this question
Top-Down Model
The Growth Metric
The Evaluation
The Rating

17. Let’s begin at the beginning
The Test
The Growth Metric
The Evaluation
The Rating

18. What is measured should be
aligned to what is to be taught
3rd Grade
ELA
Teacher?
3rd Grade
ELA
Standards
3rd Grade
Social
Studies
Teacher?
1. Answer questions to demonstrate
understanding of text….
2. Determine the main idea of a
Elem. Art
Teacher?
text….
3. Determine the meaning of general
academic and domain specific
words…
Would you use a general
reading assessment in the
evaluation of a….
~30% of teachers teach in tested subjects and grades
The Other 69 Percent: Fairly Rewarding the Performance of Teachers of Nontested Subjects and Grades,
http://www.cecr.ed.gov/guides/other69Percent.pdf

19. What is measured should be
aligned to what is to be taught
• Assessments should align with the
teacher’s instructional responsibility
– Specific advanced content
• HS teachers teaching discipline specific content
– Especially 11th and 12th grade
• MS teachers teaching HS content to advanced students
– Non-tested subjects
• School-wide results are more likely “professional
responsibility” rather than reflecting competence
– HS teachers providing remedial services

20. The purpose and design of the
instrument is significant
• Many assessments are
not designed to
measure growth
• Others do not measure
growth equally well for
all students

21. Let’s ensure we have similar
meaning
Adult
Reading
5th Grade
x
Beginning Literacy
x
Time 1 Time 2
Status
Two assumptions:
1. Measurement accuracy,
and
2. Vertical interval scale

22. Accurately measuring
growth
depends on
accurately measuring
achievement

23. What does it take to accurately
measure achievement?
Questions
surrounding the
student’s
achievement level
The more
questions the
merrier

24. Teachers encounter a distribution
of student performance
Adult
Reading
5th
Grade
x
x
x
x
x x x
x
x
x
x
x
x
x
x
Beginning Literacy
Grade Level
Performance

25. Adaptive testing works differently
Item bank can
span full
range of
achievement

26. How about accurately
measuring height?
What if the yardstick
stopped in the middle of
his back?

27. Items available need to match student
ability
California STAR NWEA MAP

28. How about accurately
measuring height?
What if we could only
mark within a pre-defined
six inch range?

29. 5th Grade
Level Items
These differences impact
measurement error
.12
.10
.08
.06
.04
.02
.00
Information
190 200 210 220 230 240 260
Scale Score
Fully
Adaptive Test
Significantly
Different Error
250
Constrained
Adaptive or
Paper/Pencil
Test

30. To determine growth,
achievement
measurements
must be related through
a scale

31. Let’s measure height again
If I was measured as:
5’ 9”
And a year later I was:
1.82m
Did I grow?
Yes. ~ 2.5”
How do you know?

32. Traditional assessment uses items
reflecting the grade level standards
Adult
Reading
6th Grade
5th Grade
4th Grade
Grade Level Standards
Beginning Literacy
Traditional
Assessment Item Bank

33. Traditional assessment uses items
reflecting the grade level standards
Adult
Reading
6th Grade
5th Grade
4th Grade
Grade Level Standards
Grade Level Standards
Beginning Literacy
Overlap allows
linking and scale
construction
Grade Level Standards

34. Error can change your life!
• Think of a high stakes test –
State Summative
– Designed mainly to identify if a student is proficient or
not
• Do they do that well?
• 93% correct on Proficiency determination
• Does it go off design well?
• 75% correct on Performance Levels determination
*Testing: Not an Exact Science, Education Policy Brief, Delaware Education Research & Development Center, May 2004,
http://dspace.udel.edu:8080/dspace/handle/19716/244

35. Using tests in high stakes ways
creates new dynamic
• Tests specifically designed to inform classroom
instruction and school improvement in
formative ways
No incentive in the system for
inaccurate data

36. 8.00
6.00
4.00
2.00
0.00
-2.00
-4.00
-6.00
New phenomenon when used as part of
a compensation program
Mean value-added growth by school
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71
Students taking 10+ minutes longer spring than fall All other students

37. Other consequence
When teachers are evaluated
on growth using a once per
year assessment, one teacher
who cheats disadvantages the
next teacher

38. Lessons?
• What were some things you learned?
• What practices do you want to reinforce?
• What do you need to do differently?
• Think – Pair
– 2 min to make some notes
– 3 min to share with a neighbor

39. Testing is complete . . .
What is useful to answer our question?
The Test
The Growth Metric
The Evaluation
The Rating

40. The problem with spring-spring
testing
Teacher 1 Summer Teacher 2
4/14 5/14 6/14 7/14 8/14 9/14 10/14 11/14 12/14 1/15 2/15 3/15 4/15

41. A better approach
• When possible use a spring – fall – spring
approach
• Measure summer loss and incentivize schools
and teachers to minimize it
• Measure teacher performance fall to spring,
giving as much instructional time as possible
between assessments
• Monitor testing conditions to minimize
gaming of fall or spring results

42. 100
90
80
70
60
50
40
30
20
10
0
Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Reading
Math
The metric matters -
Let’s go underneath “Proficiency”
Difficulty of New York Proficient Cut Score
National Percentile
College
Readiness
New York Linking Study: A Study of the Alignment of the NWEA RIT Scale with the New York State (NYS) Testing Program, November 2013

43. The metric matters -
Let’s go underneath “Proficiency”
Dahlin, M. and Durant, S., The State of Proficiency, Kingsbury Center at NWEA, July 2011

44. What actually happened?
Estimated Proficiency Rates For Six NY Districts
With Proficiency Cut Scores Changed
87
4th Grade Mathematics
55
100
80
60
40
20
0
-20
-40 -31
2012 2013 Reported Change

45. What actually happened?
Estimated Proficiency Rates
For Six NY Districts
4th Grade Mathematics
With Proficiency Cut Scores
87
Changed
55
100
80
60
40
20
0
-20
-40 -31
2012 2013 Reported Change
46
55
10
60
50
40
30
20
10
0
Estimated Proficiency Rates
For Six NY Districts
4th Grade Mathematics
With 2013 Proficiency Cut
Scores Applied
2012 2013 Actual Change

46. Number of Students
What gets measured and attended to
Mathematics
Fall RIT
No Change
Down
Up
really does matter
Proficiency College Readiness
One district’s change in 5th grade mathematics performance
relative to the KY proficiency cut scores

47. Number of Students
Changing from Proficiency to Growth
means all kids matter
Mathematics
Student’s score in fall
Below projected
growth
Met or above
projected growth
Number of 5th grade students meeting projected
mathematics growth in the same district

48. Lessons?
• What were some things you learned?
• What practices do you want to reinforce?
• What do you need to do differently?
• Think – Pair – Share
– 2 min to make some notes
– 3 min to share with a neighbor

49. How can we make it fair?
The Test
The Growth Metric
The Evaluation
The Rating

50. Without context what is
“Good”?
Scale Norms Study
Adult
Literacy
Beginning
Reading
National
Percentile
College Readiness
ACT
Benchmarks
State Test
Performance Levels
“Meets”
Proficiency
Common
Core
Proficient
Performance Levels

51. Normative data for growth is a
bit different
Fall
Score
Basic
Factors
Grade: 5th
Subject:
Reading
7
points
FRL vs. non-FRL?
IEP vs. non-IEP?
ESL vs. non-ESL?
Starting
Achievement
Instructional
Weeks
Typical growth
Outside of a teacher’s direct control

52. A Visual Representation of
Value Added
Spring 5th Grade
Test
Student A
Spring Score 209
Score 207
(Average Spring Score for Similar
Students)
Value Added
(+2 Score)
Student A
Fall Score 200
Fall 5th Grade
Test

53. Consider . . .
• What if I skip this step?
– Comparison is likely against normative data
so the comparison is to “typical kids in
typical settings”
• How fair is it to disregard context?
– Good teacher – bad school
– Good teacher – challenging kids

54. Value-added is science
• Value added models can control for a variety
of classroom, school level, and other
conditions
– Proven statistical methods
– All attempt to minimize error
– Variables outside controls are assumed as random

55. A variety of errors means more
stability only at the extremes
• Control for measurement
error
– All models attempt to address
this issue
• Population size
• Multiple data points
– Error is compounded with
combining two test events
– Many teachers’ value-added
scores will fall within the range
of statistical error

56. 12.00
11.00
10.00
9.00
8.00
7.00
6.00
5.00
4.00
3.00
2.00
1.00
0.00
-1.00
-2.00
-3.00
-4.00
-5.00
-6.00
-7.00
-8.00
-9.00
-10.00
-11.00
-12.00
Average Growth Index Score and Range
Range of teacher value-added
estimates
Mathematics Growth Index Distribution by Teacher - Validity Filtered
Q5
Q4
Q3
Q2
Q1
Each line in this display represents a single teacher. The graphic
shows the average growth index score for each teacher (green
line), plus or minus the standard error of the growth index estimate
(blue line). We removed students who had tests of questionable
validity and teachers with fewer than 20 students.

57. With one teacher,
error means a lot

58. Assumption of randomness can
have risk implications
• Value-added models assume that variation is
caused by randomness if not controlled for
explicitly
– Young teachers are assigned disproportionate
numbers of students with poor discipline records
– Parent requests for the “best” teachers are
honored
• Sound educational reasons for placement are
likely to be defensible

59. Instability at the tails of the
distribution
“The findings indicate that these modeling
choices can significantly influence outcomes for
individual teachers, particularly those in the tails
of the performance distribution who are most
likely to be targeted by high-stakes policies.”
Ballou, D., Mokher, C. and Cavalluzzo, L. (2012) Using Value-Added Assessment for Personnel
Decisions: How Omitted Variables and Model Specification Influence Teachers’ Outcomes.
LA Times Teacher #1
LA Times Teacher #2

60. How tests are used to evaluate
teachers
The Test
The Growth Metric
The Evaluation
The Rating

61. Translation into ratings can be
difficult to inform with data
• How would you
translate a rank order
to a rating?
• Data can be provided
• Value judgment
ultimately the basis
for setting cut scores
for points or rating

62. Decisions are value based,
not empirical
• What is far below a
district’s expectation is
subjective
• What about
• Obligation to help
teachers improve?
• Quality of replacement
teachers?

63. Even multiple measures need to
be used well
• System for combining elements and
producing a rating is also a value based
decision
– Multiple measures and principal judgment
must be included
– Evaluate the extremes to make sure it
makes sense

64. Leadership Courage Is A Key
5
4
3
2
1
0
Ratings can be driven by the assessment
Teacher 1 Teacher 2 Teacher 3
Observation Assessment
Real
or
Noise?

65. Big Message
If evaluators do not differentiate
their ratings,
then all differentiation comes from
the test

66. Lessons?
• What were some things you learned?
• What practices do you want to reinforce?
• What do you need to do differently?
• Think – Pair – Share
– 2 min to make some notes
– 3 min to share with a neighbor
– 2 min for two report outs on anything so far

67. Improving your
assessment program

68. • Read the sheet
and highlight
anything
interesting to
you
Let’s Define
Types of Assessments

69. The pursuit of compliance is exhausting
because it is always a moving target.
Governors move on, the party in power
gets replaced, a new president is elected,
and all want to put their own stamp on
education.
It is saner and less exhausting to define
your own course and align compliance
requirements to that.

70. Seven standards that define the
purpose driven assessment system

71. The purposes of all assessments are
defined and the assessments are valid
1 and useful for their purposes

72. Teachers are educated in the proper
administration and application of the
assessments used in their classrooms 2

73. Redundant, mis-aligned, or unused
3assessments are eliminated

74. Assessment results are aligned to
the needs of their audiences 4

75. Assessment results are delivered in a
timely and useful manner 5

76. The metrics and incentives used
encourage a focus on all learners 6

77. The assessment program contributes
to a climate of transparency and
objectivity with a long-term focus 7

78. 1. Typical assessment
purposes
• Identify student learning needs
• Identify groupings of students for instruction
• Guide instruction
• Course placement
• Determine eligibility for programs
• Award credits and/or assign grades
• Evaluate proficiency
• Monitor student progress
• Predict proficiency
• Project achievement of a goal
• Formative and summative evaluation of programs
• Formative evaluation to support school and teacher improvement
• Report student achievement, growth, and progress to the
community and stakeholders
• Summative evaluation of schools and teachers

79. 1. Assessment Purpose Survey
To increase value…
Identify gaps between:
1. How critical is this data
to your work?
2. How do you actually
use this data?
Take 10 min to fill this
out and 5 min to pair and
discuss areas of biggest
gap

80. Compare
assessments and
their purposes to
find unnecessary
overlaps
Take 10 min to
fill this out and 5
min to pair and
discuss areas of
redundancy
3. Eliminate waste

81. Lessons?
• What were some things you learned?
• What practices do you want to reinforce?
• What do you need to do differently?
• Think – Pair – Share
– 2 min to make some notes
– 3 min to share with a neighbor
– 2 min for two report outs on this section

82. Data Culture

83. Education Organizations
Mature
Poor to
Fair
Fair to
Good
Good to
Great
Great to
Excellent
Achieving the
basics of literacy
and numeracy
Getting the
foundations
in place
Shaping the
professional
Improving
through
peers and
innovation
Data use does too
Barber, M., Chijioke, C., & Mourshed, M. (2011). How the world’s most improved school systems keep getting better. McKinsey & Company.

84. Data Use Continuum
Poor to
Fair
Fair to
Good
Good to
Great
Great to
Excellent
One on One
Within
Teams
Within
the Walls
Across the
Walls
Requires a shift in the culture

85. What would you do with this?

86. Reflection Time
• Where are your pockets of most maturity?
• Least maturity?
• What is causing the differences?
Think - 2 min

87. Research on data use in
school improvement
• Education problems are “Wicked”
– Problem boundaries are ill-defined
– No definitive solutions
– Highly resistant to change
– Problem and solutions depend on
perspective
– Changes are consequential
Data can only take you so far

88. Research on data use in
school improvement
• Use data as a platform for deeper
conversations
• Define your problem well
– Problem title and description
– Magnitude
– Location
– Duration

89. Research on data use in
school improvement
• Part of a continuous improvement
process
–Data conversations
• Collaborative
• Embedded in culture
• Structured process
• Love, Nancy – Using data to improve learning for all
• Lipton & Wellman – Got data? Now what?
• NWEA Data Coaching

90. Final Question

91. What are your biggest
take-aways?
• Think – 2 min
• Pair – 3 min
• Share – 2 min - Two people

92. Last thing
• Materials in your Conference App
Presentation available on Slideshare.net

93. More information
Contacting me:
E-mail: andy.hegedus@nwea.org
Contacting us:
Exhibit Booth or Liz Kaplan