1. Atlanta Public Schools
Manufactured Academic Gaps?
27 November 2017 | Final
By Ed Johnson
Advocate for Quality in Public Education
(404) 505-8176 | edwjohnson@aol.com
Data sources:
https://gov.georgia.gov/sites/gov.georgia.gov/files/related_files/press_release/2017%20SSAS%20Award%20Schools.pdf
and
http://www.gadoe.org/CCRPI/Pages/default.aspx
2. Content Mastery – Student performance
on state assessments
Voice of Content Mastery Rates Variation
Something’s wrong. Very wrong. Detectable variation signals of special causes operating.
“Is this chart difficult? Patrick mastered it at age 11. This was his science project at school. A good start in life.
Some essential theory of variation could obviously be taught in the 5th grade. Pupils would come out of school
with knowledge in their heads, not merely information.” –W. Edwards Deming
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 2
3. CL 43.71
27.6
26.2
27.0
60.7
59.8
61.0
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
CRCT & EOCT GMAS (EOG & EOC)
All Schools
Voice of Content Mastery Rates Variation
Focus: All APS district schools
White, better
Black, sharply worse White v. Black CMR gaps behavior
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 3
125% inflation
due to special
causes operating
68.9
70.1
69.3
39.1
39.1
41.5
96.5
96.3
96.2
99.8
98.9
102.6
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
CRCT & EOCT GMAS (EOG & EOC)
All Schools
4. CL 41.72 38.94
49.65
24.3
22.3
24.5
59.7
58.1
61.5
22.1
21.3
20.3
56.4
57.5
56.0
36.3
35.3
32.8
64.9
64.7
64.0
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
CRCT &
EOCT
GMAS CRCT &
EOCT
GMAS CRCT &
EOCT
GMAS
Elementary Schools Middle Schools High Schools
Voice of Content Mastery Rates Variation
Focus: APS elementary, middle, and high schools
White, better
Black, sharply worse White v. Black CMR gaps behavior
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 4
144% inflation
due to special
causes operating
178% inflation
due to special
causes operating
98% inflation
due to special
causes operating
72.8
75.0
73.5
41.2
40.7
42.8
75.2
76.3
77.9
39.8
39.3
40.6
56.4
57.0
56.5
35.5
36.3
39.9
97.1
97.3
97.9
100.9
98.8
104.2
97.4
97.5
98.2
96.2
96.8
96.5
92.7
92.3
89.3
100.4
100.9
103.9
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
CRCT &
EOCT
GMAS CRCT &
EOCT
GMAS CRCT &
EOCT
GMAS
Elementary Schools Middle Schools High Schools
5. Voice of Content Mastery Rates Variation
Focus: APS public schools and charter schools
Public with White, better – Black, worse
Charter with White, worse – Black, worse White v. Black CMR gaps behavior
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 5
67.4
68.3
67.1
36.7
36.1
38.6
76.8
79.0
78.6
48.2
50.5
53.2
96.4
96.0
95.7
103.1
101.3
107.2
96.9
97.6
98.0
92.0
94.0
93.8
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017CRCT & EOCT GMAS CRCT & EOCT GMAS
Public Schools Charter Schools
CL 47.59
31.01
29.0
27.7
28.6
66.5
65.2
68.6
20.1
18.7
19.5
43.8
43.5
40.6
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
CRCT & EOCT GMAS CRCT & EOCT GMAS
Public Schools Charter Schools
133% inflation
due to special
causes operating
125% inflation
due to special
causes operating
6. Voice of Content Mastery Rates Variation
Focus: APS Black students and White students
White in Public, better – Charter, worse
Black in Public, worse – Charter, worse Public v. Charter CMR gaps behavior
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 6
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
CRCT EOG CRCT EOG CRCT EOG CRCT EOG EOCT EOC EOCT EOC
Black White Black White Black White
Elementary Schools Middle Schools High Schools
Center Line: Average gap size
-11.9
1.9
-12.3
13.0
-3.9
14.1
-6.5
-13.6
-14.6
-9.8
-14.2
-12.9
-0.6
-2.0
-0.8
5.3
2.7
6.7
-11.7
-6.3
-7.1
-15.1
-16.9
-16.8
2.9
3.7
2.5
27.6
22.8
18.6
12.2
12.2
4.4
-13.8
-16.4
-21.6
1.1
27.1
-40
-20
0
20
40
60
80
100
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
CRCT EOG CRCT EOG CRCT EOG CRCT EOG EOCT EOC EOCT EOC
Black White Black White Black White
Elementary Schools Middle Schools High Schools
7. Closer Look
Voice of Content Mastery Rates Variation for
2017 SSAS Award Recipients
Something’s wrong even among the best.
Single Statewide Accountability System (SSAS) Awards are given to schools with high academic performance
and/or high academic growth.
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 7
8. Carver Early College High School Content Mastery Rates
(2017 SSAS Bronze “Highest Performing” Award Recipient)
Black CMR over time (White not applicable) Black CMR systemic behavior
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 8
98.8
98.7
100.0
88.8
71.9
66.7
100.0
100.0
97.5
74.7
75.0
47.9
94.3
92.7
97.1
87.8
75.0
71.3
98.5
100.0
100.0
90.7
65.5
78.0
96.8
100.0
91.9
64.3
81.0
57.7
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017EOCT EOC EOCT EOC EOCT EOC EOCT EOC EOCT EOC
9th Grade Literature and
Composition
American Literature and
Composition
Biology Economics/Business/Free
Enterprise
US History
UCL 105.94
110.55
103.27
114.59
123.94
CL 87.47
82.52
86.37
88.78
81.94
LCL 68.99
54.48
69.48
62.98
39.94
98.8
98.7
100.0
88.8
71.9
66.7
100.0
100.0
97.5
74.7
75.0
47.9
94.3
92.7
97.1
87.8
75.0
71.3
98.5
100.0
100.0
90.7
65.5
78.0
96.8
100.0
91.9
64.3
81.0
57.7
0
20
40
60
80
100
120
140
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
2012
2013
2014
2015
2016
2017
EOCT EOC EOCT EOC EOCT EOC EOCT EOC EOCT EOC
9th Grade Literature and
Composition
American Literature and
Composition
Biology Economics/Business/Free
Enterprise
US History
15. Closer Look
Seeing Content Mastery affect
Achievement and Performance
Systemic views
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 15
16. 0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
What patterns and specifics do you see happening?
7 blue-dot schools, 2017 Single Statewide Accountability System (SSAS) Award Recipients
16 red-dot schools, eligible for state takeover
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 16
2012, 2013, and 2014: Elementary & middle schools were assessed by Criterion-Referenced Competence Tests (CRCT); high schools were assessed by End of Course Tests (EOCT).
2015, 2016, and 2017: Elementary & middle schools were assessed by Ga Milestones End of Grade (EOG) tests; high schools were assessed by Ga Milestones End of Course (EOC) tests.
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
17. 0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
What patterns and specifics do you see happening?
7 blue-dot schools, 2017 Single Statewide Accountability System (SSAS) Award Recipients
16 red-dot schools, eligible for state takeover
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 17
2012, 2013, and 2014: Elementary & middle schools were assessed by Criterion-Referenced Competence Tests (CRCT); high schools were assessed by End of Course Tests (EOCT).
2015, 2016, and 2017: Elementary & middle schools were assessed by Ga Milestones End of Grade (EOG) tests; high schools were assessed by Ga Milestones End of Course (EOC) tests.
18. 0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
What patterns and specifics do you see happening?
7 blue-dot schools, 2017 Single Statewide Accountability System (SSAS) Award Recipients
16 red-dot schools, eligible for state takeover
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 18
2012, 2013, and 2014: Elementary & middle schools were assessed by Criterion-Referenced Competence Tests (CRCT); high schools were assessed by End of Course Tests (EOCT).
2015, 2016, and 2017: Elementary & middle schools were assessed by Ga Milestones End of Grade (EOG) tests; high schools were assessed by Ga Milestones End of Course (EOC) tests.
19. Closer Look
An elementary Common Core/Math-based case for
Achievement-Progress Transits
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 19
Achievement Points (A)
Progress Points (P)
20. What is a Transit?
• The straight-line transition from one
(CCRPI Achievement Points, CCRPI
Progress Points) coordinate pair of
values to another such coordinate
pair of values
• Each example Transit at right
• Communicates seeing unified
transition of both
• Achievement and
• Progress
• Communicates seeing both
• Transit Direction and
• Transit Magnitude
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 20
Year 1
Year 2
Year 3
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
Transit Examples
21. CC/M-based case for
Achievement-Progress
Transits
• Issue: Unstable state mandates about dominance of
CCRPI components
• Achievement used to be 1st order dominant
• Progress used to be 2nd order dominant
• Achievement now 2nd order dominant
• Progress now 1st order dominant
• Issue: Achievement and Progress strangely related,
counterintuitive in common vernacular
• Both positive Progress and negative Achievement
happens
• Both negative Progress and positive Achievement
happens
• Solution: Transits unify and stabilize the 1st and 2nd
order dominant CCRPI components Achievement Points
and Progress Points in both magnitude and direction
• Solution Basis in Common Core Standards/Math
• Grade 1 » Geometry » Reason with shapes and their
attributes, through
• Grade 7 » Introduction » 1) » slope, and
• Grade 8 » Geometry » Understand and apply the
Pythagorean Theorem
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 21
61.6% 60.0% 61.3%
36.8% 36.9% 37.5%
24.9% 25.1% 24.9%
52.0% 50.8% 50.8%
86.5% 85.1% 86.2%
88.8% 87.7% 88.3%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2012 2013 2014 2015 2016 2017
CRCT/EOCT EOG/EOC
Progress Points as Unstable % of CCRPI Scores
Achievement Points as Unstable % of CCRPI Scores
Achievement-Progress Points as Stable % of CCRPI Scores
22. Closer Look
Achievement-Progress Transits of
2017 SSAS Award Recipients
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 22
Achievement Points (A)
Progress Points (P)
23. Carver Early College High School Transits
(2017 SSAS Bronze “Highest Performing” Award Recipient)
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 23
2012
20132014
2015
2016
2017
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
Assessment
Type
School Year
End
Achievement
Points
Progress
Points
CCRPI
Score
Transit
Slope
Transit
Vector
EOCT 2012 54.6 19.8 94.5 N/A N/A
EOCT 2013 53.7 18.4 94.3 -1.56 -1.66
EOCT 2014 52.4 18.1 94.3 -0.23 -1.33
EOC 2015 42.5 40.0 101.7 -2.21 -24.03
EOC 2016 42.5 37.2 97.3 -Undefined -2.80
EOC 2017 37.2 37.1 91.9 -0.02 -5.30
24. Cleveland Avenue Elementary School Transits
(2017 SSAS Gold “Greatest Gains” Award Recipient)
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 24
2012
2013
2014
2015
2016
2017
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
Assessment
Type
School Year
End
Achievement
Points
Progress
Points
CCRPI
Score
Transit
Slope
Transit
Vector
CRCT 2012 37.9 15.2 54.6 N/A N/A
CRCT 2013 35.2 16.5 52.2 -0.48 -3.00
CRCT 2014 36.6 19.5 67.6 2.14 3.31
EOG 2015 22.2 39.2 72.2 -1.37 -24.40
EOG 2016 24.3 40.0 80.8 0.38 2.25
EOG 2017 26.7 38.3 76.0 -0.71 -2.94
25. Jackson Elementary School Transits
(2017 SSAS Bronze “Greatest Gains” & Gold “Highest Performing” Awards Recipient)
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 25
2013
2012 2014
2015
2016
2017
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
Assessment
Type
School Year
End
Achievement
Points
Progress
Points
CCRPI
Score
Transit
Slope
Transit
Vector
CRCT 2012 57.5 18.6 94.0 N/A N/A
CRCT 2013 57.5 18.6 93.2 +Undefined 0.00
CRCT 2014 57.9 18.4 93.5 -0.50 -0.45
EOG 2015 45.4 40.0 97.7 -1.73 -24.96
EOG 2016 46.5 35.5 95.4 -4.09 -4.63
EOG 2017 47.3 40.0 100.7 5.63 4.57
26. Mary Lin Elementary School Transits
(2017 SSAS Bronze “Greatest Gains” & Gold “Highest Performing” Awards Recipient)
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 26
2012
2013
2014
2015
2016
2017
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
Assessment
Type
School Year
End
Achievement
Points
Progress
Points
CCRPI
Score
Transit
Slope
Transit
Vector
CRCT 2012 57.7 17.4 90.9 N/A N/A
CRCT 2013 56.9 14.2 86.4 -4.00 -3.30
CRCT 2014 57.5 17.5 91.5 5.50 3.35
EOG 2015 43.8 37.2 93.2 -1.44 -24.00
EOG 2016 44.9 39.5 98.9 2.09 2.55
EOG 2017 46.2 38.3 97.8 -0.92 -1.77
27. Morningside Elementary School Transits
(2017 SSAS Platinum “Highest Performing” Award Recipient)
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 27
2012
2013
2014
20152016
2017
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
Assessment
Type
School Year
End
Achievement
Points
Progress
Points
CCRPI
Score
Transit
Slope
Transit
Vector
CRCT 2012 56.8 19.6 93.7 N/A N/A
CRCT 2013 58.9 20.4 96.1 0.38 2.25
CRCT 2014 59.4 18.9 94.1 -3.00 -1.58
EOG 2015 48.3 38.8 98.2 -1.79 -22.79
EOG 2016 46.7 35.9 96.5 -1.81 -3.31
EOG 2017 46.3 35.6 94.7 -0.75 -0.50
29. Springdale Park Elementary School Transits
(2017 SSAS Gold “Highest Performing” Award Recipient)
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 29
2012
2013 2014
2015
2016
2017
0
10
20
30
40
50
0 10 20 30 40 50 60 70
ProgressPoints
Achievement Points
Assessment
Type
School Year
End
Achievement
Points
Progress
Points
CCRPI
Score
Transit
Slope
Transit
Vector
CRCT 2012 58.7 15.8 90.9 N/A N/A
CRCT 2013 55.2 17.5 90.5 -0.49 -3.89
CRCT 2014 58.5 16.3 91.4 -0.36 -3.51
EOG 2015 46.0 35.4 94.5 -1.53 -22.83
EOG 2016 45.7 36.6 95.6 -4.00 -1.24
EOG 2017 45.8 36.7 95.8 1.00 0.14
30. 2017 SSAS Award Recipients
Achievement-Progress Transits systemic behaviors
27 Nov 2017 | Final Advocate for Quality in Public Education (AfQPE) 30
UCL 9.6 8.3
-22.3
12.6 10.6
CL -1.0 -0.2
-23.8
-0.9 -0.6
LCL -11.5
-8.8
-25.4
-14.5
-11.7
2.87
-1.66
-3.00
0.00
-3.30
2.25
-3.89
-1.50
-1.33
3.31
-0.45
3.35
-1.58
-3.51
-23.69
-24.03
-24.40
-24.96
-24.00
-22.79
-22.83
0.64
-2.80
2.25
-4.63
2.55
-3.31
-1.24
1.78
-5.30
-2.94
4.57
-1.77
-0.50
0.14
-50
-40
-30
-20
-10
0
10
20
30
BrandonElementarySchool
CarverEarlyCollegeHighSchool
ClevelandElementarySchool
JacksonElementarySchool
LinElementarySchool
MorningsideElementarySchool
SpringdaleParkElementarySchool
BrandonElementarySchool
CarverEarlyCollegeHighSchool
ClevelandElementarySchool
JacksonElementarySchool
LinElementarySchool
MorningsideElementarySchool
SpringdaleParkElementarySchool
BrandonElementarySchool
CarverEarlyCollegeHighSchool
ClevelandElementarySchool
JacksonElementarySchool
LinElementarySchool
MorningsideElementarySchool
SpringdaleParkElementarySchool
BrandonElementarySchool
CarverEarlyCollegeHighSchool
ClevelandElementarySchool
JacksonElementarySchool
LinElementarySchool
MorningsideElementarySchool
SpringdaleParkElementarySchool
BrandonElementarySchool
CarverEarlyCollegeHighSchool
ClevelandElementarySchool
JacksonElementarySchool
LinElementarySchool
MorningsideElementarySchool
SpringdaleParkElementarySchool
From 2012 to 2013 From 2013 to 2014 From 2014 to 2015 From 2015 to 2016 From 2016 to 2017
What happened to ALL schools?!
First ever
Ga Milestones happened!!
31. 27 Nov 2017 | Final
Thank You!
Questions?
Ed Johnson
Advocate for Quality in Public Education
(404) 505-8176 | edwjohnson@aol.com
Advocate for Quality in Public Education (AfQPE) 31