A First Look at Trends and Bright Spots in St. Louis School Performance Post-Pandemic
•Download as PPTX, PDF•
0 likes•200 views
In partnership with Exponent Education, a highly regarded education data group, you are invited to a discussion on the recently released state education data – our first look at how children and schools are doing post-pandemic.
Recommended
Recommended
More Related Content
Similar to A First Look at Trends and Bright Spots in St. Louis School Performance Post-Pandemic
Similar to A First Look at Trends and Bright Spots in St. Louis School Performance Post-Pandemic (20)
Recently uploaded
Recently uploaded (20)
A First Look at Trends and Bright Spots in St. Louis School Performance Post-Pandemic
- 1. Learning To Improve: A First Look at Trends and Bright Spots in St. Louis School Performance Post-Pandemic January 2023 Analysis and visualization by
- 2. PAGE 2 Welcome & Agenda 1. State and Regional Education Data Analysis 2. Insights from Fast-Improving Systems 3. A New Public Data Tool and Improvement Resource
- 3. PAGE 3 Meet Steve Cartwright Steve Cartwright Founder, Exponent Education • Early research experience at Urban Institute and DC Public Schools • Served as Fellow, Strategic Data Project, Harvard Center for Education Policy Research • Analytics consultant with schools, systems, and support orgs across the country
- 4. PAGE 4 Today’s content 1 Pandemic impact on national and state achievement 2 New evidence on regional achievement 3 Finding and learning from outliers
- 5. PAGE 5 • Losses in these key milestones ranged from about half to nearly all of the gains made between 2000 and 2019. The Covid-19 pandemic significantly impacted student academic achievement across the country 272 276 278 280 282 283 284 281 282 281 273 211 216 217 220 220 220 221 221 221 219 216 2000 2003 2005 2007 2009 2011 2013 2015 2017 2019 2022 National Assessment of Educational Progress Composite Score, 2009-2022, All Students -8 points (loss of 86% of 2000 to 2019 gains) -3 points (loss of 41% of 2000 to 2019 gains) Grade 8 Math Grade 4 Reading No Child Left Behind: national emphasis on assessment and accountability Race to the Top: national emphasis on standards and teacher evaluation Current era interrupted by global pandemic • Average achievement increased significantly between 2000 and 2013 before stagnating and declining. • The pandemic most affected students’ math performance, but may have lingering effects in reading. Information on NAEP cut scores for Basic and Proficient performance can be found here. Grade 8 Proficient Grade 8 Basic Grade 4 Basic Grade 4 Proficient
- 6. PAGE 6 • Black students in Missouri were disproportionately affected and experienced some of the nation’s largest declines. In Missouri, average declines were similar to the nation, but losses for Black students were particularly large 222 221 221 224 220 222 223 223 218 213 203 200 200 204 199 200 202 200 200 186 279 276 281 286 282 283 281 281 281 272 250 247 253 260 254 260 258 256 258 245 2003 2005 2007 2009 2011 2013 2015 2017 2019 2022 National Assessment of Educational Progress - Missouri Composite Score, 2009-2022, All Students -9 points (loss of >400% of 2003 to 2019 gains) -12 points (loss of >160% of 2003 to 2019 gains) -5 points (loss of ~110% greater than 2003 to 2019 loss) -14 points (loss of ~500% of 2003 to 2019 loss) Grade 8 Math Grade 4 Reading • In grade 4 reading, Black elementary students in Missouri lost more ground than Black students in nearly every other state. • Missouri’s Black middle school students are now performing among the lowest in the country. All Students Black Students All Students Black Students -19 point difference -27 point difference
- 7. PAGE 7 Today’s content 1 Pandemic impact on national and state achievement 2 New evidence on regional achievement 3 Finding and learning from outliers
- 8. PAGE 8 ELA performance continued to decline from 2021 to 2022 statewide and remained relatively flat locally Interact with these data on www.theopportunitytrust.org • In the state and county, ELA achievement continued to decline from 2021 to 2022. +1 +2 • Locally, both charters and SLPS improved modestly (+1% and +2% points, respectively). • No sector has reached pre-pandemic levels of performance. -2 -1
- 9. PAGE 9 • Mirroring national trends, students lost more ground in mathematics than reading. Math proficiency shows some signs of “rebounding” statewide and locally, unlike ELA Interact with these data on www.theopportunitytrust.org +4 +7 +6 +3 • Some signs of ”rebounding” exist in math: students statewide, in St. Louis county, and across the city are improving, through no sector is back to pre-pandemic achievement.
- 10. PAGE 10 • Selective admissions schools perform 6-8x better than open enrollment schools, which enroll the vast majority of SLPS students. Aggregate results mask a large inequality of outcomes between open enrollment and selective admissions schools 6x larger 8x larger • Those selective admissions schools also saw greater rates of improvement from 2021 to 2022.
- 11. PAGE 11 Twenty years of research identifies key milestones in children's development that predict meaningful life outcomes Kindergarten readiness 3rd grade reading proficiency 8th grade math proficiency 4th grade math proficiency A local study found that students entering kindergarten behind their peers continued to lag 18 months later (University of Missouri 2019). Another study from Ohio found nearly three in four (74%) students who tested positively for a potential problem on a kindergarten readiness assessment scored below the passing promotion score on the third grade ELA assessment (Justice et al 2019). Students who were not proficient in third grade reading are four times as likely not to graduate from high school as those who were reading proficiently in third grade. For children who lived in a high-poverty neighborhood for at least a year and were not reading proficiently in third grade, they are six times as likely not to graduate (Hernandez 2012). Researchers studying ten- year-old students’ mathematical knowledge in the US and the UK found that their mastery of fractions was highly predictive of their overall mathematical and algebraic knowledge in high school, five to six years after initial testing (Siegler et al 2012). Researchers consider eighth grade math the “gatekeeper” to higher math and science courses and other postsecondary opportunities for both college and career track students (Wang and Goldsmith 2003).
- 12. PAGE 12 Interact with these data on www.theopportunitytrust.org • Nearly two thirds (65%) of SLPS students scored below basic - the lowest of four levels of performance. • Both sectors have more than 50% of 3rd grade students at the lowest level of performance. Milestone: 3rd grade reading proficiency • The 3rd grade results mirror the overall trend of stagnation statewide with some modest improvements locally.
- 13. PAGE 13 Interact with these data on www.theopportunitytrust.org Milestone: 8th grade math proficiency • The charter sector also saw a reduction in the percentage of students below basic, but in SLPS that figure expanded and is now almost three quarters of students. • “Rebounding” in math is shown not just in proficiency improvements, but also in reductions in the percentage of students below basic statewide and in the county.
- 14. PAGE 14 Today’s content 1 Pandemic impact on national and state achievement 2 New evidence on regional achievement 3 Finding and learning from outliers
- 15. PAGE 15 The schools best helping students recover aren’t just those increasing proficiency rates Interact with these data on www.theopportunitytrust.org • Proficiency rates are crude measures to use when evaluating change over time. 44% 43% 43% 46% 9% 9% 4% 2% 2022 2021 Basic Below Basic Proficient Advanced 42% 42% 37% 37% 16% 15% 5% 5% 2022 2021 Basic Below Basic Proficient Advanced ELA Performance by Level, 2021 to 2022 Black Students Missouri Momentum (formerly Eagle) No change +1% pt. No change No change +2% pts. No change +1% pt. -3% pts. • Movement across all four performance levels provides a more detailed view of change. • Unlike the state, Momentum increased proficiency rates and decreased the rate of students below basic.
- 16. PAGE 16 Fast-recovering schools/systems for Black students include a mix of districts and charters • Outlier schools achieve outsized performance index changes with Black students (and serve a majority of Black students overall). Interact with these data on www.theopportunitytrust.org Note: Only schools with <10% of students omitted are shown. School composition at least 50% Black students Compute a performance index to capture movement across all levels of performance
- 17. Compared to the state, county, and open enrollment SLPS schools, outliers are improving 2x-15x faster PAGE 17 Interact with these data on www.theopportunitytrust.org Note: Only schools with <10% of students omitted are shown. Kairos Academies SDUC Momentum +5.5 index points +5.9 index points +8.1 index points Missouri +0.4 index points St. Louis County +0.4 index points SLPS non- selective +2.7 index points
- 18. PAGE 18 Outliers present an example of what’s possible citywide when the right conditions for improvement are in place 15 9 28 43 0 10 20 30 40 50 60 70 80 90 100 ELA Math 2022 Actual 2027 Projection St. Louis City Black Student Proficiency Rate Projection SLPS and Charter • Current proficiency rates for Black students citywide are 15% in ELA and 9% in math. * +2.6% points in ELA and +6.9% points in math. **See Chetty, R., Friedman, J. N., & Rockoff, J. (2011). New evidence on the long-term impacts of tax credits. IRS white paper. In an average class of 30 students, this amounts to 1 additional student proficient in ELA and 2 additional students proficient in math per year. Statewide all student average 43 39 • If every SLPS school and charter improved their proficiency rates for Black students at Momentum’s rate* for the next five years, citywide Black student achievement would dramatically improve. • These increased test scores translate into significant improvements in college attendance and early career earnings.**
Editor's Notes
- Guest let into zoom from wait room at 11:58 Eric welcome guests and open presentation
- Eric
- Eric introduce Steve Steve: Thanks Eric! Good <morning> all -- I’m excited to be here and share the most recent data we’ve obtained on student achievement throughout Missouri and here locally in St. Louis I’ve been working with educational data for 15 years and think that there’s never been a more important time to tune into what the data are telling us about the state of our schools A few quick words about my background -- I started my research career in the Urban Institute in Washington DC, which is where I still live today, and also led applied analytics work in the DC Public Schools during the leadership of Michelle Rhee and Kaya Henderson During my time in DCPS, I was concurrently a fellow with Harvard’s Center for Education Policy Research, which supports districts with analytic expertise More recently, I have been consulting across the country through my company, Exponent Education, doing all manner of strategic planning analysis, data strategy and dashboard development, and applied research and evaluation
- This presentation has three main components, after which we’ll pause and create space for questions and discussion. <click> The first component focuses on the impact of the pandemic on student achievement across the country based on the latest data from the National Assessment of Educational Progress -- also called the Nation’s Report Card, which I’ll share more about in a moment. Essentially, those data provide the most apples-to-apples comparison of how students are performing across state lines, so we’ll investigate how Missouri’s students fare relative to the nation. <click> In the second component, we’ll examine data released in December by Missouri’s Department of Elementary and Secondary Education -- these data allow us to examine achievement at the school, district, and regional level and also provide a window into how students are recovering from 2021 to 2022. <click> Finally, we’ll explore what we can learn from schools that are outliers -- these are schools and systems that are changing the odds and helping Black students in the St. Louis region achieve outsized gains from 2021 to 2022 on the road to recovery. OK - let’s jump in and look at the national data.
- The National Assessment of Educational Progress - or NAEP for short - is valid, reliable assessment that’s been administered in every state and the District of Columbia for decades. It provides the most comprehensive national view of student achievement and enables comparisons state-to-state. It’s administered in grades 4 and 8 in reading and math, as well as other content areas. <click> Before the pandemic, leading up to 2019, there were significant improvements in the average achievement of students nationally in 4th grade reading and 8th grade math, shown here. <click> This progress was significant -- you can see here that the average student moved from closer to a “basic” level of content mastery on the assessment to midway toward the proficiency standard These improvements roughly corresponded with two major chapters of federal education policy: <click> first, No Child Left Behind, which was authorized in 2002 and placed a heavy emphasis on assessment and accountability for the performance of all student groups in a school. <click> The second chapter began in 2009 when the Obama administration announced a program to offer states flexibility from the requirements of No Child Left Behind through its Race to the Top program, which prioritized more rigorous standards aligned to college and career readiness and more rigorous systems for teacher evaluation. Congress also reauthorized No Child Left Behind as the Every Student Succeeds Act in 2015, at which point the achievement increases of a decade and a half generally plateau. The modern era of national achievement has largely been characterized by the staggering impact of the Covid-19 pandemic. <click 3 times> You can see here the significant declines in test performance from 2019 to 2022 - particularly in mathematics. I really want to spend a moment thinking about the gravity of these declines. <speak slowly> <click> 8th graders declined 8 points in math performance, which is nearly all of the gains made over two decades from 2000 to 2019. <click> 4th graders declined less in reading performance, but this 3 point loss is still almost half of the gains made over those same two decades. And of course, we know from the research and our first hand experience the myriad other impacts the pandemic has had on student, teacher, and family health and wellbeing that go well beyond what’s reflected in these test scores. REF: gr 4 reading (208 basic 238 prof); gr8 math (262 basic 299 prof)
- In Missouri, average student achievement was roughly on par with students nationwide prior to the pandemic. <click> However, you can see here the significant achievement gap faced by Black students statewide. We’ll focus here on Black students because they face the greatest inequities in the St. Louis region and given the mission of The Opportunity Trust. <click> In 4th grade reading, Black student performance declined 14 points from 2019 to 2022, which is almost 3x the rate of decline experienced by the average student. <click> As a result, the achievement gap for Black students widened from 19 points in 2003 to 27 points in 2022. <click> The amount of this decline is more than almost every other state. Only two states’ Black students were impacted more and only one state, Maine, has lower performing Black students overall as of 2022. <click> In 8th grade math, the trend is similar. Black students experienced an 8 point gain from 2003 to 2019; during the pandemic, however, those students lost 12 points, which is more than all of the gains seen during those two decades. Only 5 states saw larger declines for Black students and Missouri now ranks 4th from the bottom in 8th grade math performance nationwide for Black students.
- With that national picture in mind, we can now turn to new data released in December from DESE. The 2022 state test results allow us to make comparisons to not just pre-pandemic performance in 2019 as NAEP did, but to 2021 performance so we can determine just how much schools are rebounding or recovering from the effects of the pandemic. They also allow us to examine results by community, district, and school.
- Let’s start with reading, also known as English Language Arts, or ELA for short. <click> From 2021 to 2022, the percentage of students scoring proficient or advanced on the Missouri state assessment -- which identifies students who have mastered state standards -- continued to fall. Less than half of students statewide and in St. Louis county met these standards in 2022. I’m not particularly surprised by this -- as we saw in 2021 and from results in communities across the nation, the pandemic had less of an overall effect on students’ reading proficiency compared with math. We’re now seeing the accumulating effect of rising 3rd graders who spent most of 1st and 2nd grades in the pandemic. <click twice> Locally, while overall performance is worse, St. Louis charters and SLPS schools both saw modest improvements from 2021 to 2022: +1 point in charters and +2 points in SLPS. <click> Still, no sector in the city is back to their pre-pandemic level of performance, which was not particularly strong. Even before the pandemic, less than 1 in 3 students was proficient in charters and less than 1 in 4 was proficient in SLPS. Before we continue, I want to note the “customize this chart” button you see on the slide. All of these data and charts are available on The Trust’s new “Learning to Improve” website for you to analyze yourself so you can explore results for specific grade levels, student groups, districts, or schools over time and by content area.
- There is better news statewide and locally in mathematics. <click> Consistent with data we’re seeing in other states and communities, students experienced larger initial declines in math compared to ELA from 2019 to 2021 when schools first went remote <click twice> AND students are also seeing some “rebounding”, or positive improvement, from 2021 to 2022. While no sector is back to its 2019 pre-pandemic achievement level, these improvements are significant. Math proficiency increased by 4 points statewide, by 7 points in St. Louis county schools, by 6 points among St. Louis charters, and by 3 points in SLPS schools. To put this in context, that 6% point increase among St. Louis charters means more than 360 students tested proficient in 2022 versus 2021. That’s a lot. It’s also worth saying a word about participation rates here: SLPS tested a significantly smaller percentage of students in 2022 than 2021 -- almost 1k fewer students tested and the rate went from 94% to 87%. The largest declines were among historically lower performing groups. In charters, participation was 98%.
- One important caveat to note before we continue is that looking at SLPS results in the aggregate belies a large inequality between the vast majority of students enrolled in open enrollment schools and the small minority admitted into various selective admissions schools across the city. Open enrollment schools account for more than 80% of the tested student population in SLPS, about 6500 students. Selective schools test just about 1200 students. <click> As you can see here, proficiency rates tend to be 6-8x higher in these selective admissions schools, which pulls up the district average by a non-trivial amount. <click> Those selective admissions schools also improved more from 2021 to 2022. So, the vast majority of students in SLPS are enrolled in schools with very low levels of performance.
- Beyond the aggregate results - representing students in all grades - it’s important to examine progress in the grade-level data, particularly in grades 3 and 8. There’s an accumulation of research on early predictors of later academic and life outcomes, such as on-time high school graduation and college and career readiness. <click> These predictive milestones begin with kindergarten readiness, which measures students’ developmental preparedness for school. Students who attend a high quality pre K program are more likely to demonstrate school readiness and are more likely to master grade-level content in elementary school. <click> The second milestone is reading proficiently by the end of grade 3. Research shows that students achieving proficiency are much more likely to be able to access the rigor of the curriculum in future grade levels and achieve higher levels of educational attainment. <click> Math proficiency in grade 4 is also predictive of students’ ability to be successful in upper-level math courses because of the importance of fractions as a ”power standard”; that is, a content standard that is foundational to other math concepts. <click> Finally, mastery of 8th grade math skills ensures students have the essential math concepts needed for everyday life and are also in a position to complete a full high school math sequence, which best prepares students for college and/or career. We are going to drill into grade 3 reading and grade 8 math - two specific milestones - because of the strength of evidence on their predictive power.
- In 3rd grade, ELA results are similar to what we saw overall, which are continued declines statewide and small increases locally. On this page, you’re looking not just at the overall percent of students proficient or advanced, but the percentage of students at each level. <Click> Importantly, as we drill down into specific grade levels, we’re unable to compute a fully accurate result for the charter sector -- the grey boxes here indicate the percentage of students whose results could not be determined based on how the state suppresses their scores. <click> Still there’s a lot we can learn from these results by level <click> here you can see, of the percent of students not proficient, what percentage performed at a basic level of mastery or a below basic level of mastery. <click> In both sectors, at least 50% of our youngest students are reading at the below basic level. Given that grade 3 marks an important shift from when students move from learning to read to reading to learn, these below basic students are unprepared to access the rigors of the curriculum in grades 4 and up. According to Missouri standards, students at the below basic level “demonstrate little or no ability to organize and/or develop writing or exhibit little command of the conventions of standard English” <click> In the charter sector, the below basic percentage is approximately 50, but it could be as high as 57 given that we don’t know the scores of 7% of students. In SLPS, that figure is nearly two thirds of students, or 65%. Of more than 1200 3rd grade students tested in SLPS’s open enrollment schools -- the vast majority of this statistic -- nearly 900 3rd graders are below basic.
- Now we’re looking at 8th grade math results - so this is how today’s 9th graders performed. <click> As we saw earlier, proficiency rates have been “rebounding” statewide and locally. That improvement in proficiency rates is accompanied by significant reductions in the percentage of students scoring below basic statewide and in the county. <click twice> Locally, the charter sector also appeared to significantly reduce the percentage of below basic students, but as with 3rd grade results, not all test scores could be incorporated into this analysis. In SLPS, however, the percentage of below basic 8th graders increased to nearly three quarters of the 8th grade class. This means that the vast majority of SLPS students entering high school will need significant intervention to master the skills required to engage in a high school math sequence.
- The last component of our analysis looks at the variability in results to find schools and systems that are achieving outlier status and improving student achievement at faster rates than average.
- To find outliers, we want to look beyond crude proficiency rates. <click> As you saw with the grade-level results just a moment ago, there are actually four levels of performance on the Missouri assessment. Given the impact of the pandemic, we want to find schools that are not just moving more students above the proficiency threshold, but we also want to detect improvement for the students farthest behind. We can do this by computing an index that captures how many students are at each of the four levels of performance in any given year. <click> Here we are looking at Black student performance by level. Statewide, you can see that Black students improved modestly in ELA - with no real change in the percentage of students achieving below basic or basic mastery of standards. <click> On the other hand, at Momentum - a local charter network - we see not just an improvement in the percent of Black students who are advanced (+2%), but also a meaningful reduction in the percent of Black students scoring below basic and a slight increase in the percent of Black students scoring basic. It's this positive upward movement of students across the levels of performance that helps best tell the story of year-over-year change.
- We’ve computed these index measures for all schools and systems and you can find these data on our website. When we subset that list to just those systems whose student enrollment is at least 50% Black students <click> , and sort them on how much their performance index changed from 2021 to 2022 in ELA <click>, we identify a mix of outliers <click> - both charter and district. I’ll drill into the top three examples in ELA, but know that you can access this full dataset on our website to look for outliers in other content areas, as well.
- Statewide and in St. Louis county, proficiency rates for Black students in ELA improved modestly from 2021 to 2022 - and with very little change in the percent of students scoring below basic (shown in red) <click> Compare that to University City, where the percentage of students scoring advanced increased, as well as proficient, as well as a reduction in the percent of students below basic. Black students in university city improved at 15x the rate of Black students county-wide. <click> As another comparison, consider open enrollment SLPS schools, which also saw some modest improvements in ELA. (Note, however, that the state did not identify scores for about 6% of students in these schools, so our data reflect the vast majority - 94% - but not all students.) <click> When you look at Kairos Academies and Momentum - Black students improved at 2-3x the rate of district-run schools. For both campuses, you can see the significant reductions in the percent of black students scoring at the lowest level of performance, and significant increases in students at the proficient or advanced levels, or both. These gains are substantial and they can be instructive if we are able to get under the hood, so to speak, and learn about what practices are driving their success. The mere existence of outlier schools shows us what is possible -- many of these schools are serving entirely historically disadvantaged student populations and achieving life-changing results.
- As we wrap up the data story, we want to share a short thought experiment. We just looked at the inspiring progress that three examples of fast-recovering systems are experiencing: University City, Kairos Academies, and Momentum. These schools are changing the odds for Black students and showing what’s possible. <click> Now consider our current citywide proficiency rates - inclusive of both district schools and charters. They’re 15% in ELA and just 9% in math. What if every school were able to implement the approaches used in these outliers? <click> As an illustration, if every school in the city improved at the rate of Momentum for the next 5 years (that’s improving proficiency by less than 3% points in ELA and less than 7% points in math), <click> citywide achievement would improve dramatically. In ELA, we’d be able to close the statewide achievement gap by half -- and in math we’d be outperform the state average. <click> Improvements of this magnitude aren’t just reflected in test scores. The best available research shows that higher scores increase students’ probability of college attendance, raise earnings, reduce teenage birth rates, and improve the quality of the neighborhood in which students live in adulthood. <click> So while the road to recovery will of course be challenging, these examples show it's possible. To finish the thought experiment, imagine you are teaching a class of 30 students, which is larger than most. <speak slowly> In a given year, we’d need you to move 1 additional student to proficiency in ELA and 2 students to proficiency in math to achieve this level of improvement. Surely we can do better. That concludes our data analysis, so I look forward to hearing your questions and reactions.