The New York City Charter Schools
       Evaluation Project

How New York City’s Charter
 Schools Affect Achievement




 ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                   INTRODUCTORY MATERIAL




HOW NEW YORK ...
NEW YORK CITY'S CHARTER SCHOOLS
                                           INTRODUCTORY MATERIAL


TABLE OF CONTENTS
Bibli...
NEW YORK CITY'S CHARTER SCHOOLS
                                           INTRODUCTORY MATERIAL


Chapter III: The Superi...
NEW YORK CITY'S CHARTER SCHOOLS
                                           INTRODUCTORY MATERIAL

! Lottery-based results:...
NEW YORK CITY'S CHARTER SCHOOLS
                                           INTRODUCTORY MATERIAL


THE NEW YORK CITY CHART...
NEW YORK CITY'S CHARTER SCHOOLS
                                           INTRODUCTORY MATERIAL

The New York City Charte...
NEW YORK CITY'S CHARTER SCHOOLS
                                           INTRODUCTORY MATERIAL


EXECUTIVE SUMMARY
The d...
NEW YORK CITY'S CHARTER SCHOOLS
                                           INTRODUCTORY MATERIAL


EXECUTIVE SUMMARY, CONT...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I




CHAPTER I. NEW YORK CITY'S ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I

How are students admitted to N...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I

A typical feature of charter s...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I

Where are New York City's char...
NEW YORK CITY'S CHARTER SCHOOLS
                                                   CHAPTER I

What are the charter schools...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I

Wildcat Academy charter school...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I

proportion are CMOs and even s...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I


                             ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I

one school are hired by anothe...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I

vast majority attend a school ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                    CHAPTER I

unionization in place. Sinc...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I




                           ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I




                           ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER I


                             ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER II




CHAPTER II. THE STUDENTS O...
NEW YORK CITY'S CHARTER SCHOOLS
                                                    CHAPTER II

The bottom line is that wh...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER II


                            ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER II


Prior test scores of New Yor...
NEW YORK CITY'S CHARTER SCHOOLS
                                                     CHAPTER II

                         ...
NEW YORK CITY'S CHARTER SCHOOLS
                                                       CHAPTER II

attending charter schoo...
NEW YORK CITY'S CHARTER SCHOOLS
                                                 CHAPTER II

The main conclusion to draw f...
NEW YORK CITY'S CHARTER SCHOOLS
                                                CHAPTER III




CHAPTER III. THE SUPERIORI...
NEW YORK CITY'S CHARTER SCHOOLS
                                                CHAPTER III

that only some percentage of ...
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009
Upcoming SlideShare
Loading in …5
×

Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009

1,327 views
1,048 views

Published on

Published in: Education, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,327
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Education, Hoxy Study, How Nyc Charter Schools Affect Achievement Sept2009

  1. 1. The New York City Charter Schools Evaluation Project How New York City’s Charter Schools Affect Achievement September 2009 reporting on results through the 2007-08 school year Principal Investigators: Caroline M. Hoxby, Sonali Murarka, Jenny Kang
  2. 2. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL HOW NEW YORK CITY'S CHARTER SCHOOLS AFFECT ACHIEVEMENT Caroline M. Hoxby, Sonali Murarka, and Jenny Kang Suggested Citation: Hoxby, Caroline M., Sonali Murarka, and Jenny Kang. “How New York City's Charter Schools Affect Achivement, August 2009 Report.” Second report in series. Cambridge, MA: New York City Charter Schools Evaluation Project, September 2009. The New York City Charter Schools Evaluation Project 434 Galvez Mall Stanford, CA 94305 charterevaluation@gmail.com Principal Investigators: Caroline M. Hoxby, National Bureau of Economic Research and Stanford University Sonali Murarka, The Wharton School of the University of Pennsylvania Jenny Kang, National Bureau of Economic Research This research was funded by the Institute for Education Sciences under Contract R305A040043, a subcontract of the National Center on School Choice at Vanderbilt University. We are thankful to the New York City Department of Education, especially Jennifer Bell-Ellwanger and Janet Brand, for their assistance in providing us with administrative data. The New York City Charter School Center and the individual charter schools participating in the study have been indispensable for their cooperation in compiling data. We gratefully acknowledge grant and administrative help from staff of the National Bureau of Economic Researh and from the National Center on School Choice, Vanderbilt University. We also gratefully acknowledge excellent research assistance from Natalie Cox, Ryan Imamura, Christina Luu, Conrad Miller, Brendon Pezzack, and Ardalan Tajalli. The authors are responsible for the content of this report. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL i
  3. 3. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL TABLE OF CONTENTS Bibliographic information i Table of contents ii Introductory Material The New York City Charter Schools Evaluation Project v Executive summary of the report vii Chapter I: New York City's Charter Schools ! When did New York City's charter schools open? I-1 ! Which New York City charter schools are participating in the study? ! How are students admitted to New York City's charter schools? I-2 ! Is New York City a typical environment for charter schools? ! What grades are served by New York City's charter schools? • Table Ia: Charter school applicants by grade I-3 ! Where are New York City's charter schools located? I-4 • Figure Ia: Map of New York City charter schools ! What are the charter schools' neighborhoods like? I-5 • Table Ib: Charter school neighborhoods compared to New York City as a whole ! Who authorizes New York City's charter schools? I-6 • Figure Ib: Charter school authorizers ! Who operates New York City's charter schools? I-7 • Figure Ic: Charter school operating agencies ! Do all charter schools have the same mission? • Figure Id: Charter school missions I-8 ! Does each charter school have its own policies and practices? • Table Ic: Policies and characteristics of New York City charter schools I-11 • Table Id: New York City charter schools, in order of when they opened I-12 Chapter II: The Students of New York City's Charter Schools ! The race, ethnicity, and gender of New York City's charter school applicants II-1 • Table IIa and Figure IIa: The race, ethnicity, and gender of charter school applicants and students in the traditional public schools II-2 ! Prior test scores of New York City's charter school applicants II-4 • Table IIb: Prior test scores of charter school applicants and students in the traditional public schools II-5 ! Free and Reduced-Price Lunch, special education, and English Learner services participation of New York City's charter school applicants • Table IIc: Prior program participation of charter school applicants and students in the traditional public schools II-6 ! Summing up: charter school applicants II-7 THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL ii
  4. 4. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL Chapter III: The Superiority of Lottery-Based Analysis ! Why is lottery-based evaluation the gold standard? III-1 ! Guaranteeing an "apples-to-apples" comparison ! What about students who are offered a charter school place via a lottery but who decide not to attend? ! What about other evaluation methods, for occasions when lottery-based analysis is not available? III-2 ! How about analyzing gains in achievement? ! Are there any evaluation methods that should not be used? ! How do we know which evaluation methods work well? III-4 ! What is the bottom line on methods? III-5 ! How exactly do we use lottery-based data to estimate charter schools' effects? ! For the technically inclined Chapter IV: The Effects of New York City's Charter Schools on Achievement ! A preview of the main findings IV-1 ! Why test scores and Regents diplomas? IV-2 ! Are the estimates representative of New York City's charter schools? • Table IVa: Number and grades of charter schools IV-3 • Table IVb: Number of students available for assessing the achievement effects of New York City's charter schools IV-4 ! Lottery-based results: the effects of New York City's charter schools on math and English test scores in grades 3 through 8 IV-5 • Tables IVc and IVd: Lottery-based estimates of the effect of attending New York City's charter schools on math and English language arts IV-6 • Figure IVa: Estimate-based math progress of lotteried-out students versus students who attend charter schools IV-8 • Figure IVb: Estimate-based English language arts progress of lotteried-out students versus students who attend charter schools IV-9 ! Effects of New York City's charter schools on students of different types IV-10 • Figure IVc: Estimated annual effect of charter schools on math and English Scores, black versus Hispanic students IV-10 • Figure IVd: Estimated annual effect of charter schools on math and English Scores, black versus Hispanic students IV-11 ! What is the achievement of the lotteried-out students? IV-12 • Figure IVe: Math progress of lotteried-out students versus proficiency standard and "Scarsdale standard" IV-13 • Figure IVf: English progress of lotteried-out students versus proficiency standard and "Scarsdale standard" IV-14 ! Lottery-based results: the effects of New York City's charter schools on science and social studies test scores in grades 4, 5, and 8 IV-15 • Table IVe: Lottery-based estimates of the effect of attending New York City's charter schools on science and social studies IV-15 THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL iii
  5. 5. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL ! Lottery-based results: the effects of New York City's charter schools on Regents examination scores IV-16 • Tables IVf and IVg: Lottery-based estimates of the effect of attending New York City's charter schools on Regents examination scores IV-16 ! Lottery-based results: the effects of New York City's charter schools on graduating with a Regents diploma IV-19 • Table IVh: Lottery-based estimates of the effect of attending New York City's charter schools on graduating with a Regents diploma IV-19 ! Do all of New York City's charter schools have similar effects? IV-21 • Figure IVg: Distribution of charter schools' effect on math IV-21 • Figure IVh: Distribution of charter schools' effect on English IV-22 Chapter V: Associating Charter Schools' Effects with their Policies ! What method do we use for this investigation? V-1 ! Associations, not causation ! What kinds of answers can we give? ! Which is more informative, one-variable or multiple-variable regression? V-2 ! What's the bottom line on the association between achievement and charter policies? V-3 ! The long school year ! Other associations between achievement effects and charter school characteristics V-4 ! Charter school characteristics, considered one by one • Figure Va: Associations between charter schools' characteristics and their effects on achievement V-5 ! Summing up V-10 Chapter VI: Students who Leave Charter Schools or Leave the Study ! Students who leave charter schools and return to the traditional public schools VI-1 ! Students who leave the study • Table VIa: Probability that student has left the study, overall and by reason VI-2 • Table VIb: Effect on probability that student has left the study for any reason other than graduation VI-3 Appendix ! Endnotes A-1 ! Appendix Figure 1: Relationship between scale scores and performance levels, math A-2 ! Appendix Figure 2: Relationship between scale scores and performance levels, English Language Arts ! Appendix Figure 3: Relationship between scale scores and performance levels, science A-3 ! Appendix Figure 4: Relationship between scale scores and performance levels, social studies ! Frequency Asked Questions A-4 THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL iv
  6. 6. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT The New York City Charter Schools Evaluation Project is a multi-year study in which nearly all of the city's charter schools are participating. This is the second report in the study and analyzes achievement and other data from the 2000-01 school year up through the 2007-08 school year. The next report in the study will analyze achievement up through the 2008-09 school year. The previous report (July 2007) and a technical report may be downloaded from the following site: www.nber.org/~schools/charterschoolseval. This report (August 2009) analyzes the achievement of 93 percent of the New York City charter school students who were enrolled in test-taking grades (grades 3 through 12) in 2000-01 through 2007-08. The remaining students are not covered by this report for one of two reasons. 5 percent of charter school students in test-taking grades were enrolled in schools that opened from 2006-07 onwards. Their achievement will be covered by the next report of the New York City Charter Schools Evaluation Project. 2 percent of charter school students in test-taking grades were enrolled in schools that declined to participate in the study. The most distinctive feature of the study is that charter schools' effects on achievement are estimated by the best available, "gold standard" method: lotteries. 94 percent of charter school students in New York City are admitted to a school after having participated in a random lottery for school places. This is because the city's charter schools are required to hold lotteries whenever there are more applicants than places, and the charter schools are routinely oversubscribed. In a lottery-based study like this one, each charter school's applicants are randomly divided into the "lotteried-in" (who attend charter schools) and the "lotteried-out" (who remain in the regular public schools. These two groups of students are essentially identical at the time of the lottery. They are not identical just on dimensions that we can readily observe, such as race, ethnicity, gender, poverty, limited English, and disability. They are also identical on dimensions that we cannot readily observe like motivation and their family's interest in education. The lotteried-in and lotteried-out students who participated in the same lottery are identical on these subtle dimensions because they all applied to the charter school. They are separated only by a random number. We follow the progress of lotteried-in and lotteried-out students. We compute the effect that charter schools have on their students' achievement by comparing the lotteried-in students to their lotteried- out counterparts. This is a true "apples-to-apples" comparison. Lottery-based studies are scientific and reliable. There are no other methods of studying the achievement of charter school students that have reliability that is "in the same ballpark" (details below). The New York City Charter Schools Evaluation Project reports on the city's charter schools in the aggregate. We do not identify individual charter schools with their individual results. However, we do describe the variation in charter schools' performance in this report, and we show the association between charter schools' policies and their effects on achievement. In general, it is important to remember that charter schools differ, and no charter school is a mirror image of the aggregate results. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL v
  7. 7. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL The New York City Charter Schools Evaluation Project is funded by a grant from the Institute for Education Sciences, which is the research arm of the United States Department of Education. The study would not be possible without the generous cooperation and help of the New York City Department of Education, the New York City Charter School Center, and the charter schools located in New York City. More information about the project may be found in the Frequently Asked Questions. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL vi
  8. 8. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL EXECUTIVE SUMMARY The distinctive feature of this study is that charter schools' effects on achievement are This is a true "apples-to- estimated by the best available, "gold standard" method: lotteries. 94 percent of apples" comparison. charter school students in New York City are Lottery-based studies are admitted to a school after having participated scientific and reliable. There in a random lottery for school places. In a lottery-based study like this one, each charter are no other methods of school's applicants are randomly divided into studying the achievement of the "lotteried-in" (who attend charter schools) and the "lotteried-out" (who remain in the charter school students that regular public schools. These two groups of have reliability that is in the students are identical not just on dimensions same ballpark. that we can readily observe, such as race, ethnicity, gender, poverty, limited English, and disability. They are also identical on dimensions that we cannot readily observe like motivation and their family's interest in education. The lotteried-in and lotteried-out students who participated in the same lottery are identical on these subtle dimensions because they all applied to the charter school. They are separated only by a random number. We follow the progress of lotteried-in and lotteried-out students. We compute the effect that charter schools have on their students' achievement by comparing the lotteried-in students to their lotteried- out counterparts. This is a true "apples-to-apples" comparison. Lottery-based studies are scientific and reliable. There are no other methods of studying the achievement of charter school students that have similar reliability. The key findings of this report are as follows. ! Charter school applicants are much more likely to be black and much less likely to be Asian or white than the average student in New York City's traditional public schools. [Chapter II] ! Charter school applicants are more likely to be poor than the average student in New York City's traditional public schools. [Chapter II] ! Charter schools' lotteries appear to be truly random, as they are designed to be. Our tests for randomness are based on students' race, ethnicity, gender, prior test scores, free and reduced-price lunch participation, special education participation, and English Learner status. [Chapter II] ! Students who actually enroll in charter schools appear to be a random subset of the students who were admitted. [Chapter II] THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL vii
  9. 9. NEW YORK CITY'S CHARTER SCHOOLS INTRODUCTORY MATERIAL EXECUTIVE SUMMARY, CONTINUED ! Lottery-based analysis of charter schools' effects on achievement is, by far, the most reliable method of evaluation. It is the only method that reliably eliminates "selection biases" which occur if students who apply to charter schools are more disadvantaged, more motivated, or different in any other way than students who do not apply. [Chapter III] ! On average, a student who attended a charter school for all of grades kindergarten through On average, a student who eight would close about 86 percent of the attended a charter school for all of "Scarsdale-Harlem achievement gap" in math and 66 percent of the achievement gap in grades kindergarten through eight English. A student who attended fewer grades would close about 86 percent of the would improve by a commensurately smaller "Scarsdale-Harlem achievement amount. [Chapter IV] gap" in math and 66 percent of the ! On average, a lotteried-out student who achievement gap in English. stayed in the traditional public schools for all of grades kindergarten through eight would stay on grade level but would not close the "Scarsdale-Harlem achievement gap" by much. However, the lotteried-out students' performance does improve and is better than the norm in the U.S. where, as a rule, disadvantaged students fall further behind as they age. [Chapter IV] ! Compared to his lotteried-out counterpart, a student who attends a charter high school has Regents examination scores that are about 3 points higher for each year he spends in the charter school before taking the test. For instance, a student who took the English Comprehensive exam after three years in charter school would score about 9 points higher. [Chapter IV] ! A student who attends a charter high school is about 7 percent more likely to earn a Regents diploma by age 20 for each year he spends in that school. For instance, a student who spent grades ten through twelve in charter high school would have about a 21 percent higher probability of getting a Regents diploma. [Chapter IV] ! The following policies are associated with a charter school's having better effects on achievement. We emphasize that these are merely associations and do not necessarily indicate that these policies cause achievement to improve. • a long school year; • a greater number of minutes devoted to English during each school day; • a small rewards/small penalties disciplinary policy; • teacher pay based somewhat on performance or duties, as opposed to a traditional pay scale based strictly on seniority and credentials; • a mission statement that emphasizes academic performance, as opposed to other goals. [Chapter V] THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL viii
  10. 10. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I CHAPTER I. NEW YORK CITY'S CHARTER SCHOOLS When did New York City's charter schools open? The New York State Charter Schools Act of 1998 authorized the establishment of charter schools in New York State. The first year of operation for charter schools in New York City was 2000-01, and twelve schools began operating that year. Four of these schools were converted from previously-existing public schools: Wildcat Academy, Renaissance, KIPP Academy, and Beginning with Children. By the 2005-06 school year, 36 more charter schools had opened (including one conversion school, Future Leaders Institute) and one charter school had closed. As of the writing of the report, New York City has 78 charter schools operating and another 26 whose operations are scheduled to begin in fall 2009 or fall 2010. Which New York City charter schools are participating in this study? Nearly all of New York City's charter schools are participating in the study. The study contains multiple reports, including a technical report of July 2007 that we will mention several times. All the reports can be downloaded from the website listed on page i. This report analyzes achievement data up through 2007-08, and it analyzes charter schools that were operating as of the 2005-06 school year. These data represent 93 percent of charter school students who were in test-taking grades (grade three through twelve) up through 2007-08. There are a few schools that were operating in 2005-06 and that are not covered by this report. Two schools, UFT Elementary Charter School and South Bronx Charter School for International Cultures and the Arts, declined to participate in the study. Readnet Bronx Charter School closed in 2005-06. The NY Center for Autism Charter School is not included in the study because it serves a very special population and is not compatible with many elements of the study. A full list of New York City's charter schools is in Table Id at the end of this chapter. They are listed by their year of opening. All of the schools that opened by 2005-06 are included in this report, with the exception of the four named above. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-1
  11. 11. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I How are students admitted to New York City's charter schools? Any student who lives in one of the five boroughs of New York City can apply to its charter schools. The typical charter school application simply asks for the student's name, his date of birth, the parent or guardian's contact information, and the grade to which the student is applying. Some applications ask for more information of this basic type but charter schools never ask students to provide their school transcript, write an essay, or otherwise demonstrate their academic performance. Think of a single sheet of paper, not a college application. Charter schools are required to take all applicants if they have space for them. (Charter Charter schools are required to take schools that specifically serve disabled students all applicants if they have space for or drop-outs only consider applications from them.... If a charter school does not students who fit their service model.) If a charter school does not have enough space for have enough space for all all applicants, it is required to hold a random applicants, it is required to hold a lottery among the applicants. 94 percent of random lottery among the students who apply to New York City charter schools are put into one of these lotteries. applicants. 94 percent of students who apply to New York City charter In this study, we are to check whether the schools are put into one of these lotteries are random, and we find no evidence that they are not. lotteries. A lottery is specific to a school and a grade. For example, Explore Charter School may hold one lottery for its kindergarten places, another for its first grade places, and so on. A student may apply to more than one charter school in a single year, but we find that this is not at all common. Much more detail on the lotteries can be found in our technical report. Is New York City a typical environment for charter schools? Nothing about New York City is typical! However, we shall see that its charter schools enroll a disadvantaged population of students that is very similar demographically to charter school students you would see in other cities in the U.S. Also, nearly all charter schools in the U.S. hold admissions lotteries, as New York City's schools do, when they are oversubscribed. Most urban charter schools are routinely oversubscribed, as are New York City's charter schools. There are a few things that are distinctive about the city's charter school environment. The Chancellor of the New York City schools, Joel I. Klein, is supportive of charter schools, and his Department of Education has an Office of Charter Schools that performs important functions. Also, New York City charter schools often share buildings or campuses with traditional public schools. What grades are served by New York City's charter schools? Of the charter schools covered by this report, the majority serve elementary or middle school grades. Less than a quarter serve high school grades. We describe the grades served in more detail later in the report. (See Table IIIa.) THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-2
  12. 12. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I A typical feature of charter schools in New York City is that they open with only their lowest grade, the “intake” grade, and add a grade each subsequent year. This is known as "rolling-up." For example, a charter high school may open with only ninth grade in its first year. In its second year, the ninth graders will roll up to become tenth graders and the school will welcome a new batch of ninth graders, thereby serving ninth and tenth graders. By its fourth year, the school will be serving all of the high school grades from nine to twelve. Because kindergarten and first grade are both traditional intake grades, charter elementary schools in New York City often open with both kindergarten and first grade and then add one grade each year. Thus, they serve kindergarten through grade five by their fifth year of operation. The logic of rolling-up is that it gives schools a manageable way to grow and to instill the school’s culture in students. Charter schools do not always roll up, however. Some open by admitting students into intake and non-intake grades alike. This makes their first year of operation different from subsequent years in which their admissions will be dominated by the intake grades. Typically, non-intake grades admit only a small number of students to fill places that open up when students depart. Conversion charter schools in New York City typically convert to charter school status with their full complement of grades. Table Ia shows that kindergarten alone accounts for a third of all applicants, and kindergarten and first grade by themselves accounts for about half of all applicants. In general, higher grades account for fewer applicants. (Grades five and six are intake grades for middle schools so they have somewhat more applicants than grades four and seven.) Table Ia Grade accounts for this percentage of all applicants Kindergarten 31.6% Grade 1 16.5% Grade 2 9.7% Grade 3 7.6% Grade 4 5.4% Grade 5 13.9% Grade 6 6.9% Grade 7 2.4% Grade 8 1.0% Grade 9 2.5% Grade 10 1.8% Grade 11 0.4% Grade 12 0.2% Note: Table includes the New York City charter schools covered by this report. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-3
  13. 13. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I Where are New York City's charter schools located? New York City's charter schools are generally concentrated in Harlem and the South Bronx. They are scattered throughout Brooklyn, and there are a few in Queens. Starting in the 2009-10 school year, Staten Island will have a charter school. Figure Ia shows a map of New York City with the locations of the charter schools that were operating as the 2008-09 school year. The schools covered by this report are indicated by a red star. The others are indicated by a blue dot. Figure Ia Map of NYC Charter Schools THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-4
  14. 14. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I What are the charter schools' neighborhoods like? When a charter school locates in a neighborhood, it can expect to serve students who are disproportionately from that neighborhood. This is especially true for schools that serve elementary grades. Thus, it is important to know something about the neighborhoods where charter schools locate. We use tract data from the U.S. Census to describe these neighborhoods. (A Census tract is a small, fairly homogeneous neighborhood with about 4000 people living in it.) Comparing the average charter school's census tract to New York City as a whole, we see that charter schools locate in neighborhoods that have unusually low proportions of white and Asian residents and unusually high proportions of black and Hispanic residents. For instance, charter schools' neighborhoods are 50.3 percent black whereas New York City as a whole is only 28.7 percent black. Charter schools' neighborhoods are 37.0 percent Hispanic whereas New York City as a whole is only 27.9 percent Hispanic. Charter schools' neighborhoods are also economically disadvantaged. For instance, the median income of families in charter schools' census tracts is $28,947 while the median income of families in New York City overall is $43,018. 42.0 percent of households in charter schools' neighborhoods have incomes less than $20,000, but only 28.4 percent of New York City households have such low incomes. Table Ib Charter School Neighborhoods Compared to New York City as a Whole Neighborhoods of the New York City charter schools as a Whole Neighborhood Characteristic covered by this report % White (non-Hispanic) 13.9% 33.2% % Black (non-Hispanic) 50.3% 28.7% % Hispanic 37.0% 27.9% % Asian 5.3% 12.5% Median family income $28,947 $43,018 % households with income less than $20,000 42.0% 28.4% % of adults who have no high school diploma or GED 38.4% 28.0% % of adults with bachelor’s degree or higher 17.1% 27.9% % of families with children are single parent families 57.1% 39.2% Source: Authors' calculations based on Geolytics 2008 estimates of U.S. Census of Population and Housing data. Finally, charter schools' neighborhoods are educationally and socially disadvantaged. 38.4 percent of their adults have no high school diploma or GED. In contrast, only 28 percent of New York City adults have such a low level of education. Only 17.1 percent of adults in charter school neighborhoods have a four-year college degree, whereas 27.9 percent of New York City adults have such a degree. Perhaps most dramatic is the difference in the share of families that are headed by single parents. 57.1 percent of families with children are headed by single parents in charter schools' neighborhoods, whereas only 39.2 percent of such families are headed by single parents in New York City as a whole. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-5
  15. 15. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I Wildcat Academy charter school is located in an affluent neighborhood but is exclusively targeted at high school drop-outs. Its downtown Manhattan location makes sense not because its students reside near the school but because they do internships with nearby firms as part of their education. Who authorizes New York City's charter schools? There are three agencies that have the power to authorize new charter schools in the city: the State University of New York (SUNY) trustees, the Chancellor of the New York City Schools, and the New York State Board of Regents. When a group of individuals decides to form a charter school, they submit a proposal to only one authorizer at a time. As Figure Ib shows, most of the charter schools covered by this report were authorized by SUNY or the Chancellor. The Board of Regents authorized only three of the schools. However, the Board of Regents authorizes many charter schools in New York State outside of the city. Figure Ib Charter School Authorizers Who operates New York City's charter schools? Distinct from the authorizer is the operating agency. There are three broadly-defined types of operating agencies in New York City: non-profit Community Grown Organizations (CGOs), non- profit Charter Management Organizations (CMOs), and for-profit Education Management Organizations (EMOs). CMOs and EMOs are formal organizations that exist to manage charter schools. Some larger ones in New York City are the KIPP Foundation (CMO), Achievement First (CMO), and Victory Schools (EMO), all of which operate multiple schools in the city. CGOs, on the other hand, are much more varied. They may consist of a group of parents and teachers, or a community organization that already provides social services to local residents, or an individual from the business world who partners with people working in education. Some CGOs operate multiple schools--for instance, Harlem Children's Zone and Harlem Village Academies. As Figure Ic shows, a little over half of the charter schools covered by this report are CGOs. A smaller THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-6
  16. 16. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I proportion are CMOs and even smaller proportion are EMOs, though the EMOs enroll more students than the CMOs. Conversion charter schools have been grouped under CGOs because many of them were started as non-traditional public schools by groups of parents and teachers or community organizations. Figure Ic Charter School Operating Agencies Do all charter schools have the same mission? Charter schools craft a mission statement that describes their overall vision and focus as a school. Schools with the same operating agency sometimes share the same mission statement, but they do not always do so. A school’s mission statement is a strong indicator to parents, students, school staff members, and the public of the school’s educational philosophy. In New York City, there are several broad educational philosophies held by clusters of schools. Obviously, there is no way to summarize the schools' carefully crafted mission statements in a simple framework, and we encourage people to read each statement for itself. They reveal a variety of thoughtful educational strategies and policies. While we cannot reduce mission statements to simple variables, we can categorize them roughly. Using the statements, we grouped charter school into five broad missions (in descending order of prevalence): a child-centered or progressive philosophy (29 percent of students), a general or traditional educational mission (28 percent of students), a rigorous academic focus (25 percent), a mission to serve a targeted population of students (11 percent of students), and a mission to offer a specific curriculum (7 percent of students). Figure Id shows the proportions of students and schools in each category. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-7
  17. 17. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I Figure Id Charter School Missions While clearly there is much overlap in schools' missions, there are a few key features of schools’ statements that helped us categorize them. Child-centered or progressive schools typically seek to develop students’ love of learning, respect for others, and creativity. Such schools’ mission statements may also focus on helping students realize their potential and on building strong connections between students and their families and communities. Schools with a general or traditional educational mission typically seek to develop students’ core skills and would like to see their students meet or exceed New York State academic standards. Schools with a rigorous academic focus are characterized by mission statements that almost exclusively mention academic pursuits such as excelling in school and going to college. These schools also frequently state that they would like students to become leaders. Schools with a mission to serve a targeted population of students use their statement to describe their target: low-income students, special needs students, drop-outs, male students, female students. The targeted curriculum category contains schools that use a special focus, such as science or the arts, to structure their whole curriculum. Does each charter school have its own policies and practices? We cannot emphasize too often that charter schools are not all alike. In fact, there are many reasons to expect charter schools to differ. They are independent and fairly autonomous. Their founding groups or agencies have a variety of histories. They are most often start-ups and therefore more likely to experiment with new policies than are established schools. On the other hand, there are some reasons to think that charter schools will share certain policies. We have seen that they commonly serve disadvantaged students; they are all under pressure to attract parents and to satisfy their authorizers; they may imitate one another consciously (as when they purposely adopt another school's policy that seems to be working) or unconsciously (as when teachers who have worked at THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-8
  18. 18. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I one school are hired by another and bring their knowledge with them). It is no surprise, therefore, that charter schools in New York City exhibit a variety of policies, but that there are still distinguishable patterns. Table Ic shows the policies of the charter schools covered by this year's report. Of course, we are only able to include characteristics that can be measured objectively with data that are available for most schools. Schools have many characteristics that plausibly affect student achievement but that are not measurable. For instance, the leadership style of the school head is important, but it is very difficult to measure in a way that is consistent across schools. There are two things to take away from an examination of charter schools' policies, as shown in Table Ic. First, we can gauge what policy innovations the charter schools have made. Second, we can appreciate the range of policies pursued by charter schools. In Chapter V, we will show that different charter schools have different effects on achievement, and we will attempt to see which policies are associated with more positive effects on achievement. There are a few policies that the vast majority of charter schools in New York City have adopted but that are uncommon in the traditional public schools. 89 percent of charter school students wear school uniforms or follow a strict dress code. 92 percent of charter school students take approximately two internal evaluations each year. An internal evaluation is a test that used to track students' progress and to identify students who need extra or different instruction. Popular tests include Terra Nova, the Iowa Test of Basic Skills, and the Stanford 9. Such evaluations are not required by the state or city. All charter schools administer the New York State standardized exams. 89 percent of students who attend charter schools that serve middle or high school grades experience an advisory system. In an advisory system, a teacher or pair of teachers is assigned to a group of students for an entire school year. Teachers meet frequently (daily or weekly) with their students and are responsible for making sure that each student is making progress and is not "falling through the cracks." Because elementary schools typically assign students to a single teacher for most of the school day, advisory systems would be duplicative and are therefore not used by them. The average charter school student experiences a school year that is 192 days long. In other The average charter school student words, they attend school for an extra two and experiences a school year that is a half weeks each year. (A 180-day school year is used by the traditional public schools.) 192 days long. In other words, It is interesting to note that a small number of they attend school for an extra two charter schools have very extended school and a half weeks each year. years of 200 to 220 days (four to eight extra weeks). The average charter school student experiences a school day that is 8 hours long. This is about 90 minutes more per day than the traditional public schools. The average charter school student learns English language arts (reading) for 112 minutes per day. 90 minutes is the length of the literacy block mandated for elementary school grades by the Children First initiative in New York City. Half the charter school students learn math for ninety or more minutes per day, where regular public elementary schools in New York City are required to have between 60 and 75 minutes of math instruction daily, depending on the grade. The majority of charter school students attend a school that offers Saturday School (sometimes mandatory, sometimes optional), and the THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-9
  19. 19. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I vast majority attend a school that has an after-school program. There are no dominant math or English language arts curricula in the New York City charter schools although substantial minorities of students experience Saxon Math, Everyday Math, SRA reading, or Open Court reading. About a quarter of students experience a curriculum developed by their own school or its operating agency. See the Frequently Asked Questions for short descriptions of each curriculum. Two-thirds of charter school students learn via a direct instruction teaching style in at least their math class. Direct instruction is a method of teaching that emphasizes the explicit introduction of skills through lectures, scripted exercises, or demonstrations. It is often contrasted with methods in which a student learns by doing. These other methods are variously known as exploratory learning, discovery learning, or inquiry-based learning.1 The average class size experienced by charter school students is 23. We are wary of comparing this number to the numbers for traditional public schools that are reported in New York City's annual class size report. Our caution is due to the fact that the charter school number is based on schools' informal self-reports and the traditional public school numbers are based on registers. We conclude that class size is in about the same range for the same grades in the charter and traditional public schools. At least 22 percent of charter school students experience a "small rewards/small punishments" disciplinary strategy. This strategy is based on the idea that rewarding small courtesies and penalizing small infractions is important. Such discipline is usually carried out in the classroom and sometimes employs an explicit system of points. This is in contrast to disciplinary strategies that focus more on preventing or punishing large infractions and that are carried out mainly by administrators above the classroom level. A school may call its disciplinary policy by a variety of names but we classified it as "small rewards/small punishments" if it clearly fit the description given above. Since we erred on the side of not classifying a school if its strategy was hard to characterize, we believe that the 22 percent number understates the share of charter schools with small rewards/small punishments. About half of charter schools students attend a school where parents are asked to sign a contract. These contracts are not legally enforceable, but they may help to set parents' beliefs about what the school expects of them. A typical parent contract specifies expectations about attendance, on-time arrival at school, homework, and similar issues. For about 60 percent of charter school teachers, some of their pay is based on their performance For about 60 percent of charter and duties they undertake. The standard pay school teachers, some of their pay scale experienced by teachers in the traditional public schools is largely based on seniority and is based on their performance and credentials, such as whether a teacher has a duties they undertake. master's degree. The standard scale does allow pay to depend a little on duties but it does not allow pay to depend on performance, such as whether a teacher raises her students' achievement. Although a small minority of New York City charter schools do have unionized teachers, this phenomenon is not as interesting as it seems at first glance. The vast majority of the unionized charter school teachers are in the conversion charter schools, all of which converted with THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-10
  20. 20. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I unionization in place. Since all the conversion schools were experimental and fairly autonomous before they converted, it is not clear that they have ever had typical unionized environments. Many of the policies just described tend to appear in “packages.” For example, it is very common for schools with a long school year also to have a long school day. Because such packages exist, charter schools do fall into types that parents and authorizers should be able to recognize. We discuss the association between schools' policies and their effect on achievement in Chapter V. Table 1c Policies and Characteristics of New York City Charter Schools average for NYC charter school students Years that school has been operating 6 Operated by a Charter Management Organization (CMO) 29% Operated by an Education Management Organization (EMO) 21% Operated by a Community Grown Organization (CGO) 49% Number of days in the school year 192 Number of hours in the school day 8 Saturday school (mandatory for all or certain students) 57% Optional after-school program available 80% Number of minutes of English language arts per day 112 Long mathematics period (90 minutes or more) 50% Saxon math curriculum 39% Scott Foresman math curriculum 8% Everyday Math curriculum 30% SRA reading curriculum 15% Scott Foresman reading curriculum 10% Open Court reading curriculum 25% Core Knowledge curriculum 17% School's/operating agency's own math and language arts curriculum 28% Direct instruction style of teaching 66% Class size 23 Internal evaluations regularly administered 92% Number of internal evaluations per year 2 Student-faculty advisory (middle and high schools) 82% School uniforms or strict dress code 89% Small rewards/small punishments disciplinary philosophy 22% Parent contract 52% Seat on the Board of Trustees reserved for a parent 58% Teacher pay based on performance/duties (not just seniority and credentials) 59% Number of school leaders 2 Notes: Table describes the schools covered by this report. Schools' characteristics are weighted by their 2007-08 enrollment, so the table represents the experience of New York City charter school students. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-11
  21. 21. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I Table 1d New York City Charter Schools, in order of when they opened School Name Year Grades Grades the Opened Offered as of School Plans to 2008-09 Offer Sisulu-Walker CS 1999 K-5 K-5 Amber CS 2000 K-6 K-6 Bronx Preparatory CS 2000 5-12 5-12 Brooklyn CS 2000 K-5 K-5 Community Partnership CS 2000 K-5 K-5 Harbor Sciences and Arts CS 2000 1-8 1-8 John V. Lindsay Wildcat Academy CS 2000 9-12 9-12 Kipp Academy CS 2000 5-8 5-9 Merrick Academy CS 2000 K-6 K-6 Renaissance CS, The 2000 K-12 K-12 Beginning with Children CS 2001 K-8 K-8 Carl C. Icahn CS 2001 K-8 K-8 Family Life Academy CS 2001 K-5 K-5 Harlem Day CS 2001 K-5 K-5 Harriet Tubman CS 2001 K-8 K-8 Explore CS 2002 K-8 K-8 Our World Neighborhood Charter 2002 K-8 K-8 Bronx CS for Better Learning 2003 K-5 K-5 Bronx CS for the Arts 2003 K-6 K-6 Brooklyn Excelsior CS 2003 K-8 K-8 Harlem Village Academy CS 2003 5-10 5-12 KIPP S.T.A.R. College Preparatory CS 2003 5-8 5-9 Bronx CS for Children 2004 K-5 K-5 Bronx CS for Excellence 2004 K-5 K-5 Bronx Lighthouse CS, The 2004 K-6 K-12 Excellence CS of Bedford Stuyvesant 2004 K-5 K-8 Grand Concourse CS of New York 2004 K-5 K-5 Opportunity CS, The 2004 6-11 6-11 Peninsula Preparatory Academy CS 2004 K-5 K-5 Williamsburg Charter High School 2004 9-12 9-12 Achievement First Crown Heights CS 2005 K-3, 5-7 K-9 Achievement First East New York CS 2005 K-3 K-7 Future Leaders Institute CS 2005 K-8 K-8 Girls Preparatory CS of New York (Lower E Side) 2005 K-4 K-5 Harlem Children's Zone/ Promise Academy CS 2005 K-5, 8-9 K-10 Harlem Children's Zone/ Promise Academy II 2005 K-4 K-5 Harlem Link CS 2005 K-4 K-5 Harlem Village Academy Leadership CS 2005 5-8 5-9 THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-12
  22. 22. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I Table 1d, continued New York City Charter Schools, in order of when they opened School Name Year Grades Grades the Opened Offered as of School Plans to 2008-09 Offer Harlem Village Academy Leadership CS 2005 5-8 5-9 Hellenic Classical CS 2005 K-7 K-7 Kipp AMP (Always Mentally Prepared) CS 2005 5-8 5-9 Kipp Infinity CS 2005 5-8 5-9 Manhattan CS 2005 K-4 K-5 South Bronx CS for Int'l Culture & the Arts 2005 K-4 K-5 UFT CS 2005 K-4 K-12 Williamsburg Collegiate CS 2005 5-8 5-9 Achievement First Bushwick CS 2006 K-6 K-8 Achievement First Endeavor CS 2006 5-7 5-8 Community Roots CS 2006 K-3 K-5 Democracy Prep CS 2006 6-8 6-9 East New York Preparatory CS 2006 K-3 K-8 Harlem Success Academy 1 CS 2006 K-3 K-4 Hyde Leadership CS 2006 K-2, 6-8 K-12 International Leadership CS 2006 9-11 9-12 Leadership Prep CS 2006 K-3 K-4 New Heights Academy CS 2006 6-7, 9-11 5-12 Ross Global Academy CS 2006 K-3, 6-8 K-12 South Bronx Classical CS 2006 K-3 K-5 Carl C. Icahn Bronx North CS II 2007 K, 2-3 K-5 Kings Collegiate CS 2007 5-6 5-8 Achievement First Brownsville CS 2008 K-1 K-7 Bedford Stuyvesant Collegiate CS 2008 5 5-12 Bronx Academy of Promise CS 2008 K-2 K-8 Bronx Community CS 2008 K-1 K-8 Bronx Global Learning Institute for Girls CS 2008 K-1 K-5 Brooklyn Ascend CS 2008 K-2 K-12 Carl C. Icahn South Bronx CS III 2008 K, 1 K-5 DREAM CS 2008 K-1 K-8 Green Dot NY CS 2008 9 9-12 Harlem Success Academy 2 CS 2008 K-1 K-5 Harlem Success Academy 3 CS 2008 K-1 K-5 Harlem Success Academy 4 CS 2008 K-1 K-5 La Cima CS 2008 K-1 K-5 Mott Haven Academy CS 2008 K-1 K-8 THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-13
  23. 23. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER I Table 1d, continued New York City Charter Schools, in order of when they opened School Name Year Grades Grades the Opened Offered as of School Plans to 2008-09 Offer NYC Charter High School for Architecture, Engineering, 2008 9 9-12 & Construction Industries PAVE Academy CS 2008 K-1 K-5 St. HOPE Leadership Academy CS 2008 5-7 6-12 VOICE CS of New York 2008 K-1 K-8 Academic Leadership CS 2009 K-5 Believe Northside CS 2009 9-12 Believe Southside CS 2009 9-12 Brooklyn Prospect CS 2009 6-12 Brooklyn Scholars CS 2009 K-8 Brownsville Ascend CS 2009 K-6 Brownsville Collegiate CS 2009 5-9 Carl C. Icahn CS IV 2009 K-6 Coney Island Prep CS 2009 5-12 Crown Heights Collegiate CS 2009 5-8 Equality CS 2009 6-11 Equity Project CS, The 2009 5-8 Ethical Community CS 2009 K-12 Excellence CS for Girls 2009 K-8 Explore II CS 2009 K-8 Fahari Academy CS 2009 5-12 Girls Preparatory CS of East Harlem/Bronx 2009 K-4 Growing Up Green CS 2009 K-5 Hebrew Language Academy CS 2009 K-5 John W. Lavelle Preparatory CS 2009 6-12 Leadership Prep. East New York/Brownsville CS 2009 K-8 Summit Academy CS 2009 6-12 Achievement First North Crown Heights CS 2010 not yet known East New York Collegiate CS 2010 not yet known Leadership Preparatory Brownsville CS 2010 not yet known Leadership Preparatory Flatbush CS 2010 not yet known THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL I-14
  24. 24. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER II CHAPTER II. THE STUDENTS OF NEW YORK CITY'S CHARTER SCHOOLS In this chapter, we look at who applies to New York City charter schools. How do they compare to New York City students as a whole? How do they compare to the students who were lotteried-in? Finally, how do they compare to the students who ultimately chose to enroll in charter schools? There are no right or wrong answers to these questions. There is no group of students who is supposed to apply to charter schools and there is no group of students who is not supposed to apply. Furthermore, students are not supposed to attend just because they apply and are offered a place. This report analyzes achievement results up through 2007-08 and includes 93 percent of all applicants who were taking tests in that year or a previous year. However, this report does not include students who applied to a charter school in 2006 or 2007, most of whom were kindergarteners, first, or second graders in 2007-08. Thus, throughout this chapter, when we refer to the "most recent year of applicants," we are referring to 2005-06. The race, ethnicity, and gender of New York City's charter school applicants Before looking at statistics on the race and ethnicity of charter school applicants in New York City, it is important to discuss a fact that often causes confusion. As will be seen, New York City's charter schools draw from a student population that is overwhelmingly black and Hispanic. Therefore, if a school's applicants are disproportionately more black, they will automatically be disproportionately less Hispanic. If a school's applicants are disproportionately more Hispanic, they will automatically be disproportionately less black. This is somewhat confusing because, in most areas of the U.S., when we hear that a school is more black or more Hispanic, we (correctly) assume that this means that the school is less white. However, in the neighborhoods from which New York City charter schools draw, this assumption would be incorrect. The white share of these neighborhoods is so small that a school cannot become much more black or much more Hispanic by becoming much less white. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL II-1
  25. 25. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER II The bottom line is that when we say that a charter school is more black, we are automatically saying that it is less Hispanic, and vice versa. If a school is more black and less Hispanic, these are not two separate findings: they are two ways of stating the same finding. Table IIa shows the race, ethnicity, and gender of students who applied to New York City's charter schools, students who were lotteried-in, and students who actually enrolled in the charter schools. The table also shows a column for New York City as a whole. By comparing charter school applicants to this column, we learn how the presence of charter schools is changing the population of students who attend the city's whole system traditional public schools. To ensure that the columns of Table IIa can be compared, we used data from the same school years (2000-01 to 2005-06) to construct each column. Table IIa The Race, Ethnicity, and Gender of Charter School Applicants and Students in the Traditional Public Schools All applicants to Applicants Applicants New York charter schools who were who enrolled City's lotteried-in in charter traditional schools public schools % black non-Hispanic 63 64 61 34 % white non-Hispanic 4 4 4 15 % Hispanic 29 28 29 38 % Asian 3 3 4 12 % other race <1 <1 <1 <1 % female 50 50 52 50 Note: The table includes data for all years of applicants covered by this report: 2000-01 to 2005-06. Table IIa shows that New York City's charter school applicants were very largely black (63 New York City's charter school percent) and Hispanic (29 percent). Only a few applicants were very largely black percent were white, Asian, or another race. (Because percentages are rounded to the (63 percent) and Hispanic (29 nearest whole number, the race and ethnicity percent). Only a few percent were percentages may not add up to 100.) The white, Asian, or another race. lotteried-in applicants look just like the applicants as a whole. Formally, there are no statistically significant differences between the lotteried-in applicants and lotteried-out applicants. This confirms that the lotteries were indeed random. The students who enrolled in charter schools also look like a random subset of applicants. They were nearly all black (61 percent) or Hispanic (29 percent). Formally, the students who enrolled are not statistically significantly different than those who applied. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL II-2
  26. 26. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER II Figure IIa Race and Ethnicity of Charter and Traditional Public School Students Comparing the charter school students to traditional public schools citywide (Figure IIa), we see that charter school applicants were almost twice as likely to be black as the average traditional public school student. As already discussed, saying that a student is more likely to be black is roughly equivalent to saying that he is less likely to be Hispanic, and this is indeed true. Because they were more likely to be black, charter school applicants in our study were about 10 percentage points less likely to be Hispanic than the average traditional public school student. Summing up, New York City's charter school students are disproportionately black and The existence of charter schools in disproportionately not white or Asian. The the city leaves the traditional public existence of charter schools in the city therefore leaves the traditional public schools schools less black, more white, and less black, more white, and more Asian. more Asian. Charter school applicants, lotteried-in applicants, and charter school enrollees are all about equally likely to be male and female. So too are students in New York City's traditional public schools. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL II-3
  27. 27. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER II Prior test scores of New York City's charter school applicants People are often interested in the prior test scores of students who apply to charter schools because they would like to know whether a district's high-achievers or low-achievers are disproportionately applying to charter schools. Unfortunately, there is a serious problem with studying the prior test scores of charter school applicants. Because students do not take tests until grade three, we only have prior test scores for students who apply to grades four or higher from another New York City public school. This gives us prior test scores for only 22 percent of applicants. There is just no guarantee that such applicants are typical of charter school applicants. Logically we expect them to be atypical because they are disproportionately students who are not applying to an intake grade. It would be unwise to use data on only 22 percent of applicants to draw conclusions about how charter schools affect the student population that attends traditional public schools. It is simply impossible to compare the prior test scores of the average charter school applicant to the average student in New York City. We can, however, compare the prior test scores of charter school applicants to those who are lotteried-in and to those who enroll in charter schools. These are apples-to-apples comparisons, and they show that the lotteries are random. With these caveats in mind, examine Table IIb. It shows the prior test scores of charter school applicants, lotteried-in students, and students who enroll in charter schools. All of the scores are expressed as standard scores. A standard score is created by subtracting the New York City-wide mean score from a student's score and dividing the resulting difference by the New York City-wide standard deviation. This procedure puts all scores on the same basis. (The New York City-wide means and standard deviations are specific to each grade tested and each year of testing.) Standard scores are very commonly used in education studies. They also called z-values, z-scores, normal scores, and standardized variables. Expressing an achievement change in standard scores or standard deviations is often called the "effect size." Standard scores are ideal for studies like this, where we follow students over several grades and several school years. With the standard scores, we can make computations that are unaffected by superficial changes in the way New York State scored tests. Also, effects that are expressed in standard scores can be compared to the effects of any other policy that might affect achievement. If a standard score changes by 1, that means that a student's score has changed by a whole standard deviation. A standard deviation is a large difference in achievement. On most tests it corresponds to more than a grade's worth of learning and more than a performance level. Because the New York City-wide mean and standard deviation are used to create the standard scores, the traditional public school system will always have an average standard score of zero. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL II-4
  28. 28. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER II Table IIb Prior Test Scores of Charter School Applicants and Students in the Traditional Public Schools All applicants to Applicants who Applicants who New York City's charter schools were enrolled in charter traditional public lotteried-in schools schools Math standard -0.002 -0.014 -0.012 0 score English language -0.007 -0.006 -0.007 0 arts standard score Note: None of the standard scores shown in three applicant-based columns of the table is statistically significantly different from any of the other standard scores shown in these columns. The data are for all years of applicants covered by this report: 2000-01 to 2005-06. Because about 80 percent of charter school applicants have not taken a prior test when they apply, the above table is not representative of New York City charter school applicants. It is accurate for the subset of applicants who do have a prior test score. The main conclusion to draw from Table IIb is that, as regards the lotteries for which prior test scores are available, the lotteries were indeed random. Also, the students who actually enrolled in charter schools were a random subset of those who were lotteried-in. The math standard score of applicants is -0.002, which is not statistically significantly different from the math standard score of lotteried-in students (-0.014), which is again not statistically significantly different from the math standard score of students who enrolled in charter schools (-0.012). The same can be said for the English language arts scores. For the reasons given above, it is not possible to draw conclusions about how charter school applicants' achievement compares to that of students in New York City's traditional public schools. Free and reduced-price lunch, special education, and English Learner services participation of New York City's charter school applicants It is also interesting to know whether charter school applicants participate in the National School free and reduced-price Lunch program, special education, and/or services for English Learners. (Participation in the National School Lunch Program is often used a rough proxy for income because a student generally cannot participate if his family's income is higher than 185 percent of the federal poverty line. This threshold is currently $40,792 for a family of four.) It would be nice if schools had some objective, consistent way of recording whether students were eligible for these programs because we would then know which students were poor, which were disabled, and which were not native English speakers. Unfortunately, eligibility is not what is recorded. Instead, schools record whether students participate in the programs. Participation is not the same thing as eligibility, and participation is influenced by the school the student attends. Our previous report describes, in detail, the numerous ways in which participation measures are problematic when some students are attending traditional public schools and other students are THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL II-5
  29. 29. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER II attending charter schools. The problem boils down to the fact that the traditional public schools and charter schools not only do not record eligibility, they do not even record participation in the same way. Because of these measurement difficulties, we will look at charter school applicants' program participation at the time they applied. By focusing on the time when they applied, we can see how participation was recorded by the traditional public school system. This allows us to perform comparisons on data that were recorded in a consistent manner. Unfortunately, although examining program participation at the time students applied to charter schools is helpful to some extent, we are left with the problem that the modal charter school applicant does not have a history of program participation. This is because he is applying to kindergarten or first grade (see Table Ia) and therefore has little or no history in the traditional public schools. In short, the data on program participation is not representative of charter school applicants, and we will not be able to say what the average charter school applicant was like. The problem is akin to the problem of not having prior test scores for all applicants. We present information on students' program participation at the time they applied, making a simple adjustment for the fact that charter schools have a disproportionate number of students in their early grades. (For instance, a charter school that is "rolling-up" may only have students in kindergarten through grade three. We need to compare the program participation of its applicants to students who are in the same grades in the traditional public schools. The adjustment is easily accomplished by weighting the data on charter school applicants so that they have the same grade composition as the traditional public schools.2) Table IIc Prior Program Participation of Charter School Applicants and Students in the Traditional Public Schools All applicants Applicants Applicants who New York City's to charter who were enrolled in traditional public schools lotteried-in charter schools schools % who participated in the Free or 92 91 91 72 Reduced-Price lunch program (at the time they applied if applicants) % who participated in special 11 11 11 13 education (at the time they applied if applicants) % who used services for English 4 4 4 14 Learners (at the time they applied if applicants) Note: For the columns dealing with charter school applicants, the participation information is recorded at the time the applicant applied to a charter school. The data are reweighted so that the charter school applicants have the same grade composition as students in the traditional public schools. The table includes data for all years of applicants covered by this report: 2000-01 to 2005-06. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL II-6
  30. 30. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER II The main conclusion to draw from Table IIc is that, as regards applicants for whom a program participation history existed at the time they applied, the lotteries were indeed random. Also, the students who actually enrolled in charter schools were a random subset of those who were lotteried- in. For instance, 92 percent of applicants participated in the free or reduced-price lunch program, 91 percent of the lotteried-in participated in the same program, and 91 percent of students who enrolled in charter schools participated in the same program. These numbers are not statistically significantly different from one another. Similarly, the same percentage of applicants, lotteried-in students, and charter school enrollees participated in special education (11 percent) and in services for English Learners (4 percent). For the reasons given above, it is not possible to draw firm conclusions about how charter school applicants' program participation compares to that of students in New York City's traditional public schools. However, the data suggest that--if anything-- charter school applicants were poorer than the average student in New York City's traditional public schools or the composite feeder school. It appears that they had about the same probability of participating in special education, and they were less likely to be participating in services for English Learners. This last conclusion makes sense because we know that they were less likely to be Hispanic, and most English Learners in the New York City schools are Hispanic. However, all of the above conclusions are tentative because they are not based on representative data. Summing up: charter school applicants The data indicate that charter school applicants are much more likely to be black (and, thus, somewhat less likely to Hispanic) than the average student Charter school in New York City's traditional public schools. Charter school applicants applicants are are also much less likely to be white or Asian than the average student in the city's traditional public schools. It is safe to conclude that charter poorer than school applicants were poorer than the traditional public school students. the average However, on prior test scores, special education, and English learner New York City services, it is impossible to make firm comparisons between the average charter school applicant and the average traditional public school student. public school The numbers suggest that the charter school applicants and traditional student. public school students are fairly similar on these grounds, and it is not possible to be more exact because there are no data that accurately represent The data indicate that the the average charter school applicant. charter schools' lotteries were The data indicate that the charter schools' lotteries indeed random. The data also were indeed random. See the study's previous report indicate that the students who and technical report for more detail on this point. actually enrolled in the charter The data also indicate that the students who actually schools were a random subset of enrolled in the charter schools were a random subset the students who were lotteried- of the students who were lotteried-in. in. THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL II-7
  31. 31. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER III CHAPTER III. THE SUPERIORITY OF LOTTERY-BASED ANALYSIS Why is lottery-based evaluation the gold standard? Lottery-based analysis is by far the best way to determine the effect of a charter school on the achievement of the sort of students who tend to apply to charter schools. In fact, lottery-based analysis produces results that are so much more reliable than those of any other method that no other results should be given any credence when lottery-based results are available. Only when lottery- based results are unavailable should one turn to other methods--and even then only with caution. If a charter school runs a lottery, its effects on achievement should be evaluated via the lottery method, even if this involves gathering some data. Guaranteeing an "apples to apples" comparison While students at traditional public schools normally attend the school closest to where they live, students attending charter schools have specifically chosen to apply to them. Thus, if we simply compare students who attend a charter school to students who attend traditional public schools, we may be comparing "apples to oranges." We will almost certainly confuse evidence on the effects of the charter school with evidence on who selects into the charter school. This is known as selection bias. Lottery-based analysis completely eliminates this bias so long as a charter school holds a random lottery among applicants and has a sufficient number of lotteried-in and lotteried-out students. Intuitively, the reason why lottery-based analysis eliminates bias is that both the lotteried-in and lotteried-out students have applied to the same charter school, so there is no difference in selection between the two groups. What about students who are offered a charter school place via a lottery but who decide not to attend? We conduct a lottery-based analysis that accounts for the fact that some students who are lotteried-in decide not to attend the charter school after all. The basic intuition is this: we adjust the achievement differences between lotteried-in and lotteried-out students to take account of the fact THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL III-1
  32. 32. NEW YORK CITY'S CHARTER SCHOOLS CHAPTER III that only some percentage of lotteried-in students enroll in charter schools. For instance, if 90 percent of lotteried-in students actually enroll in charter schools, the adjustment factor would be 90 percent. This adjustment is the standard procedure in lottery-based studies, including medical trials. Readers who have some technical expertise will want to know that, formally, we estimate treatment- on-the-treated effects by using lotteried-in status as an instrument for enrollment. This gives us the same results as we would get using the Wald method (described in the previous paragraph), but it allows us to test the robustness of the results by controlling for covariates. More information is available in our technical report. What about other evaluation methods, for occasions when lottery-based analysis is not available? When lottery-based analysis is not available, researchers usually compare charter school students to students in traditional public schools, most of whom did not apply to charter schools. These comparisons are usually biased. Researchers may be able to remedy some of the bias by using statistical methods to adjust for differences that we can readily observe between students who do and do not apply to charter schools. Unfortunately, a researcher does not really know when the statistical adjustments are making the bias better and when they are making the bias worse! The only way to know for sure is to compare the statistically adjusted results to results from a lottery- based study of the same data. However, if we have lottery-based results, there is really no point in computing statistically adjusted results. How about analyzing gains in achievement? Lottery-based analysis automatically analyzes gains since we start with two groups who are identical (lotteried-in and lotteried-out) and follow their progress. However, when people say that they are doing "gains analysis," they usually are not referring to lottery-based analysis. As a rule, they are talking about comparing the gains made by charter school students to the gains made by traditional public school students. Comparing gains can be helpful, just as statistical adjustments can be helpful. Unfortunately, like statistical adjustments, comparing gains can make the selection bias worse. We have just described a couple of methods that can work well but do not reliably work well. These two methods (statistical adjustment and comparison of gains) tend to work better when used by a really expert researcher, but it is not easy for people to judge who is expert and who is not. Even in the hands of an expert researcher, these methods can go wrong. Are there any evaluation methods that should not be used? There are a couple of methods that should not be used because, instead of making the selection bias better, they make it dramatically worse. These methods are: (1) pure value-added and (2) matching based on students' prior history in the traditional public schools ("TPS-history-matching"). Both methods have been used by a variety of researchers. We describe them below. First, let us understand the basic problem, something that requires no technical expertise. Think about applying to charter school from the family's point of view. If a family decides to apply for a kindergarten place in a charter school, it is probably thinking that the charter school provides a better THE NEW YORK CITY CHARTER SCHOOLS EVALUATION PROJECT SEPTEMBER 2009 REPORT WWW.NBER.ORG/~SCHOOLS/CHARTERSCHOOLSEVAL III-2

×