- Intervention: Small public high schools in New York City, created citywide in mostly high-poverty communities to replace large, low-performing high schools. The small schools compete for students through the city’s system of school choice.
- Evaluation Methods: A large, well-conducted randomized controlled trial.
- Key Findings: Four years after random assignment, a 6-10 percentage point increase in the four-year high school graduation rate, and a 4-6 percentage point increase in the rate of four-year graduation with a New York State Regents diploma (requiring proficiency on each of five state Regents exams in various subjects).
- Other: (i) The small schools’ operating cost is approximately the same as that of the larger, more traditional high schools attended by control group students. (ii) A study limitation is that it evaluated the intervention in a single city that was concurrently implementing other system-wide education reforms. Replication of these findings in a second trial, in another setting and different conditions, would be desirable to confirm the initial results and establish that they generalize to other situations where the intervention might be implemented.
Download a printable version of this evidence summary (pdf, 5 pages)
The Top Tier initiative’s Expert Panel has identified this intervention as Near Top Tier
The Panel finds that this intervention meets the “Near Top Tier” evidence standard, defined as:
Interventions shown to meet almost all elements of the Top Tier standard (i.e., well-conducted randomized controlled trials… showing sizable, sustained effects), and which only need one additional step to qualify. This category includes, for example, interventions that meet all elements of the standard in a single site, and just need a replication trial to confirm the initial findings and establish that they generalize to other sites.
Description of the Intervention
Between 2002-2008, New York City created Small Schools of Choice (SSCs) to replace large public high schools with graduation rates below 45 percent located in disadvantaged communities. SSCs are small (roughly 100-120 students per grade, as compared to the usual 350 or more in traditional city high schools), academically nonselective, and designed to ensure students receive individualized attention from teachers. The schools were newly created through a competitive process, in which the city invited applications from prospective school leadership teams.
Additional SSC features include: (i) new principals and teachers (as opposed to transfers from a large high school that the SSC replaced); (ii) start-up funding from the city’s Department of Education and philanthropic organizations (such as the Bill and Melinda Gates Foundation); (iii) assistance with leadership development, staff hiring, and program start-up from intermediary organizations (such as New Visions for Public Schools); and (iv) partnerships with local businesses or nonprofit organizations that offer students learning opportunities inside and outside the classroom.
SSCs operate in the context of the city’s high school choice system, under which all rising 9th graders rank order their preferences for high schools to attend, and then are placed in their most preferred school with an available spot. Researchers call these high schools “small schools of choice” because they are small and they are open to all students who choose them regardless of the students’ past academic performance.
SSCs’ operating cost is about $58,000 per student over five years (in 2014 dollars), which the study described below found to be approximately the same as that of the larger, more traditional high schools attended by students in the control group.1
EVIDENCE OF EFFECTIVENESS
This summary of the evidence is based on a systematic search of the literature to identify all well-conducted randomized controlled trials of SSCs. Our search identified one such trial. What follows is a summary of the study design and the intervention’s effects on the main outcomes measured in the study, including any such outcomes for which no or adverse effects were found. All effects shown are statistically significant at the 0.05 level unless stated otherwise.
Overview of the Study Design: Randomized controlled trial with a sample of 18,000 rising 9th graders who applied to one of 105 over-subscribed SSCs in New York City during 2005-2008.
The study sample comprised approximately 18,000 students who applied to enter 9th grade in one of 105 oversubscribed SSCs in the city during 2005-2008 (out of a total of 123 SSCs created during these years). These students were randomly assigned via lottery to (i) a group that was eligible to attend the SSC, or (ii) a control group that was not.2
The study estimated the effects of SSCs using the sample of 14,969 students who applied to SSCs during the first three of the four study years – i.e., 2005, 2006, and 2007 – since these were the students for whom four-year outcome data were available for the most recent study report.3 46% of students in this sample were Hispanic, 44% were African American, 70% performed below grade level in reading and 68% did so in math, and 84% were eligible for free or reduced-priced school lunches due to low family income.
Effects of SSCs four years after random assignment, compared to the control group::
- A 6 percentage point increase in the four-year high school graduation rate (69.6% of the SSC group graduated from high school in four years, versus 63.6% of the control group).
- A 4 percentage point increase in the rate of four-year graduation with a New York State Regents diploma, requiring proficiency on each of five state Regents exams in various subjects (44.9% of the SSC group versus 41.3% of the control group).
- A 4 percentage point increase in the rate of students scoring high enough on the Regents exam in English to be considered college ready by the City University of New York (CUNY) system (39.5% of the SSC group versus 35.4% of the control group).
- No significant effect on the percent of students graduating in four years with an Advanced Regents diploma (awarded to exemplary students), or on the percent scoring high enough on the Regents math exam to be considered college ready in math by the CUNY system.4
The effects of winning the SSC lottery, shown above, likely understate the effects of actually attending an SSC, because some of the students who won the lottery did not attend an SSC, and some students who lost the lottery attended an SSC anyway (e.g., by winning a second-stage lottery). The study estimates that the effects of actually attending an SSC (the “treatment on treated” effects) are greater than shown above – namely, a 10 percentage point increase in the four-year high school graduation rate, a 6 percentage point increase in the rate of four-year graduation with a Regents diploma, and a 7 percentage point increase in the college-readiness rate in English. These effects may be valid, but confidence in them is partially tempered by their reliance on an analytical assumption that cannot be directly verified.5
Discussion of Study Quality
- The study had low sample attrition and a reasonably long-term follow-up: Outcome data were obtained for 81% of the SSC group and 81% of the control group, four years after random assignment.
- The SSC and control group students in both the original randomized sample and the four-year follow-up sample were highly similar in their observable pre-program characteristics (e.g., demographics, academic achievement).
- The study appropriately kept sample members in the groups to which they were randomly assigned, when analyzing the intervention’s effects (i.e., the study used an “intention-to-treat” analysis).
- All study outcomes were measured using official administrative records from the New York City Department of Education.
- The study evaluated SSCs as delivered on a very large scale in New York City, thus providing evidence of the intervention’s effectiveness under real world implementation conditions.
- Study limitations: The study evaluated SSCs as implemented in a single city that was concurrently implementing other system-wide education reforms (e.g., centralization of school district governance under the mayor, changes in teacher hiring policies to de-emphasize seniority). The Top Tier initiative’s Expert Panel believes that replication of the above findings in a second trial, conducted in another setting and different conditions, would be desirable to confirm the initial results and establish that they generalize to other situations where SSCs might be implemented.
SUMMARY OF THE INTERVENTION’S BENEFITS AND COSTS
If taxpayers fund the delivery of this intervention, what benefits to society can they expect to result, and what would be their net cost? The following table provides a summary. This is intended to be a general overview of social benefits in relation to taxpayer cost, rather than a comprehensive benefit-cost analysis. It assigns monetary value to particular benefits and costs only when doing so requires minimal assumptions. The monetary amounts shown are in 2014 dollars.
Benefits To Society
Four years after random assignment:
Cost To Taxpayers
- Bifulco, Robert, Rebecca Unterman, and Howard S. Bloom. The Relative Costs of New York City’s New Small Public High Schools of Choice, MDRC, October 2014 (available here).
- Bloom, Howard S., Saskia Levy Thompson, and Rebecca Unterman. Transforming the High School Experience: How New York City’s New Small Schools Are Boosting Student Achievement and Graduation Rates, MDRC, June 2010 (available here).
- Bloom, Howard S., and Rebecca Unterman. “Can Small High Schools of Choice Improve Educational Prospects for Disadvantaged Students?” (2014). Journal of Policy Analysis and Management, vol. 33, no. 2, pp. 290–319.
- Bloom, Howard S., and Rebecca Unterman. Sustained Progress: New Findings About the Effectiveness and Operation of Small Public High Schools of Choice in New York City, MDRC, August 2013 (available here).
- Unterman, Rebecca. Headed to College: The Effects of New York City’s Small High Schools of Choice on Postsecondary Enrollment (an MDRC Policy Brief), October 2014 (available here).
Note: Top Tier Panel members Ron Haskins and Sean Reardon did not participate in the Panel’s review of this intervention.
1 This cost estimate is based on the sample of students who applied to SSCs during the first two study years, for whom five-year outcomes were available. One reason SSCs did not cost more than the schools attended by control group students is that a greater percentage of SSC students graduated in four years, and thus fewer stayed in high school for a fifth year requiring an additional year of school expenditures.
The cost estimate for SSCs does not include the one-time start-up costs of creating an SSC, the ongoing costs of facility usage (e.g., gyms, science labs), nor the cost of resources contributed to SSCs by external partners to support ongoing operations. However, a careful analysis found that these factors are unlikely to change the cost estimate substantially because (i) the one-time start-up costs are relatively small when amortized over many years; (ii) the difference in facility usage costs between SSCs and other schools is minimal; and (iii) external partners’ funding contributions for ongoing operations are relatively small, based on a survey of such partners conducted by New Visions for Public Schools (Bifulco, Unterman, and Bloom 2014).
2 Some of the students who lost the lottery for their first-choice SSC ended up attending an SSC that was not their first choice (e.g., through a subsequent stage of the lottery process). The study appropriately kept these “cross-over” students in the control group when estimating SSC’s effects, consistent with an intention-to-treat approach. As described below, the study also presents “treatment on treated” effects that adjust for these cross-overs.
3 A forthcoming study report will present four-year outcomes for the full sample, as well as longer-term outcomes for students who entered the study during its initial years. A preliminary brief on these findings (Unterman 2014) shows similar effects on high school graduation to those presented here, and positive effects on college enrollment. We plan to revise this summary to include the new results once they are fully reported.
4 The Regents English and math exams are the two Regents exams that the CUNY system uses to determine whether a student has met its proficiency requirements for enrollment in a bachelor’s degree program.
5 An assumption underlying these “treatment-on-treated” effects is that the intervention’s effect on each student in the control group who attended an SSC (i.e., each control group “cross-over” student) is the same as the intervention’s effect on those students who were randomly assigned to the same SSC (i.e., the treatment group students in the same school). This assumption can neither be verified nor disproved.