- Intervention: A charter middle school in New York City, serving mainly low-income, minority students.
- Evaluation Methods: A well-conducted randomized controlled trial, based on the lottery used to determine which students are offered admission.
- Key Findings: By the end of eighth grade – three years after random assignment – increased students’ math achievement by 0.53 standard deviations (about 1½ grade levels) and English language arts achievement by 0.20 standard deviations (about ¾ of a grade level).
- Other: A limitation of this study is that it was conducted in a single site – one school in New York City. Replication of these findings in a second trial, conducted in another setting, would be desirable to confirm that the intervention is effective across various settings where it might normally be implemented.
Download a printable version of this evidence summary (pdf, 4 pages)
The Top Tier initiative’s Expert Panel has identified this intervention as Near Top Tier.
The Panel finds that this intervention meets the “Near Top Tier” evidence standard, defined as: Interventions shown to meet almost all elements of the Top Tier standard (i.e., well-conducted randomized controlled trials… showing sizable, sustained effects), and which only need one additional step to qualify. This category includes, for example, interventions that meet all elements of the standard in a single site, and just need a replication trial to confirm the initial findings and establish that they generalize to other sites.
Description of the Intervention
The Promise Academy is a charter middle school, serving predominantly low-income, minority students from grades six through eight. The school opened in 2004 as one of the Harlem Children’s Zone programs to improve communities and schools in a 97-block area of Harlem, in New York City. The school provides an extended school day and year, with coordinated after-school tutoring and additional Saturday classes for children struggling in math or English language arts. As a result, students spend 50-100% more time in school per year than students in traditional public schools in New York City, depending on how far behind they are academically. The school emphasizes recruiting and retaining high quality teachers, who are incentivized and evaluated based on their success in raising students’ test scores. Students are consistently reminded of the importance of hard work in achieving success, and are given rewards for achievement, such as money or trips. The school also provides them with free medical, dental, and mental health services; and provides their parents with meals, bus fare, and other benefits.
The school spent $19,272 per pupil during the 2008-2009 school year, compared to an average of $16,144 per pupil in New York City middle schools during that year.1
EVIDENCE OF EFFECTIVENESS
This summary of the evidence is based on a systematic search of the literature, and correspondence with leading researchers, to identify all well-conducted randomized controlled trials of the Promise Academy charter middle school. Our search identified one such trial, as follows.
Overview of Study Design: Randomized controlled trial, based on the 2005 and 2006 lotteries used to determine which students were offered admission to the school.
612 students were randomly assigned via lottery to (i) a group of “lottery winners” offered admission to the Promise Academy in 6th grade; or (ii) a control group of “lottery losers” not offered admission.
83% of sample members were African American, 15% were Hispanic, and 80% were eligible for free or reduced-price lunch. Prior to the study (i.e., 5th grade), they scored about a half-year behind the average New York City student in both math and English language arts achievement.2 69% of the lottery winners accepted the offer of admission and enrolled in the Promise Academy.
Effects three years after random assignment – i.e., end of 8th grade:
These are the Academy’s effects on all outcomes that the study measured, and (unless otherwise noted) on all lottery winners, including those who enrolled in the Promise Academy and those who did not. The effects shown are statistically significant at the 0.05 level unless stated otherwise.
- In math:
- Lottery winners were 27% more likely than lottery losers to be on grade level, as measured by the New York state test.
- Lottery winners scored higher than lottery losers by 0.53 standard deviations, which equates to about 1½ grade levels.3
- The effect grew over the three years of the study (from 0.24 standard deviations in year 1, to 0.26 in year 2, to 0.53 in year 3), and was statistically significant in all three years.
- In English language arts:
- Lottery winners were 15% more likely than lottery losers to be on grade level, as measured by the New York state test.
- Lottery winners scored higher than lottery losers by 0.20 standard deviations, which equates to about ¾ of a grade level.3
- The effect grew over the three years of the study (from 0.02 standard deviations in year 1, to 0.03 in year 2, to 0.20 in year 3). It was statistically significant only in year 3.
- Absences: Lottery winners were absent 3.9 fewer days than lottery losers during the first six months of 8th grade (the time period when absence data are collected by the city).
- Percent entering 8th grade “on-time” for their age: There was no difference between lottery winners and losers on this measure.
The effects on the subsample of lottery winners who actually accepted the admission offer and enrolled in the Academy were 45% larger than the effects shown above.4
Discussion of Study Quality
- The study had a reasonably long-term follow-up: three years after random assignment.
- The lottery winners and lottery losers in the three-year follow-up sample were similar in their observable pre-program characteristics (e.g., demographics, prior academic achievement).
- The study evaluated the Promise Academy as it normally operates in Harlem Children’s Zone in New York City, thus providing evidence of its effectiveness under real-world implementation conditions. We note, however, that the Academy only opened in 2004; thus the effects shown above apply to its early years of operation.
- The study appropriately measured outcomes for all students who won the lottery, regardless of whether or how long they actually attended the Promise Academy (i.e., the study used an “intention-to-treat” analysis).
- The study measured all outcomes using administrative data from the New York City Department of Education, including (for achievement outcomes) student scores on the state tests.
- Study limitations include the following:
- There was moderate sample attrition, which differed somewhat between the program and control groups: outcome data were obtained for 80% of lottery winners and 72% of lottery losers. The study report provides some reassurance that the attrition did not undermine the equivalence of the two groups and lead to erroneous results, as follows: (i) the study analysis shows that the estimated effects change little even under the conservative assumption that the control group’s greater attrition was due to its loss of high-achieving students; and (ii) as noted above, lottery winners and losers in the three-year follow-up sample (i.e., after sample attrition) were similar in their observable pre-program characteristics. Nevertheless, the attrition conceivably could have caused unobservable differences between the two groups, possibly leading to inaccurate estimates of the program’s effects.
- The New York state tests used to measured achievement outcomes have been criticized as being susceptible to preparation that improves students’ test-specific skills but not necessarily their general knowledge or ability.5Ideally, the study would have also measured achievement using another test whose validity is well-established.6
- This study was conducted at a single site – i.e., one school in New York City. The Top Tier initiative’s expert advisory panel believes that replication of the above findings in a second randomized controlled trial, conducted in another setting by the same or other researchers, would be desirable to (i) rule out the possibility that the findings occurred by chance; and (ii) confirm that this intervention is effective in other settings where it would normally be implemented.
SUMMARY OF THE INTERVENTION’S BENEFITS AND COSTS
If taxpayers fund the delivery of this intervention, what benefits to society can they expect to result, and what would be their net cost? The following table provides a summary. This is intended to be a general overview of social benefits in relation to taxpayer cost, rather than a comprehensive benefit-cost analysis. It assigns monetary value to particular benefits and costs only when doing so requires minimal assumptions. The monetary amounts shown are in 2009 dollars.
Benefits To Society
Cost To Taxpayers
(Click on linked authors’ names for their contact information)
- Bloom, Howard S., “Accounting for No-Shows in Experimental Evaluation Designs,” Evaluation Review, vol. 8, April 1984, pp. 225-246.
- Bloom, Howard S., Carolyn Hill, Alison Rebeck Black, and Mark Lipsey, “Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions,” MDRC Working Paper on Research Methodology, October 2008.
- Dobbie, Will and Roland G. Fryer, Jr., “Are High Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone,” May 2010. The full study report is posted here, and is forthcoming in American Economic Journal: Applied Economics. (An earlier version of the report is listed as National Bureau of Economic Research working paper 15473.)
- Medina, Jennifer, “On New York School Tests, Warning Signs Ignored,” New York Times, October 10, 2010, linked here.
- Orr, Larry L., Social Experimentation: Evaluating Public Programs With Experimental Methods, Sage Publications, Inc., 1999, pp. 62-64.
1Both figures are direct service expenditures per pupil, which is a comprehensive measure of expenditures for services provided directly to students during the school year. The source for the Promise Academy estimate is Dobbie et al. 2010. The source for the city average is the New York City Department of Education’s School Based Expenditure Reports.
2 Specifically, they scored 0.25-0.30 standard deviations below the average New York City student in math and English language arts, which equates to approximately a half-year of learning in fifth grade for the typical U.S. student (see Bloom, Hill, Black, and Lipsey 2008, referenced at the end of this summary).
3 Specifically, the average annual achievement gain for U.S. students between the end of seventh grade and the end of eighth grade on seven nationally normed tests is 0.32 standard deviations in math and 0.26 in reading (see Bloom, Hill, Black, and Lipsey 2008, referenced at the end of this summary). The difference in achievement between lottery winners and losers, shown above, is about 1.6 times this amount in math, and 0.75 times this amount in reading.
4 This is known as the “treatment-on-treated” effect, and was calculated using Bloom’s “no-show adjustment.” This adjustment relies on the reasonably-safe assumption that winning the lottery had no effect on educational outcomes for the 31% of lottery winner who never enrolled in the Promise Academy – i.e., the no-shows. More information on the no-show adjustment can be found in Bloom 1984 and Orr 1999, referenced at the end of this summary.
5 See, for example, the New York Times article by Jennifer Medina, referenced at the end of this summary
6 The study does report that that Promise Academy students’ national percentile rank scores on another test – the Iowa Test of Basic Skills – also rose during the study period, albeit by a smaller amount than on the state tests. However, this was not a randomized analysis (this test was not administered to lottery losers), and so can only be considered suggestive.