HIGHLIGHTS
  • Intervention: A low-cost performance pay program for primary school teachers in rural India, which awards them an annual bonus of about $13 for each percentage point gain in their students’ math and language test scores.
  • Evaluation Methods:  Well-conducted randomized controlled trial with a large, representative sample of rural schools in the Indian state of Andhra Pradesh.
  • Key Findings:  The program produced gains in all four subjects measured (math, language, science, and social studies), increasing the average achievement score by between 6 and 13 percentile points in performance pay schools compared to control schools, over a 2-3 year period.
  • Other:  A study limitation is that schools in the sample were all located in one Indian state. Thus, replication of these findings in a second trial, in another setting, would be desirable to confirm the initial results and establish that they generalize to other settings where the intervention might be implemented.

Download a printable version of this evidence summary (pdf, 6 pages)

The Top Tier initiative’s Expert Panel has identified this intervention as Near Top Tier

The Panel finds that this intervention meets the “Near Top Tier” evidence standard, defined as:

Interventions shown to meet almost all elements of the Top Tier standard (i.e., well-conducted randomized controlled trials… showing sizable, sustained effects), and which only need one additional step to qualify. This category includes, for example, interventions that meet all elements of the standard in a single site, and just need a replication trial to confirm the initial findings and establish that they generalize to other sites.

Description of the Intervention

This program, administered by the Azim Premji Foundation (a non-governmental organization), provided teachers in government-run primary schools in rural India (grades 1-5) with financial bonus payments for increasing the math and language achievement of their students. The bonuses were designed to address the inter-related problems of (i) low teacher effort in rural India, such as pervasive absenteeism, and minimal teaching activity even among many of those present [Kremer et. al., 2005]; and (ii) low student achievement as reflected, for example, in the recent finding from an all-India survey of rural households that approximately half of students enrolled in 5th grade cannot read at a 2nd grade level [Pratham 2014].

The bonus payments to teachers were worth approximately $13 for each percentage point gain in their students’ average math and language test scores, compared to the students’ test scores at the end of the previous year. The bonuses were awarded annually, over the five years of the project. A typical teacher in the study earned a base salary plus benefits of about $3,000 per year, and the average annual bonus was calibrated to be about 3% of this amount (roughly $90).1

The cost of the program, including both the bonuses and the administration of the tests used to calculate the bonuses, was approximately $375 per school per year. This equates to a per-student cost of about $3 per year, making it a very low-cost intervention. A more detailed description of the program can be found on pages 50-52 of Muralidharan and Sundararaman, 2011.

EVIDENCE OF EFFECTIVENESS

This summary of the evidence is based on a systematic search of the literature, and correspondence with leading researchers, to identify all well-conducted randomized controlled trials of this specific teacher performance pay program. Our search identified one such trial.2 This trial evaluated the program as implemented in a developing-world setting characterized by both low teacher effort and student achievement, as described above. Thus, the study’s findings apply to the program as implemented in such a setting, and cannot be presumed to apply, for example, to other schools in India or other countries where teachers exert greater effort and students are higher performing.

What follows is a summary of the study design and the program’s effects on the main outcomes measured in the study, including any such outcomes for which no or adverse effects were found. All effects shown are statistically significant at the 0.01 level.

Overview of the Study Design: Randomized controlled trial of the above teacher performance pay program in a representative sample of 200 schools in rural Andhra Pradesh, conducted 2005-2010.

This was a randomized controlled trial of 200 government-run primary schools in the Indian state of Andhra Pradesh, containing 621 teachers and 22,580 students in grades 1-5. Andhra Pradesh is the fifth most populous state in India, and is close to the national average in literacy and school enrollment. The 200 schools were randomly sampled from five rural districts within Andhra Pradesh, so as to be representative of schooling conditions in rural areas of the state. The schools were randomly assigned to (i) a group that received the performance pay program, or (ii) a control group that did not.3

On average, each school employed three teachers and enrolled 113 students in grades 1- 5. The teachers typically taught all subjects for a given grade, and often more than one grade simultaneously. On pre-program tests of the students’ competency at their current grade level, they correctly answered only 18% of math questions and 34% of language questions.

Effects of the performance pay program on student achievement, two school years after random assignment:4

The average score in the performance pay schools was higher than the average score in the control group schools by –

  • 13 percentile points in math (equating to a standardized effect size of 0.32);
  • 9 percentile points in language (equating to a standardized effect size of 0.23);
  • 8 percentile points in science (equating to a standardized effect size of 0.19); and
  • 9 percentile points in social studies (equating to a standardized effect size of 0.22).5

The gains in science and social studies achievement were spillover benefits, since the program only incentivized math and language achievement.

Effects of the performance pay program on student achievement, three school years after random assignment:6

The average score in the performance pay schools was higher than the average score in the control group schools by –

  • 10 percentile points in math (equating to a standardized effect size of 0.25); and
  • 6 percentile points in language (equating to a standardized effect size of 0.16).5,7

Other findings at the two- and three-year follow-ups:

  • The program increased the achievement of students at all levels of ability (as measured by their pre-program test scores).
  • The program produced approximately the same increases in math and language achievement as measured by “mechanical” test questions (which follow the format of standard exercises in the students’ text books) and “conceptual” test questions (which test the same underlying knowledge or skill in an unfamiliar way). This, along with the positive effects on non-incentivized subjects (science and social studies), provides confidence that the test score gains represent genuine improvements in learning.
  • The intervention had no impact on teacher absences, but it did have an impact on other self-reported measures of teacher effort – e.g., amount of classwork and homework assigned, the teaching of classes outside school hours, and the administration of practice tests – illuminating a possible mechanism through which the program increased student learning.

Discussion of Study Quality

  • The study had a reasonably long-term follow-up period – three school years after random assignment.8
  • At the two-year follow-up, the study had no school attrition, minimal teacher attrition,9 and moderate student attrition: Outcome data were obtained for 72% of the students in the performance pay group and 71% of student in the control group. (At the three-year follow-up, the study had higher student attrition, as discussed below, under “study limitations.”)
  • At the start of the study, schools in the performance pay and control groups were highly similar in their observable characteristics (e.g., student enrollment, student/teacher ratio, and students’ math and language test scores).
  • The study appropriately sought outcome data for all schools assigned to the performance pay group, regardless of whether or how long they actually participated in the program (i.e., the study used an “intention-to-treat” analysis).
  • The study measured outcomes using standardized tests that were developed for the study by India’s leading educational testing firm, and measured mechanical as well as conceptual learning, as described above. Tests were proctored and graded by external evaluators unaffiliated with the sample schools, in order to prevent teachers from cheating.
  • The study’s statistical analysis appropriately accounted for the fact that schools, rather than individual students or teachers, were randomly assigned to the performance pay versus control group.
  • The study evaluated the performance pay program as delivered on a large scale in a representative sample of rural schools in Andhra Pradesh, India, thus providing evidence of the program’s effectiveness under real-world implementation conditions.
  • Study limitations

    • Although the program was delivered on a large scale, the participating schools were all located in a single Indian state. The Top Tier initiative’s Expert Panel believes that replication of the above findings in a second trial, conducted in another setting, would be desirable to confirm the initial results and establish that they generalize to other settings where the intervention might normally be implemented.
    • At the three-year follow-up, although the study still had no school attrition and minimal teacher attrition, it did have moderate-to-high student attrition: Outcome data were obtained for just 60% of the original student sample (61% of the performance pay group and 59% of the control group). Statistical tests suggest the attrition did not create systematic differences between the two groups in their observable characteristics (e.g. pre-program test scores). However, the attrition conceivably could have caused unobservable differences between the two groups, possibly leading to inaccurate estimates of the program’s effects. (This limitation does not apply to the findings at the two-year follow-up).

SUMMARY OF THE INTERVENTION’S BENEFITS AND COSTS

If taxpayers (in India and/or the international community) fund implementation of this performance pay program, what benefits to society can they expect to result, and what would be their net cost? The following table provides a summary. This is intended to be a general overview of social benefits in relation to taxpayer cost, rather than a comprehensive benefit-cost analysis. It assigns monetary value to particular benefits and costs only when doing so requires minimal assumptions.

Benefits To Society

The program increased the average achievement score in performance pay schools (compared to control schools) by:

  • 10 percentile points in math, over a three-year period;
  • 6 percentile points in language, over a three-year period;
  • 8 percentile points in science, over a two-year period; and
  • 9 percentile points in social studies, over a two-year period.

Cost To Taxpayers

The program cost approximately $375 per school per year, or about $3 per student per year.

REFERENCES

Main Study

Other References

1 Monetary amounts reported in this summary are 2013 dollars, obtained by converting rupees to dollars, and adjusting for inflation.

2 Our search identified other randomized controlled trials of teacher performance pay programs, but we do not summarize them here because these programs were substantively different than the intervention described above in the design of their incentives (e.g., providing bonuses to teachers based on their whole school’s performance, rather than individual teacher performance) and/or the settings in which they were delivered (e.g., outside the developing world).

3 The study also had three other randomly-assigned groups, each containing 100 schools and implementing a different intervention. Two of the interventions provided additional school inputs (an extra teacher or a cash grant for school materials), and the third provided bonuses to teachers based on their whole school’s performance (as opposed to individual teacher performance). The study found substantially smaller effects for these other interventions; thus, in the remainder of this write-up, we omit the other study groups from both the sample description and the summary of results.

4 These effects were measured for the full sample of students in grades 1-4 at the time of randomization. The students in grade 5 had graduated from primary school at the two-year follow-up, and so were no longer available for testing.

5 These are percentile points in the control group’s distribution of scores.

6 These effects were measured for the full sample of students in grades 1-3 at the time of randomization. The students in grades 4 and 5 had graduated from primary school at the three-year follow-up, and so were no longer available for testing.

7 At the three-year follow-up, the study also found effects on science and social studies achievement (standardized effect sizes of 0.11 and 0.16, respectively). However, unlike the three-year math and language effects shown above, the three-year science and social studies effects were estimated for a sample that included not just the students and teachers randomized at the start of the study, but also many students and teachers who entered the performance pay and control schools in the two years after randomization, potentially undermining the equivalence of the two groups and leading to inaccurate findings. For this reason, we do not include these findings in the evidence summary above.

8 The study did measure the program’s effects over two additional years (i.e., years 4 and 5 after random assignment), and found that the effects in these years grew substantially compared to year 3. However, we believe these findings are only suggestive because of a study limitation in years 4 and 5 – namely, student attrition of approximately 50%, and evidence that the attrition may have partly undermined the equivalence of the performance pay and control groups.

9 Shortly after the study’s launch, approximately one-third of the teachers in the original sample were transferred out of their schools and replaced with other teachers, consistent with the state’s policy of transferring teachers every three years. However, these transfers occurred before the study had announced which teachers were in the performance pay group versus the control group, and therefore were unrelated to group assignment and unlikely to have undermined equivalence of the two groups. The government agreed to minimize such transfers in subsequent years of the study. As a result, additional teacher attrition through the three-year follow-up was less than 5%.