This site’s criteria for identifying evidence-based interventions:
The process and criteria we use for identifying interventions to include on the Social Programs That Work site are similar to those used in the Coalition for Evidence-Based Policy’s Top Tier Evidence initiative, with one key difference noted below. For reference, the Top Tier process and criteria are summarized in the following three bullets, excerpted from the Top Tier website:
- The standard we use to evaluate candidates for the Top Tier, based on the Congressional legislative language, is: “Interventions shown in well-designed and implemented randomized controlled trials, preferably conducted in typical community settings, to produce sizeable, sustained benefits to participants and/or society.”
- In applying this standard, we use the Checklist For Reviewing a Randomized Controlled Trial, which closely tracks guidance from the U.S. Office of Management and Budget (OMB), National Academy of Sciences, and other respected research organizations, and reflects well-established principles on what constitutes a high-quality trial (e.g., adequate sample size, low sample attrition, valid outcome measures, intention to treat analysis, and so on). It also addresses the importance of replication in establishing strong evidence – namely, demonstration of effectiveness in at least two well-conducted trials or, alternatively, one large multi-site trial.
- Our main focus, for each candidate intervention, is on assessing whether there is strong evidence that the intervention’s effects are sizable and sustained. However, in some cases, we might also take into account such factors as the intervention’s cost and ease of implementation (e.g., cases where the cost is exceptionally low) . . . .
- The standard we use to evaluate candidates for Near Top Tier is: Interventions shown to meet all elements of the Top Tier standard (i.e., well-conducted randomized trials…showing sizable, sustained effects), and which only need one additional step to qualify. This category includes, for example, interventions that meet all elements of the standard in a single site, and just need a replication trial to confirm the initial findings and establish that they generalize to other sites. The purpose of this category is to help increase the number of Top Tier interventions, by enabling policymakers and others to identify particularly strong candidates for replication trials whose results, if positive, would provide the final element for the Top Tier.
- For each viable candidate, we search the literature and contact experts to identify all other well-conducted randomized trials of the intervention (in addition to those initially brought to our attention). Also, for interventions being considered for Top Tier or Near Top Tier on the basis of a limited number of well-conducted randomized trials, we check the literature of high-quality non-randomized studies of the intervention, to look for any patterns of effects that differ from those in the trials (possibly suggesting problems in generalizability) or for any adverse intervention effects.
Social Programs That Work differs from the Top Tier website in one important way: it also includes interventions that have been found promising by Coalition staff but have not yet been identified by the Top Tier initiative’s expert review panel as Top Tier or Near Top Tier – in some cases, because of study limitations (such as only short-term follow-up) that indicate the need for additional testing. While we strive to ensure that our staff-level summaries are accurate and balanced, readers should note that they have received less scrutiny than those reviewed by the expert panel for Top Tier or Near Top Tier consideration.
Each of our study summaries includes a short section — “Discussion of Study Quality” — detailing how the study measures up to the above criteria, including any flaws or limitations.
If you are aware of other interventions or studies that meet the above criteria, please let us know (by email or phone at (202) 239-1248).
How this site relates to other “what works” websites:
Our site covers the full spectrum of social policy (K-12 education, crime prevention, international development assistance, etc.), focusing on the relatively few interventions across these areas meeting the strong evidence criteria described above. The site thereby seeks to complement the excellent existing “what works” sites that provide more in-depth coverage of specific policy areas. These other sites tend to be more inclusive, in that they also cover studies that provide suggestive evidence of effectiveness, but which sometimes yield erroneous conclusions – such as comparison-group studies in which the two groups differ in key characteristics, or randomized controlled trials with only a short-term follow-up or key flaws in the trial’s implementation. We recognize that the stronger of these studies can be helpful for decisionmaking in the absence of more definitive evidence, and we therefore link to some of the excellent sites that do include them (click here).
This site was developed by the following Coalition staff, with input from the Coalition’s Advisory Board and others:
- David Anderson, Vice-President
- Jon Baron, President
- Kim Cassel, Project Leader
- Jonathan Crane, Director for Research
- Deborah Gorman-Smith, Professor in the School of Social Services Administration, University of Chicago; Director, Chicago Center for Youth Violence Prevention; and Distinguished Research Fellow with the Coalition