Why Maintain a Rigorous Standard of Evidence – Lessons Learned
While the term evidence-based is commonly used, the actual level of scientific evidence supporting a given program varies widely, often lacking any credible research standards. Throughout the 20 years of the Blueprints project, the most important lesson that we have learned, and have strived to impart, is that the standards for recognizing an evidence-based program must be high in order to maintain the public's confidence.
In the earliest years of the project, Blueprints identified a model program, Quantum Opportunities, based upon a multi-site evaluation. This evaluation examined outcomes in each of the national sites participating in the study, and the demonstration appeared successful. Later, a large multi-site replication by the Department of Labor found only a few, largely inconsistent effects, and some of the primary behavioral outcomes were negative at one of the replicating sites. This program was subsequently removed from the Blueprints list.
A similar situation occurred with another program, CASASTART, a comprehensive case management strategy for preventing drug use and delinquency for small groups of high-risk adolescents, living in highly distressed neighborhoods. This program was listed on Blueprints as promising based upon one successful large-scale randomized trial conducted by the Urban Institute, but a later randomized controlled evaluation conducted by the Blueprints team found no effects, and some iatrogenic effects. This program was subsequently retracted by the program developers and removed from the Blueprints list. However, many agencies already using the program were left with decisions of whether to continue or abandon the program. As can be imagined, there was much confusion as to how to handle the unexpected news.
In the examples above, the original trials were conducted in a rigorous manner with random assignment, and later studies failed to replicate. This reinforces the need for replication as a part of the evidentiary standard. Replication is a necessary standard to meet the Blueprints Model criteria, and independent replication (conducted by researchers independent of the program developers) is necessary for Model Plus. Although Blueprints Promising programs do not have to have replication if their success was based on a randomized controlled trial, we also suggest that these programs should not be taken to scale in national initiatives, although they may be adopted in local settings.
These examples reinforce the reasoning for maintaining a high standard. We cannot afford to take programs to scale prior to rigorous testing. When programs that have not been adequately tested are added to registries and later found to be ineffective, this shakes the confidence of the public in our ability to identify evidence-based programs. While there is always a chance of such instances occurring, to accept programs with a low standard of evidence only increases the odds. The way to avoid such confusion is for all registries and funders to endorse standards of the highest quality. Programs must demonstrate in high-quality evaluations evidence of a beneficial change in targeted outcomes, sustained effects, multiple site replication, and other prescribed factors before going to scale. Blueprints can help decision makers understand the difference between programs that only claim to be evidence-based and those that meet the true scientific standard.