Succeeding With - or Maybe in Spite of - Evidence-Based Practices

We understand the reasoning that allows funding only for proven, evidence-based practices. But too often this requirement has become a club battering community nonprofits. Evaluator Clare Nolan explains how to "tweak" evidence-based practices to your own populations:

Safer sex can be a life and death issue. And many nonprofits make safer sex education the centerpiece of their work. But how do they know whether what they're teaching is working - that lives are being saved?

A San Francisco Tenderloin neighborhood had a safer sex education program modeled after a "proven" intervention being promoted by the Centers for Disease Prevention and Control (CDC). But their own expertise with their population led them to want to change the model. That's why they asked me to design a program evaluation.

As part of my background research, I was surprised to learn that the intervention was first shown to be effective among a primarily gay white population in a small Southern city. Would this intervention really be successful at reducing HIV risk behaviors among residents of a diverse urban neighborhood struggling with poverty, homelessness and crime?

This situation reflects a broader trend in the nonprofit sector in which funders encourage and sometimes require nonprofits to use "evidence-based" practices and models. Evidence-based practices (EBPs) are strategies that have been shown through rigorous research to be effective. The premise sounds great. If there's strong evidence that something works, nonprofits should use it, right?

Not so fast. Models and practices with positive track records are a potentially good tool for those working toward social impact, but there are some things nonprofits should know before jumping onto the evidence-based bandwagon.

Here are five tips for avoiding potential pitfalls while making the most of what EBPs have to offer.

1. Decide if claims that the practice is evidence-based are trustworthy.

These days, there is no shortage of websites providing information on supposedly proven practices or program models for the nonprofit sector. Be skeptical about what these sites have on offer. For example, some individuals are advocating for practices they pioneered in order to increase their own visibility or generate personal revenue.

To determine whether a resource is credible, look to see what kind of evidence was used to determine whether a practice was effective. Commonly accepted standards include:

• The practice was evaluated using a rigorous research design (i.e., one that used an intervention and control group)
• The practice has been shown to be effective for more than one kind of population, and by different researchers
• Studies of the intervention have been published in peer-reviewed journals. Some examples of groups that are credible resources for EBPs include the What Works Clearinghouse [4], the Promising Practices Network [5] , and the Substance Abuse and Mental Health Administration's National Registry of Evidence-Based Programs and Practices [6] .

2. Decide if the EBP is relevant for your population and your program.

Even when the evidence looks good, it's important to take a critical stance. You have to assess for yourself whether you think the practice is relevant to and appropriate for the communities you serve. Keep in mind that research is often limited when it comes to discerning what works for diverse populations. Also important is assessing whether your organization has the capacity to implement the intervention. For example, does your staff have the appropriate skill set and training associated with a particular EBP? Also take into account how easy or hard will it be to obtain staff buy-in before moving forward.

3. If necessary, tweak the EBP so that it best fits your work.

Once you've identified an EBP that seems relevant to your organization, the next step is to think about implementation. You're probably familiar with the significant body of literature on the difficulties and low success rate of replicating model programs. In most cases, nonprofits will not have the resources to replicate an EBP exactly nor is this necessarily the best idea. Instead, nonprofits should think  about how to adapt the practice in ways that are responsive to local needs. To do this, gather information about the theory or logic underlying the EBP. Decide which elements are core to the intended outcomes and which are less critical. Anticipate how potential adaptations might strengthen or weaken the EBP based on the underlying logic.

For example, the EBP on which the Tenderloin group's intervention was based utilized weekly group meetings with participants -- probably good in the EBP model but possibly too difficult with this street-based, drug-using population. Focus groups confirmed for us that having fewer group meetings would not invalidate the core elements of the EBP. As you can see, adaptation is more of an art than a science at this point.

4. Use available resources to adapt EBPs! 

Many EBPs come with materials to support implementation, including program manuals, training curricula, and technical assistance programs. (See, for example, this CDC website [7] .) The best of these will include assessments regarding your organization's readiness to implement a particular practice or program as well as how to adapt EBPs without sacrificing quality in terms of outcomes. If you can't find any implementation resources online, try calling the organization that pioneered the particular practice to gather this information directly. Also, if a particular funder is asking you to employ specific EBPs, be sure to include staff training and technical assistance in your funding request.

5. Measure whether it's working, and keep learning.

Once you've adopted an EBP for your organization, be sure to evaluate what's working. At a minimum, staff should come together regularly to reflect on implementation, identify successes, and discuss intended adaptations as well unplanned ones that occurred in practice. If you hire an external evaluator, ask them to examine how the program was implemented. This information will help you decide whether you have implemented the practice as you intended in addition to whether it is achieving desired outcomes.

Hey, wait a sec . . . We're already doing that!

In the course of looking for EBPs in your field, you may find that your organization is already using practices that have been shown to be effective. In this case, pat yourselves on the back. You've just identified a way to show your funders that your organization's services are consistent with the latest findings about what works.

Given the current economic environment, nonprofits are going to face increasing pressure to demonstrate that their programs are evidence-based.  Whether you're interested in preventing the spread of HIV, educating young children, or helping young people connect with employment and education, knowing the EBPs in your field will put you at an advantage when it comes to talking with funders. 

Clare Nolan is vice president of Harder+Company Community Research [8](www.harderco.com [9]), a California consulting firm that specializes in research and strategic planning for the nonprofit, philanthropic and public sectors.

Readers: Do you agree with Clare? Disagree? What have been your successes and frustrations? Clare will respond to comments posted here.