By now, you have engaged partners, conducted an environmental scan and assessed the existing evidence to support your position. You reviewed the feasibility of your strategy, promoted it to your stakeholders and implemented it. Welcome to the final step!
Evaluate the processes employed during the implementation of your PSE change intervention as well as the short-term, intermediate and long-term outcomes expected to result from the intervention. Evaluation can provide quantitative and qualitative data to demonstrate the change(s) that occurred.
You developed specific, measurable, attainable, relevant and time-bound (SMART) objectives for your PSE intervention, using indicators to help you measure the processes and changes that occurred as you implemented these objectives.
Input indicators measure resources used to carry out the PSE change intervention and output indicators measure immediate results, which help evaluate program efficiency.
Process indicators measure the key processes and activities that are expected to contribute to the intervention outcomes, such as reach and quality.
Short-term outcomes are changes you want to enact that promote healthier behaviors. Such changes may occur in the following areas: “physical (e.g., proximity to healthier food and spaces for physical activity); economic (e.g., changes to prices, taxes); social (e.g., changes to social networks); and communication (e.g., advertisements, point-of-decision prompts).” (Leeman, 2012). For example, if your initiative focused on the elimination of food deserts, a short-term outcome could be securing a dedicated, highly trafficked space for a weekly farmers market.
Intermediate outcomes align with your behavioral objectives, which are desired changes in your audience’s behaviors, such as actions you want them to engage in or actions you want them to stop as a result of your PSE change intervention. For example, evaluation of an intermediate outcome would be assessing the change in community members’ physical activity following the installation of walking paths and bike lanes, or the change in attitudes toward or consumption of sugary drinks following passage of a city soda tax.
Long-term outcomes/impact align with your health objectives and should correspond with your comprehensive cancer control plan goals. These outcomes are the desired changes in your audience’s health status that occur as a result of your intervention. They should be “effective, equitable and cost-effective at the population level.” (Leeman, 2012). One such example would be a change in cervical cancer incidence following improvements in access to HPV vaccination and cervical cancer screening for vulnerable populations.
You may have identified some barriers during implementation that held back some of your success. Be sure to document your lessons learned and share your PSE change successes with stakeholders!
USING DATA TO EVALUATE YOUR EFFORTS
Collecting data is extremely important when evaluating your PSE change intervention. The following tips may be helpful as you consider how to best use data gathered in your evaluation to improve and sustain your PSE change.
- Use input, output and process data to monitor key processes in your implementation and if necessary, pivot or make improvements along the way.
- Think about the decision makers and audience interested in your results and choose appropriate and meaningful indicators to present to them.
- Does the evaluation enable you to meaningfully address concerns or challenges? If not, why?
- Use proxy indicators, which are indirect measures of your intervention’s desired outcomes, when direct indicators are not available or feasible to measure.
- Use outcome data to refine your objectives, plan to for sustaining or furthering your intervention impacts or to plan your next PSE change effort.
TOOLS AND RESOURCES
Below is a list of resources focused on Step 7 of the PSE change approach (measure your success):
- Building Models and Momentum: Predicting the Impact of Tobacco Control Policies – This Research to Reality webinar from September 2017 presented the new Tobacco Control Policy Tool and a successful tobacco control campaign in Hawaii.
- Community Tool Box provides guidance on Our Evaluation Model: Evaluating Comprehensive Community Initiatives to “evaluate comprehensive community initiatives working to improve quality of life in the community.”
- Evaluating Policy, Systems and Environmental Change Interventions: Lessons Learned from CDC’s Prevention Research Centers – This paper focuses on evalations of PSE change programs that use CDC’s Framework for Program Evaluation in Public Health, including lessons learned.
- An Evaluation Framework for Obesity Prevention Policy Interventions – The CDC-funded Center of Excellence for Training and Research Translation developed this framework for the evaluation of policy interventions and the identification of gaps in existing practice-based evidence.
- Evaluation Frameworks Designed to Measure Public Health Impact: Applying the RE-AIM Framework to Evaluation of Policy, System and Environmental Changes in California SNAP-Ed – These slides describe the Reach-Efficacy-Adoption-Implementation-Maintenance (RE-AIM) Framework and apply it to a nutrition-focused PSE change effort in California.
- The National Colorectal Cancer Roundtable offers an evaluation toolkit that explains how to evaluate activities intended to increase awareness and use of colorectal cancer screening.
- Policy, Systems and Environmental Change in the Mississippi Delta: Considerations for Evaluation Design – This article by Kegler et al. describes an approach to evaluation of community PSE change programs and aims to spark discussion around approaches for evaluation of PSE change efforts.
- Program Sustainability Assessment Tool – With this tool from the Center for Public Health Systems Science, stakeholders can rate their program across eight sustainability domains.
- Visit the Cancer Control Technical Assistance Portal (TAP) Resource Repository for general evaluation resources from CDC.