Evaluations can sometimes feel like a regulatory checkbox. This can seem especially true for process evaluations which may be required but provide seemingly limited value. However, process evaluations can provide programs with invaluable information if the one-size-fits-all approach is left behind and evaluators customize them with the program’s life cycle in mind.
Evaluations of demand-side management programs typically focus on their energy savings and other outcomes. In addition to quantifying the impact of programs, it is also important to put that impact into context by understanding its implementation. Process evaluations explain the key drivers of impact evaluation results, document how programs operate, if they are being delivered as planned, and areas for improvement. Most importantly, they are not simply customer satisfaction surveys.
The Status Quo
Traditionally, program administrators find process evaluations to be particularly valuable when launching a new program or when making notable changes to an existing program, such as changing its implementer. They are also helpful when the results of the program are different than expected (often too low) and when participants report dissatisfaction or issues.
An example of this traditional process evaluation approach is the Arkansas protocol. The Arkansas TRM recommends that evaluators should conduct a process evaluation when any of the following criteria are met:
- The program has new and innovative components that have not been evaluated previously
- The program has not had a comprehensive process evaluation during the previous funding cycle
- The program has a new vendor or contractor implementing it
Additionally, the Arkansas TRM describes program performance conditions that may require a process evaluation:
- The program’s impacts are lower than expected
- Other objectives (e.g., informational or educational goals) are not being met
- The program’s participation rate is lower than expected
- The program is having operational challenges
- The program’s cost-effectiveness is less than expected
- Participants are reporting issues or low levels of satisfaction
- The program’s market effects need to be measured
While all of these criteria are excellent reasons for conducting a process evaluation of a program, they all tend to be reactive and seek to explain the program’s past implementation. As a result, process evaluations tend to be snapshots of the past with potentially limited applicability to the program’s future. Even in continuous evaluations, evaluators may just be tracking the same thing over time.
Adding Value
Evaluators can make process evaluations much more valuable by understanding that demand-side management programs have life cycles. Newly launched programs have different needs and challenges than mature programs. The research conducted and interpretation of results should likewise be different for each of those programs.
The figure below illustrates the program life cycle of a demand-side management program and examples of the types of process evaluation activities that may be suitable for each phase.
In this simplified figure, the program’s participation ramps up over time until it is at its peak and operating in a steady state. After which the program begins to ramp down and eventually close as the market is transformed, new efficient offerings become available, changes in codes or standards take effect, or other reasons. Alternatively, rather than closing, the program may change its approach or offerings to meet the new market realities.
Importantly, the process evaluation activities that evaluators perform should adjust as the program matures. As the program is being designed and launched, evaluators may wish to develop program theory/logic models and establish key performance indicators to facilitate future evaluation. During the ramp up period, research into the customer journey and early feedback is important. As the program continues to gain traction, process evaluations can help identify areas of improvement to improve the program’s efficiency. And so on.
Customizing process evaluations to the life cycle of the program also inherently admits that every demand-side management program should not last indefinitely. A typical goal of DSM programs is to help participants overcome barriers to adopting energy efficiency (e.g., awareness, cost, technical support). A successful program reduces those barriers over time to the point where it is no longer needed and successful evaluations help program administrators determine when that time is.