It’s January and after a somewhat eventful 2020, flipping the calendar to a new year seems extra meaningful. Because many people set New Year’s resolutions[1], I thought a good topic for this month’s blog would be the goals of energy efficiency programs and how evaluation can help program administrators meet them.

Implications of Different Types of Goals

The reasons to implement demand-side management programs can vary and have changed over time. The goals can include reducing the need for new construction of generation and T&D capacity, reducing greenhouse gas (GHG) emissions, developing clean energy jobs, reducing energy burden of low income customers, ensuring high customer satisfaction, and many others.

These high-level goals are often set at the state level but then trickle down to individual program administrators and programs through integrated resource plans, potential studies, utility commissions and other stakeholders, and other methods.

It is important to understand the type of savings goals established for a program, as there may be unintended consequences due to the framework in place.[2] The most common types of savings goals are:

1. Cumulative First Year Savings - Many jurisdictions use cumulative first year savings goals, meaning that utilities set annual targets but are ultimately responsible for meeting a certain goal over the length of the program cycle (often 3 to 6 years). The measures’ savings from the first year of installation are summed over the duration of the program cycle and compared to the goal. In this framework, the expected lifetime of a measure does not matter, which in the past has led to some programs focusing on quick hitting savings to help meet goals.[3] This framework also required utilities to establish a way to recognize savings from long-lead projects that may span across program cycles.
2. Annual First Year Savings – Some jurisdictions set annual goals for first year savings, instead of allowing for multiple years to contribute to a single program cycle goal. While largely similar to cumulative first year savings goals, this framework disincentivizes multi-year projects (e.g., a large industrial facility that has identified large custom projects to be implemented in several phases and is looking to implement them over several years to spread out the cost) or projects with lead times that span more than one year. In highly cyclical markets or segments, like oil and gas, these multi-stage projects may be underdeveloped, leaving savings on the table for utilities.
3. Cumulative Persistent Annual Savings - Some jurisdictions use cumulative persistent annual savings goals, which sums the lifetime savings of measures. This framework encourages utilities to focus on deeper, longer-life measures but may disincentivize shorter term savings, such as those from behavioral programs or retrocommissioning. In a related framework, some utilities also set goals where an established share of savings must come from measures in different lifetime bins (e.g., under 10 years, 10-20 years, over 20 years) to promote longer-life measures.

How Can Evaluation Help?

This seems like a topic that focuses more on how programs are designed and run, so how can evaluation help? Well, in many ways.

First, the basic question evaluators can ask is “is the program doing what it is supposed to do?” This seems very obvious until one starts to think about what the program is actually supposed to do. Is it designed to save kWh, kW, or therms, all of the above, or is it fuel neutral? Is it supposed to target all possible savings or only cost effective savings? Is it okay if all savings come through three contractors or should the program increase awareness of energy efficient equipment among all contractors?

To help understand what a program is actually supposed to do, programs and evaluators can create program theory/logic models. These models are flow charts that illustrate how a program works. They typically have four categories: program activities, outputs, short term (or proximal) outcomes, and long term (or distal) outcomes. Program theory/logic models allow evaluators to determine if a program is logically set up and then allows for tracking metrics from stage to stage to measure success. If a program does not meet its goal, a program theory/logic model helps evaluators determine whether it was due to a failure in program design or execution. The figure below shows a simplified program theory/logic model of a contractor-driven program.

Once evaluators determine (or confirm) the metrics for success, they can provide an independent measurement of those metrics. The most obvious are savings-related impacts, such as verified kWh or MBtu savings, but many goals are based on include non-energy impacts, such as increased clean energy jobs and other non-energy impacts (e.g., reduced operations and maintenance costs, change in comfort, reduced utility arrearages, etc.).

Process evaluations may also provide program administrators with insight into how to better meet their goals. For example, the evaluator may research whether the program is offering measures that best align with its goals or if the program is properly targeting customers to meet its goals. Because every program operates in its own jurisdiction, even very similar programs may require different research depending on that jurisdiction’s goals.

[1] My 2020 resolution was to spend more time with at home my family, so I nailed that one.

[2] Frameworks also vary by whether they count gross or net savings, which has even more consequences, but we will cover that piece in another blog.

[3] **Ahem, HVAC programs offering free CFLs.**