Sometimes in the course of evaluating energy efficiency projects, we come across projects with negative savings. In other words, participating in an energy efficiency program resulted in an increase in overall energy usage. Given the program theory of most energy efficiency programs, it is safe to say that is not supposed to happen! So what went wrong?
Why Is This Happening?
First, let’s look into why savings might appear negative.[1] There are many potential reasons:
- Not properly accounting for a change in facility occupancy or use
- Not accounting for (or incorrectly accounting for) non-routine events
- Removing overstated savings from additionally incented measures due to the use of deemed saving values
- Delay in the realization of energy savings due to a ramp-up period of behavior changes
- Less than a full year of post-installation data to claim savings in the same year as the installation
- Low data granularity (monthly data vs. hourly interval data) limiting the accuracy of the analysis
- Inclusion of Interactive effects
- Bringing equipment up to code as part of a project (e.g., fixing ventilation)
Potential Solutions
When confronted with projects with negative savings, utilities, implementers, and evaluators do not have consistent approaches for reporting savings. There are several potential solutions, each with its own strengths and weaknesses.
Programs and evaluators will often set negative savings to zero. The idea behind this approach is that, historically, the vast majority of energy efficiency projects save energy and no one would intentionally implement an energy efficiency project or participate in an energy efficiency program with the expectation to increase their energy use. While this is the simplest solution, its primary drawback is that it may bias overall program savings upward.
Another approach is to report negative savings as negative savings. This is the most technical pure option. Estimating energy savings compared to an unmeasurable counterfactual is imperfect, whether using regression or other M&V approaches. We try to control for all factors, but there are always unknown externalities. This means that savings may be overestimated on some projects and underestimated on others. By adjusting only projects with negative savings, the project with overestimated savings may not be balanced out.
Other options consist of a hybrid of zeroing out some negative savings and counting other negative savings. For example, there is sometimes a period of negative savings at the start of some SEM and EMS projects as behavior changes ramp up. In these cases, it may be prudent to have a period during this ramp-up time where savings (positive or negative) are not counted or to zero out any negative savings. In other cases, there may be negative savings (either in absolute values or relative to earlier periods) after the first year. These negative savings are often counted as negative. ACEEE’s North American SEM Collaborative recently released findings and recommendations on these types of scenarios based on a focus group with SEM implementers.
Multiple Lenses
Implementers and evaluators generally do their best to treat negative savings in a logical and fairway. What complicates the matter even more is that there are multiple lenses through which to view energy savings, including:
- How savings are shown on M&V reports
- How programs claim savings
- What really happened
It seems like a practical solution would be to assume zero savings on a project-level basis but count negative savings at the program level. This would slightly increase the cost to the program, but participants would be happy. But…
- What is the best way to account for the interactive effects of a dual fuel project for a single fuel utility? Should a gas utility’s savings be reduced because of the lighting measures (where inefficiency displaces fuel heating) installed through an electric utility’s program?
- If negative savings exist but are not claimed by the utility, should they be accounted for in cost-effectiveness analyses?
- Are the actual savings (e.g., including all interactive effects and negative savings) being accurately estimated for ISO planning and carbon reduction accounting?
- Et cetera
Fortunately, the number of instances of projects with negative savings is low, meaning that the overall impact of these decisions is relatively small. However, it is important for our industry to develop consensus on how to treat these issues as whole building savings become an increasingly large portion of savings.
Symptoms of Larger Problems?
As a final thought, if a project has negative savings and the assumption is that EE measures save energy, rather than zeroing them out, the negative savings should be an indicator that something in our analysis is not working right. It could be a symptom of a larger problem, and likely not only affecting one project. Evaluators should go back and revisit all other projects for the same issues. For example, do other projects have a similar NRE, interactive effects, or missing variable in regression? Understanding these issues will result in more accurate savings estimates for programs overall.
[1] Note that the issue of negative savings most commonly arises when estimating facility-wide energy savings (i.e., using IPMVP Option C) from SEM or EMS projects. However, the issue can also manifest with other types of measures and M&V approaches.