Sometimes in the course of evaluating energy efficiency projects, we come across projects with negative savings. In other words, participating in an energy efficiency program resulted in an increase in overall energy usage. Given the program theory of most energy efficiency programs, it is safe to say that is not supposed to happen! So what went wrong?
First, let’s look into why savings might appear negative.[1] There are many potential reasons:
When confronted with projects with negative savings, utilities, implementers, and evaluators do not have consistent approaches for reporting savings. There are several potential solutions, each with its own strengths and weaknesses.
Programs and evaluators will often set negative savings to zero. The idea behind this approach is that, historically, the vast majority of energy efficiency projects save energy and no one would intentionally implement an energy efficiency project or participate in an energy efficiency program with the expectation to increase their energy use. While this is the simplest solution, its primary drawback is that it may bias overall program savings upward.
Another approach is to report negative savings as negative savings. This is the most technical pure option. Estimating energy savings compared to an unmeasurable counterfactual is imperfect, whether using regression or other M&V approaches. We try to control for all factors, but there are always unknown externalities. This means that savings may be overestimated on some projects and underestimated on others. By adjusting only projects with negative savings, the project with overestimated savings may not be balanced out.
Other options consist of a hybrid of zeroing out some negative savings and counting other negative savings. For example, there is sometimes a period of negative savings at the start of some SEM and EMS projects as behavior changes ramp up. In these cases, it may be prudent to have a period during this ramp-up time where savings (positive or negative) are not counted or to zero out any negative savings. In other cases, there may be negative savings (either in absolute values or relative to earlier periods) after the first year. These negative savings are often counted as negative. ACEEE’s North American SEM Collaborative recently released findings and recommendations on these types of scenarios based on a focus group with SEM implementers.
Implementers and evaluators generally do their best to treat negative savings in a logical and fairway. What complicates the matter even more is that there are multiple lenses through which to view energy savings, including:
It seems like a practical solution would be to assume zero savings on a project-level basis but count negative savings at the program level. This would slightly increase the cost to the program, but participants would be happy. But…
Fortunately, the number of instances of projects with negative savings is low, meaning that the overall impact of these decisions is relatively small. However, it is important for our industry to develop consensus on how to treat these issues as whole building savings become an increasingly large portion of savings.
As a final thought, if a project has negative savings and the assumption is that EE measures save energy, rather than zeroing them out, the negative savings should be an indicator that something in our analysis is not working right. It could be a symptom of a larger problem, and likely not only affecting one project. Evaluators should go back and revisit all other projects for the same issues. For example, do other projects have a similar NRE, interactive effects, or missing variable in regression? Understanding these issues will result in more accurate savings estimates for programs overall.
[1] Note that the issue of negative savings most commonly arises when estimating facility-wide energy savings (i.e., using IPMVP Option C) from SEM or EMS projects. However, the issue can also manifest with other types of measures and M&V approaches.