Embedded Research & Evaluation – The Great Escape

View Report

Do process evaluation reports often have you wondering, “Where is the insight? Where is the actionable information? What are my next steps? Why did we do this research?”

How did we get to this place and can we escape?

There are two basic problems with many process evaluations: (1) the evaluation is far too ‘backward looking’, and (2) the idea of third-party independence. While third-party independence for verifying energy impacts is understood, it makes no sense to me that this same idea is applied to process evaluation work. Likewise, what value is there in assessing a program participant’s experience when it occurred months, and sometimes years, ago?

Embedded research and evaluation is the solution. And the good news - it is simple and it can be fast. Program staff and evaluation professionals work together to define the research objectives, assess data and information as it is gathered, and develop action plans to address findings and recommendations as the evaluation project progresses. This open and sharing partnership is vital for meaningful fast feedback.

Getting to this shared and open relationship can be challenging, especially when evaluation has failed to deliver meaningful insight and useful recommendations. To overcome this obstacle, the program team is in charge with the evaluation group playing the supporting role.

Embedded Research and Evaluation In Practice: A Working Example

My colleagues approached me with the question, “How can we get more savings from our commercial and industrial audit program?” An embedded research and evaluation project was born!

The Focused Problem Statement: The first step in our embedded research and evaluation project was to define the problem.

Commercial and industrial energy audits lead to energy savings projects that deliver only a fraction of the savings identified.

The Hypothesis: The next step was to determine the hypothesis. As we considered the problem statement along with program tracking data and information, we realized there were several areas for exploration – too many to address all at once. In fact, addressing too many simultaneously would lead to strained resources and bog down the effort – not conducive at all for fast feedback. Therefore, we prioritized our research questions and developed a phased-approach to the embedded research and evaluation project.

Commercial and industrial energy audits can deliver more energy savings with the right post-audit follow-up and support.

The Focused Research Question - Phase 1: Meaningful research starts with a good question. From the broad question, “How can we get more savings from our commercial and industrial audit program?”, we narrowed our initial focus to:

How can we improve the post-energy audit process to drive more projects?

Once we defined our initial focus, we developed our evaluation plan. Tune in to my next post to learn about our plan, results, and recommendations.

Picture of Teresa Lutz

Teresa Lutz

Earlier in my career, I worked for a utility supporting the design and delivery of energy conservation programs through evaluation and research. At that time, I did not love the evaluation process or the evaluation community. The value of evaluation was a tough sell to my coworkers, and I agreed the evaluation process and results could be better. We wanted more timely feedback, recommendations we could implement, and insight beyond what we already knew. As a consultant, I hold those experiences close. I avoid doing ‘evaluation for evaluation’s sake’. I am fixated on figuring out the Big WHY of what we do, what works and what doesn’t. It is through knowing this that we can improve and prosper in this industry.

More posts by Teresa Lutz