A core benefit of embedded research and evaluation (ER&E) is access to fast feedback. Frequent check-ins throughout the effort ensures the opportunity for conversation about findings as they occur. I am thrilled when a laser-focused, 15 minute check-in results in immediate actions to improve processes. Fast feedback secures early rewards – which leads to greater buy-in and support of the ER&E effort throughout the program staff, and beyond.
Take-away: ER&E check-in meetings can be very quick – stand-up meetings can work! The key is to provide frequent opportunity to chat with program staff about what the evaluator is discovering from ER&E efforts. These conversations provide context for findings within the overall program framework and lead to joint learning and actionable insight.
Embedded Research and Evaluation In Practice: A Working Example, continued
In my last blog post, I introduced the story of an ER&E project initiated by colleagues from our program design and delivery group. They approached me with the question, “How can we get more savings from our commercial and industrial audit program?” From here, we identified a focused problem statement, hypothesis, and initial research question to explore.
The Focused Problem Statement: Commercial and industrial energy audits lead to energy savings projects that deliver only a fraction of the savings identified.
The Hypothesis: Commercial and industrial energy audits can deliver more energy savings with the right post-audit follow-up and support.
The Focused Research Question - Phase 1: How can we improve the post-energy audit process to drive more projects?
A slight detour in our story
Before delving into our ER&E project, I want to step back and discuss some comments I received from an astute and thoughtful reader. First, I want to thank the reader for taking the time to interact and share thoughts and ideas. This is exactly what I hope to accomplish – engaging those within and outside of our industry in conversation about their evaluation experience and their expertise. So, a big thank you!
This reader suggests that there may be more than one area for exploration – that low claimed savings from energy audits could be due to the following(paraphrased below):
- The audits are not identifying the best energy saving options and/or include recommendations that are not feasible for the customer.
- The follow-up process is failing to transform the audit findings into feasible energy saving projects.
- The installation, construction, or operation of the energy saving projects have failed.
I agree with these thoughts, and they made me think of additional barriers to project conversion. For example, what is the motivation of customers who participate? Can we identify questions that will help us assess the customers’ “likeliness to convert”?
We expect to explore some, and perhaps all, of these potential issues as additional ER&E phases are implemented. To start, though, we selected the post-audit follow-up process as our focus for two primary reasons:
- Narrowing our focus to a single key process helps us to retain traction and progress for the ER&E effort, as it does not place a high burden on resources.
- The post-audit follow-up process assessment allows us to gather data and information from energy audit customers. This data is then used to explore additional issues as we build-out the ER&E effort.
Back to our story…the Embedded Research and Evaluation Plan
The ER&E plan starts with the research question, “How can we improve the post-energy audit process to drive more projects?”
As a team, the program design and delivery staff and I worked to identify activities that we could do to explore this question. Working as a team is an important component for ER&E – for planning and for implementation. The program staff best understands the program construct and any constraints for data and information collection.
We decided that conducting in-depth interviews with the energy engineers and auditors would provide the most immediate feedback as well as inform future activities. We were not disappointed with this effort. Interviews with four energy auditors resulted in 19 findings, which led to seven immediate recommendations and action plans. Fast feedback was provided within minutes of completing the interviews and within four weeks, five recommendations had been implemented. One of these findings and recommendations is summarized below.
Customer Engagement
Our very first ER&E activity provided insight into other stages of the audit program. Our audit process allows for customer engagement at multiple stages:
- Energy audit scheduling call
- Energy audit
- Post-audit report delivery
- Post-report delivery follow-up
At each stage, we capture data and information that we use to better understand the customer decision making process for participating in an audit and for moving recommended projects forward. This data and information includes:
- Customer motivation for the energy audit.
- Customer pain points – What are they struggling with, and how might we be able to assist them?
- Project approval process – Who approves? Do they have a budgeting cycle?
- Projects completed (measures installed) and projects planned based on the energy audit.
Prior to this ER&E effort, we did not prescribe when this information was gathered during the energy audit engagement points. We discovered from the energy auditor interviews that if we adjust the timing and sequence of the questions we ask audit participants, energy auditors will be better prepared for their next interactions. -- identifying what will motivate the customer to convert recommendations into projects. Our information category gathering sequence is depicted below.
In the Next Issue
Next month, I will share insight gained from questions we ask for each of these categories, and how we will use this information to develop a next ER&E research project phase for assessing customers’ “likeliness to convert”. This is a pivot from our original ER&E plan; a pivot spurred by our reader’s shared thoughts. The ability to quickly shift focus after completing initial research is yet another brilliant outcome of ER&E.