Embedded Research & Evaluation - The Process

In my last post, I introduced our embedded research and evaluation (ER&E) ‘likeliness to convert’ project. We are investigating audit participants’ motivation to pursue energy reduction projects as a result of participating in an energy savings audit. Our objective is to determine where additional intervention might be required to persuade action after an energy audit.

As that project unfolds, I am going to write about the overarching process we have established to conduct ER&E – the backstory.

Our ER&E Process: A Step-By-Step Guide

An ER&E effort is focused on a specific program challenge or question. It allows for experimentation with attributes being studied and with evaluation approaches. Learnings from ER&E inform the program in real time, facilitating much quicker changes to program design and corrective action when issues are identified. And, future evaluation is adapted based on what is learned – and what isn’t as the ER&E effort progresses. The initial three steps below outline the process of establishing the ER&E team and developing a program logic model to align ER&E efforts.

Step 1. Identify the ER&E opportunity. ER&E can be embedded with program implementation to help us understand how and why programs work – or don’t work. This research tests the effectiveness of specific mechanisms (like administrative, behavioral, and financial activities) on successful implementation for evidence-based interventions. For existing programs, opportunities for ER&E are often identified by the program staff when an issue arises. For pilots, ER&E is identified in the design phase as pilots are all about learning.

Step 2. Establish the ER&E Team. Once an ER&E opportunity is identified, we put together the ER&E Team.

• Program staff are assigned to lead the program ER&E project. This ensures that the project remains focused on what is most important to the program staff.
• Evaluation staff are assigned to develop evaluation methods to address researchable issues and assess evaluation metrics. This provides an independent, objective perspective that is useful in seeing beyond the perspective of those deeply involved in operations. Sometimes this is an obvious factor that gets overlooked by those so close, and sometimes it is an overarching theme that can be more apparent to someone new to the process.
• Client staff can initiate an ER&E effort, or they can be invited to participate in our ER&E efforts to improve our internal processes. Keeping clients informed, whether or not they are actively engaged in the effort, is always advised.
• Other stakeholders, such as program trade partners, can be invited to participate when integral to the investigation. For example, if an ER&E effort is focused on assessing a trade partner process for marketing and outreach to customers, inviting those trade partners to the ER&E project could greatly benefit the effort. Trade partners could help identify improvement opportunities more readily, allow for testing changes to the marketing messaging and outreach processes, and assure trade partners’ support for improvement efforts.

Step 3. Develop the Program Logic Model. Document the program theory in a program logic model, if not already done. The evaluation staff interview program staff to document the program context, assumptions, inputs, activities, outputs, and outcomes. This serves as a starting point for defining the researchable issues along with where and when in the process ER&E is applicable for the given effort. The American Evaluation Association is a great resource for logic models. A clear and easy way for non-evaluators to think through a logic model is to ask:

• Why are we offering the program?
• What activities do we need to complete?
• Who is needed for successful implementation?
• When do activities need to happen?
• How are we going to evaluate the program results?

See, “Translating Logic Models for STEM Faculty: A ‘Who, What, When, Why, and How’ Approach” by Shelly Engelman, Kristin Patterson, Brandon Campitelli, and Keely Finkelstein for more information.

In the Next Issue

In the next couple posts, I will talk about the other steps in the ER&E process. Next up are ‘Defining Researchable Issues to be Explored’ and ‘Determining Methods for Tackling Researchable Issues’.