Category: State Talk

Using Learning Agendas to Communicate Effectively with Evaluation Stakeholders

By Dr. Dakota Thomas

The Governing for Results Network is a multi-state peer learning network of state evidence leaders. The network is a collaborative effort hosted by the National Conference of State Legislatures, The Council of State Governments and The Policy Lab at Brown University. The Governing for Results Network works to foster connections across the network, and within and across state governments, by engaging with state legislators, budget directors, and legislative and agency staff who advance the use of data and evidence across branches of government.

---

Key Takeaways

• Communication is critical for ensuring that evaluations are targeted toward programs and policy areas that need them, and that they are evaluated fairly and effectively. Policymakers need to know what programs are not working as intended, so that they can be improved.
• A learning agenda can be a powerful tool for structuring communication with evaluation stakeholders. It also assists in ensuring that the right people are involved in guiding what programs are evaluated and informed of relevant evaluation results and responsive to any reforms suggested by evaluators.

---

For policymakers to rely on evidence to make decisions, communication is vital. An evaluation, on its own, is only helpful when used to make decisions. For that to occur, communication between researchers, program staff and other stakeholders is key. This communication should begin before an evaluation is even considered, and must continue during and after the evaluation to be effective.

The information researchers gather from talking to policymakers, program participants and the staff that implement the program daily can be invaluable for determining the direction of the evaluation and making sure it is well received by stakeholders. Researchers need help clearly communicating the implications of their work; stakeholders need accurate information to inform their decisions; and program staff need to know what is working and what needs to change.

This brief outlines how adopting a learning agenda can be used to help ensure smooth communication throughout the evaluation lifecycle. A learning agenda is a plan for using evidence, research and evaluations to improve policy and program outcomes. This includes taking the following actions:

• Identifying the priority research questions that need to be answered based on strategic goals and the information needs of key stakeholders.
• Assessing the existing evidence base.
• Identifying gaps in knowledge that need to be addressed.
• Planning to conduct research to address those gaps.
• Engaging with stakeholders to develop and implement the learning agenda.
• Providing the findings to those stakeholders.

The key benefit learning agendas offer as a communication framework is ensuring stakeholder engagement throughout the entire research process.

“This is a complaint I have heard many times from data/analyst folks: ‘We did this great study but nobody wants to read it.’ I think a lot of the problem is a lack of engagement between those operating government and those evaluating government. You need to talk to others before doing an evaluation to know the landscape of how an evaluation will be received. If there is no perception that there is an issue in a program, practitioners are unlikely to be interested unless there are shocking conclusions from the evaluation. Of course, this shouldn’t be the only input and an analyst might have good reason to think the collective wisdom is misplaced, but it is an important factor.”

— Jonathan Womer, The Policy Lab at Brown University

Using Learning Agendas as a Communications Framework
States and state agencies could adopt learning agendas as a tool for smoothing out communication between stakeholders (e.g., legislators), program staff and researchers. Borrowing heavily from the auditing literature, this process of using a learning agenda as ongoing communication framework should look as follows:

• Step 1: The first step is to survey the organization/agency being evaluated. This helps to create the “evaluation universe,” an up-to-date list of evaluable programs and policies that can be updated every year. Communicating with stakeholders well in advance of deciding what to evaluate and how to do so is critical. Gathering perspectives from a diverse group of people early in development is a great way to ensure the learning agenda is well calibrated to the needs of the state or agency and the communities it serves.
• Step 2: Prioritize and rank evaluable programs and/or policies. We cannot evaluate everything. Depending on the focus of an agency, it makes sense to prioritize the most important, newest and least tested programs for evaluation. Again, this is a great stage to seek feedback from stakeholders, including agency leadership, legislators, program staff and so on.
  • One question you should ask agency leadership and program managers is, “What keeps you up at night?” This will help you understand the key priorities that the learning agenda can help to understand and address. By asking this question, the North Carolina Office of Strategic Partnerships was able to help the state’s Department of Public Safety decide what research to pursue (see below).
• Step 3: Create the learning agenda based on the priorities established in Step 2 (and staff capacity), create a plan for addressing the largest items on the agenda. Once the learning agenda has been drafted, seek out more feedback and incorporate that into the final draft.
• Step 4: Perform the evaluation(s) needed. Keeping lines of communication open during the evaluation process can help prevent surprises and give program staff a chance to change course if things are not working.
• Step 5: Share the results of the evaluation(s) with relevant stakeholders. If you have communicated regularly and built up a trusting relationship with the people who need to know what’s going on, this will be much more effective than just calling everyone in at the end to hear the results.
• Step 6: Repeat. Learn from the process and improve for the next round of evaluations, which may be based on lingering questions from the first round.

The purpose of stakeholder engagement is to make sure that the learning agenda addresses questions that are relevant, salient, and meaningful to those with direct interests in the agency’s functions, and that the learning that results resonates with stakeholders.”

— Excerpt from M-19-23, Office of Management & Budget, Executive Office of the President

State Successes with Learning Agendas
While learning agendas are more common at the federal level, where some agencies are required to use learning agendas under the Evidence Act of 2019, states have also had success in adopting learning agendas. Consider the following examples:

Colorado’s Department of Public Health and Environment recently used a learning agenda to guide their public health efforts focused on the state’s family support programs. Colorado’s learning agenda was helpful for ensuring an inclusive group of stakeholders were involved throughout the entire evaluation process, including program directors and managers, state intermediaries, home visitors and university researchers. Thanks to early communication with those stakeholders, the agency identified mental health services as a key concern warranting further research. After consulting stakeholders again, the team found that an evaluation was appropriate for two home visiting programs. When the initial evaluation yielded mixed results, the team then examined additional research questions that revealed issues with training for one of the programs.

Findings were shared with participants, program staff and agencies, and later used to improve the identified weaknesses. Sharing the results of this study was structured and comprehensive. Researchers shared results with participants in the program, local agencies that implement the program, state intermediaries, and other similar programs in and outside of the state. The team shared results in multiple formats, including briefs, documents and newsletters, focusing on the key takeaways from the study. A learning agenda presents a great opportunity to formalize the follow up process and dissemination of results A high quality learning agenda will include a thorough plan for making the results available in many forms to the many stakeholders involved, providing multiple opportunities for them to learn, ask questions and use the information to make decisions.

Connecticut uses a learning agenda to gather all the research questions that agencies need answered into one place, and guide the data collection and analysis process to adequately answer those questions. This ensures that the state’s data system is responsive to policymakers, agencies and, ultimately, the state’s citizens. For example, the state wanted to understand the process of college admissions. Using the framework of a learning agenda to collaborate with the relevant stakeholders, researchers in the state narrowed this topic down to just a few key research questions, including:

• What are the pre-college indicators, including student need, that correlate with the level of preparation required for a high school student to succeed in a core, credit-level course of study at a postsecondary institution?
• To what degree does high school academic history correlate with career readiness for students who do not pursue postsecondary education after high school completion?

Guided by these informed research questions. The state’s data analysts conducted an evaluation and found several predictive pre-college indicators of success, suggested evidence-based options to improve performance, and identified areas for future research.

Connecticut’s learning agenda was especially helpful for policymakers due to the sheer number of stakeholders involved. Tackling these questions required collaboration, data sharing and analysis among 13 participating organizations, including multiple state agencies, universities and nonprofits.

Without a clear system of communication, it’s possible to fail to collect crucial data needed to answer a question, or to have multiple agencies and groups pursuing the same questions independently, and policymakers may not get the information they need to make informed decisions. The learning agenda provided a structured opportunity for all stakeholders to communicate early in the process, which helped to ensure the right research questions were being asked, the right data collected, the right programs evaluated, and the right follow up was conducted.

To develop their most recent learning agenda, the North Carolina Office of Strategic Partnerships met with leaders of the Department of Public Safety to ask, “What keeps you up at night?” The discussion revealed three priority questions related to the COVID-19 pandemic that focused on providing health care and mental health resources for offenders reentering the community, prison supervision and employment barriers for ex-offenders. Knowing these priorities, the team was able to identify external partners interested in tackling those issues.

Thanks to the structure provided by the learning agenda, North Carolina was able to recruit three teams of university-based external researchers to tackle these questions. These teams reviewed public data and available research, delivering briefs with initial findings six weeks later at no cost to the state. These findings then informed future analysis and the immediate decisions of policymakers. The learning agenda framework helped structure collaboration between the state government and universities.

North Carolina’s example illustrates learning agendas’ usefulness as a tool to seek external partners and maintain an ongoing channel of communication between policymakers, program evaluators, program managers, and external partners like university researchers. By building relationships and establishing trust, it becomes easier for policymakers to help researchers decide what programs to study, and for researchers to deliver results to policymakers in a way that will actually be useful for making decisions.

Conclusion
Communication is critical for turning evidence into effective policy. A study proving that a program is effective is useless if policymakers never actually hear about it. To ensure smooth, effective communication with all stakeholders throughout the evaluations process, researchers and evaluators should consider using a learning agenda as a communications tool to keep policymakers in the loop from the outset. Doing so can help ensure that the right programs and policies get evaluated for maximum impact, that the right people are involved in the evaluation from the beginning, and that the right people are informed of evaluation results for the  findings to be used in decision-making. State policymakers could consider requiring the use of learning agendas for state agencies, similar to the federal mandate from the Evidence Act of 2019.  Properly implemented, learning agendas can help make evidence-based policymaking easier, more accessible, and more effective.

Resources for Evaluators and Researchers:

• The Governing for Results Network is happy to assist you in trying to improve communication about evaluations in your state.
• The Office of Evaluation Sciences published “A Guide to Developing your Agency’s Learning Agenda,” which contains suggestions for incorporating stakeholder feedback throughout the learning agenda process.
  • The office also offers multiple toolkits and resources for agencies adopting or implementing a learning agenda.
• The Urban Institute created a resource, “Engaging Stakeholders in Learning Agenda Development.” Though focused on federal learning agendas, many of the lessons apply to states as well. The Urban Institute also produced a more general resource on learning agendas, which provides a helpful outline for beginners.
• For many examples of learning agendas produced by federal agencies, please view the resource available on evaluation.gov.