Using Learning Agendas to Facilitate Communication between Policymakers, Researchers and State Agencies

By Dakota Thomas, Ph.D.

The Governing for Results Network is a multi-state peer learning network of state evidence leaders. The network is a collaborative effort hosted by the National Conference of State Legislatures, The Council of State Governments and The Policy Lab at Brown University. The Governing for Results Network works to foster connections across the network, and within and across state governments, by engaging with state legislators, budget directors, and legislative and agency staff who advance the use of data and evidence across branches of government.

Key Takeaways

  • Communication is critical for ensuring that evaluations are targeted toward programs and policy areas that need them, and that they are evaluated fairly and effectively. Policymakers need to know what programs are not working as intended, so that they can be improved. Researchers need to know what gaps in knowledge are most pressing for policymakers and agencies. Agency staff need to know what research and evidence suggests is the best way to accomplish their goals.
  • A learning agenda can be a powerful tool for structuring communication with evaluation stakeholders. It also assists in ensuring that the right people are involved in guiding what programs are evaluated and informed of relevant evaluation results and responsive to any reforms suggested by evaluators.

For policymakers to rely on evidence to make decisions, communication is vital. An evaluation, on its own, is only helpful when used to make decisions. For that to occur, communication between researchers — both inside and outside of state government — program staff, state policymakers and other stakeholders is key. This communication should begin before an evaluation is even considered, and must continue during and after the evaluation to be effective.

The information researchers gather from talking to policymakers, program participants and the staff that implement the program daily can be invaluable for determining the direction of the evaluation and making sure it is well received by stakeholders. Researchers need help clearly communicating the implications of their work; stakeholders need accurate information to inform their decisions; and program staff need to know what is working and what needs to change.

This brief outlines how adopting and implementing a learning agenda can be used to help ensure smooth communication throughout the evaluation lifecycle. A learning agenda is a plan for using evidence, research and evaluations to improve policy and program outcomes typically focused on a specific policy area or based on the prerogative of a specific state agency. To create and implement a learning agenda, researchers will need to:

  • Identify the priority research questions that need to be answered based on strategic goals and the information needs of key stakeholders.
  • Assess the existing evidence base.
  • Identify gaps in knowledge that need to be addressed.
  • Plan to conduct research to address those gaps.
  • Engage with stakeholders to develop and implement the learning agenda.
  • Provide the findings to those stakeholders.

The key benefit learning agendas offer as a communication framework is ensuring stakeholder engagement throughout the entire research process.

“This is a complaint I have heard many times from data/analyst folks: ‘We did this great study but nobody wants to read it.’ I think a lot of the problem is a lack of engagement between those operating government and those evaluating government. You need to talk to others before doing an evaluation to know the landscape of how an evaluation will be received. If there is no perception that there is an issue in a program, practitioners are unlikely to be interested unless there are shocking conclusions from the evaluation. Of course, this shouldn’t be the only input and an analyst might have good reason to think the collective wisdom is misplaced, but it is an important factor.”

— Jonathan Womer, The Policy Lab at Brown University

Using Learning Agendas as a Communications Framework
States and state agencies could adopt learning agendas as a tool for smoothing out communication between all relevant stakeholders, such as legislators program staff and external researchers. Borrowing heavily from the auditing literature, this process of using a learning agenda as ongoing communication framework should look as follows:

  • Step 1: The first step is to survey the landscape an agency is operating in. This helps to create the “evaluation universe,” an up-to-date list of evaluable programs and policies that can be updated every year. Communicating with stakeholders well in advance of deciding what to evaluate and how to do so is critical. Gathering perspectives from a diverse group of people early in development is a great way to ensure the learning agenda is well calibrated to the needs of the state or agency and the communities it serves.
  • Step 2: Prioritize and rank evaluable programs and/or policies. We cannot evaluate everything. Depending on the focus of an agency, it makes sense to prioritize the most important, newest and least tested programs for evaluation. Again, this is a great stage to seek feedback from stakeholders, including agency leadership, legislators, program staff and so on.
  • Step 3: Create the learning agenda based on the priorities established in Step 2 (and in consideration of staff capacity) and plan for addressing the largest items on the agenda. Once the learning agenda has been drafted, seek out more feedback and incorporate that into the final draft.
  • Step 4: Perform the evaluation(s) needed. Keeping lines of communication open during the evaluation process can help prevent surprises and give program staff a chance to change course if things are not working.
  • Step 5: Share the results of the evaluation(s) with relevant stakeholders. If you have communicated regularly and built up a trusting relationship with the people who need to know what’s going on, this will be much more effective than just calling everyone in at the end to hear the results.
  • Step 6: Repeat. Learn from the process and improve for the next round of evaluations, which may be based on lingering questions or new puzzles uncovered in the first round.

The purpose of stakeholder engagement is to make sure that the learning agenda addresses questions that are relevant, salient, and meaningful to those with direct interests in the agency’s functions, and that the learning that results resonates with stakeholders.”

— Excerpt from M-19-23, Office of Management & Budget, Executive Office of the President

State Successes with Learning Agendas
While learning agendas are more common at the federal level, where some agencies are required to use learning agendas under the Evidence Act of 2019, some states have also adopted learning agendas for specific policy areas and used them successfully to communicate between the relevant stakeholders, before, during and after evaluations of various programs.  Consider the following examples:

Colorado’s Department of Public Health and Environment recently used a learning agenda to guide their public health efforts focused on the state’s family support programs. Colorado’s learning agenda was helpful for ensuring an inclusive group of stakeholders were involved throughout the entire evaluation process, including program directors and managers, state intermediaries, home visitors and university researchers. Thanks to early communication with those stakeholders, the agency identified mental health services as a key concern warranting further research. After consulting stakeholders again, the team found that an evaluation was appropriate for two home visiting programs. When the initial evaluation yielded mixed results, the team then examined additional research questions that revealed issues with training for one of the programs.

Colorado’s experience proves the importance of a thorough plan for making the results available in many forms to the many stakeholders involved, providing multiple opportunities for state policymakers to learn, ask questions and ultimately use the information to make decisions. Using a learning agenda to share the results of this study ensured that communication was both structured and comprehensive. Researchers shared results with participants in the program, local agencies that implement the program, state policymakers, and other similar programs in and outside of the state. The team shared results in multiple formats, including briefs, documents and newsletters, focusing on the key takeaways from the study. A learning agenda presents a great opportunity to formalize the “follow-up” process and dissemination of results.

Connecticut used a learning agenda to gather all the research questions that agencies needed answered into one place, and guide the data collection and analysis process to adequately answer those questions. This ensures that the state’s data system is responsive to policymakers, agencies and, ultimately, the state’s citizens. For example, the state wanted to understand the process of college admissions. Using the framework of a learning agenda to collaborate with the relevant stakeholders, researchers in the state narrowed this topic down to just a few key research questions, including:

  • What are the pre-college indicators, including student need, that correlate with the level of preparation required for a high school student to succeed in a core, credit-level course of study at a postsecondary institution?
  • To what degree does high school academic history correlate with career readiness for students who do not pursue postsecondary education after high school completion?

Guided by these informed research questions, the state’s data analysts conducted an evaluation and found several predictive pre-college indicators of success, suggested evidence-based options to improve performance and identified areas for future research.

Connecticut’s learning agenda was especially helpful for policymakers due to the sheer number of stakeholders involved. Tackling these questions required collaboration, data sharing and analysis among 13 participating organizations, including multiple state agencies, universities and nonprofits.

Without a clear system of communication, it’s possible to fail to collect crucial data needed to answer a question, or to have multiple agencies and groups pursuing the same questions independently, and policymakers may not get the information they need to make informed decisions. The learning agenda provided a structured opportunity for all stakeholders to communicate early in the process, which helped to ensure the right research questions were being asked, the right data collected, the right programs evaluated and the right follow up was conducted.

Conclusion
Communication is critical for turning evidence into effective policy. A study proving that a program is effective is useless if policymakers never actually hear about it. To ensure smooth, effective communication with all stakeholders throughout the evaluations process, researchers and evaluators should consider using a learning agenda as a communications tool to keep policymakers and relevant agency staff and/or other stakeholders in the loop from the outset. Doing so can help ensure that the right programs and policies get evaluated for maximum impact, the right people are involved in the evaluation from the beginning, and the right people are informed of evaluation results so the findings can be used in decision-making. State policymakers could consider requiring the use of learning agendas for state agencies, similar to the federal mandate from the Evidence Act of 2019. Properly implemented, learning agendas can help make evidence-based policymaking easier, more accessible, and more effective.

Resources for Evaluators and Researchers:

Recommended Posts

Japan Hosts Delegation, Promotes Education, Engagement with the World as Keys to its Future
Relaunch of The National Center for Interstate Compacts Database
Remembering Brigid Kelly, a Former Ohio State Representative and CSG Henry Toll Fellow