Key Takeaways
Process evaluations for delivery networks help us learn how to best support, strengthen, and expand the Aging Mastery Program®.
A process evaluation needs careful planning and a structure that all stakeholders in a statewide delivery network agree on upfront and adapt as needed.
Learn more about what a participatory and collaborative process evaluation would look like.
In early 2020, I was invited to partner with the Aging Mastery® team on the first process evaluation of building a sustainable, statewide delivery network for the Aging Mastery Program® (AMP).
How we approached the evaluation helped uncover learnings that we did not anticipate at the start of the project because we were open to challenging our assumptions and considering insights that complicated early hypotheses about how change would happen. For example, early learning questions about how to support the ongoing availability and sustainability of AMP were conceptualized as something NCOA and the state lead partner should provide to host sites. However, building a strong and sustainable delivery network was more nuanced and layered than this.
We learned that current and prospective host sites rely on one another for support and guidance in important ways, and the state lead partner in Colorado conceptualizes their relationship with host sites as bidirectional, looking to their knowledge and experiences to inform their statewide coordination and support work.
Our process evaluation was participatory and collaborative rather than solely relying on my expertise as the evaluator. I believe that learning and knowledge are not built in isolation but in collaboration with those working in the field to create the change they seek. This value is why we prioritized an approach to learning and evaluation that looked to the Aging Mastery® team, host sites delivering AMP, and additional statewide and local leaders as not solely data points but thought partners in planning and implementing our work together.
What does a participatory and collaborative process look like?
Here are key action steps:
- Give the process adaptable structure. We built an Evaluation Learning Agenda with all stakeholders to guide our work, and adapted that structure as needed.
- Interpret data collaboratively. We shared analyses based on data collected to date throughout the evaluation with people building the delivery network and delivering AMP to solicit feedback on what did or did not resonate with their experiences and interpret the findings collectively—rather than waiting until the end of the evaluation.
- Iterate! We built reflection and learning throughout the evaluation to inform real-time decision making about the evaluation and the network building efforts. For example, the Aging Mastery® team hosted calls with host site facilitators to provide space for community building and support. We collected quantitative (short polls) and qualitative (group conversations) data during these calls through which we learned that host sites wanted more opportunities to share best practices and lessons learned. This learning contributed to a format change with host sites facilitating short presentations at the start of the calls.
- Be flexible. We modified our learning questions and methods to be responsive to the COVID-19 pandemic as we collected data and reflected on what we were learning. Adaptive and iterative (as well as collaborative and participatory) approaches that are attuned to the sociocultural context in which evaluation takes place are characteristic of culturally responsive evaluation. Adapting and iterating our plans is something that we should be prepared for and willing to do—even when we are not in the middle of a pandemic—to help ensure evaluations are feasible, valuable, valid, and just.
At the end of the process, we hosted a Sensemaking session to convene the Aging Mastery® team, prospective and current host sites delivering AMP, and additional statewide and local leaders in Colorado. We used this time to collectively reflect on what we learned and generate next steps to support the development of strong and sustainable delivery networks in Colorado and beyond.
The Aging Mastery® team will use the evaluation results to learn how to best support, strengthen, and expand AMP through the widespread dissemination of best practices, tools, and insights, both in Colorado and nationwide.
In my role as a learning and change strategist, I am committed to partnering with individuals and organizations working to dismantle systems of oppression and build more just and liberatory futures. I see the change that the Aging Mastery® team, host sites delivering AMP, and additional statewide and local leaders are leading as going beyond service delivery and contributing to community building between and among older adults and local leaders.
