Measuring the results and impact of capacity development has been at the center of reflections, analysis and operational tools of the World Bank Group and many other international organizations. Due to its long-standing commitment with knowledge and learning, particularly the WBG’s Leadership, Learning and Innovation (LLI) Vicepresidency has spearheaded efforts in this area. One key example is the Capacity Development Results Framework (CDRF) launched in 2009. Today, groundbreaking LLI initiatives such as the client-focused Organizational Knowledge Sharing (OKS) program are meeting an increasing demand, which also comes with a responsibility to plan, document and account for results. Precisely to face this challenge and learn from others’ experiences, an OKS task force convened M&E experts from various WBG branches, including the Independent Evaluation Group (IEG), the Operations Policy and Country Services (OPCS) Vicepresidency and LLI teams such as CL4D.
In a three-hour immersion session held on 27 October 2015, the participants looked into how to build a strong M&E basis for OKS, a program with multiple audiences (both external and in-house), stakeholders and levels of impact. OKS is currently engaged with 10 public institutions from Colombia, Ethiopia, India, Indonesia, Kenya, Nigeria and Uganda. These clients are investing in organizational arrangements and operational capacities to manage and replicate their best solutions in sectors such as agriculture, disaster risk management, governance, health and urban development (more details in this ppt).
The LLI engagements offer services ranging from day-to-day support by national consultants, to specialized training sessions and innovative tools such as smartphone applications. Key areas include strategic visioning, in-house knowledge governance, solutions capturing, and methodologies for peer learning. While primarily financed by a Korean trust fund, most support is embedded into the capacity building components of WBG lending operations, and increasingly linked to the WBG Global Practices. Following successful pilot engagements, OKS is now designing a structured approach to M&E, not only for accountability purposes, but also for learning, quality assurance and dissemination of this agenda vital for the achievement of the Sustainable Development Goals (SDG).
By sharing their day-to-day practice during the M&E event, colleagues from IEG, OPCS and CL4D enriched current thinking of the OKS task force. Among key aspects, the participants looked into the options for a strong theory of change, the scope of indicators, ways for linking OKS M&E to lending operations, and practical tools for M&E. For the theory of change, Jenny Gold from LLI suggested the OKS team might further translate its overall ambitions into specific result chains. Today, OKS aims to strengthen the client’s operational effectiveness, organizational sustainability and the scale-up of its most powerful solutions. With its support portfolio expanding, it is a good moment for OKS to build logical links between these outcomes (intermediate outcomes, according to the CDRF) and the capacity services offered by OKS.
A lively debate addressed the quality of indicators to plan for and measure results from OKS engagements. Based on suggestions from Hirut M’cleod (CL4D), there was a consensus that OKS should build useful core indicators, rather than trying to capture the manifold processes of change it tends to generate with country clients (see Hirut's ppt here). Jenny Gold stressed the need to only capture results for which OKS can claim contribution, thereby avoiding excessive ambitions (see Jenny's ppt here). The OKS team still assumes that apart from outputs and intermediate outcomes, there is a need to look into broader context-specific outcomes. This is especially true where final users benefit from clients’ services that are increasingly based on best practices and continuous learning. In many cases, OKS support helps clients to overcome rigid and suboptimal service delivery. One emerging example can be found in the work with the National AIDS Control Organization (NACO) in India where the best AIDS prevention approaches are being transferred from successful state agencies to their peer who are still in the process of improving their services to vulnerable groups such as sex workers.
As its support to clients is increasingly mainstreamed into WBG lending operations, the OKS program might indeed move quickly to identify core indicators capable of capturing high-value outcomes and impact. Christopher Nelson from IEG emphasized the need for OKS to clearly state the transformational value added by OKS to WBG operations. Borrowing from the Emergent Learning approach, the central framing question for OKS might ask two questions: How does a stronger knowledge culture at country clients contribute to better value from the money lend to these clients?, and What is OKS contributing to this precise effect? Jenny Gold invited OKS to look into how its engagements could lower the institutional capacity risk, a key concern of many Global Practices when working clients in difficult contexts. The OKS team is already advancing in this direction of mainstreaming, and plans to conduct a series of M&E pilots with country clients that are also benefitting from large-scale lending operations.
Finally and importantly, the gathering enabled OKS to learn from existing tools and their relevance in the operational practice. Vera Wilhelm from OPCS provided valuable insights into the tools of the Emergent Learning approach, such as before and after action reports, or the emergent learning tables which enable reflective action in a comprehensive, yet disciplined way (see Vera's ppt here). Hirut M’cleod explained that apart from using pragmatic, “non-burdensome” tools (e.g. tables, rather than reports), it is vital to align as far as possible to client’s systems, while also strengthening M&E capacity and commitment of the national consultants who are driving the OKS support in daily business. While the OKS team has been inspired by the manifold approaches and options available at the WBG family, one key take-away is precisely that M&E needs to be smart, focused, light and disciplined. Bridging these four characteristics is already at the core of the OKS task force’s plans for the future M&E work around country engagements.
To learn more about the OKS program and its services, check out our sister community at knowledgehubs.org!