Introduction
The 2006 Annual Report on Operations Evaluation (AROE) updates the actions taken since the 2004 and 2005 AROEs to strengthen the results focus in monitoring and evaluation. It explores first the implications for managing for results in Bank operations and second the extent to which the Bank's M&E systems provide staff with the information they need to better manage for results. The report also evaluates the products and services of IEG. The AROE also makes recommendations for further progress on implementing the results architecture and on ways IEG might encourage more timely and widespread use of IEG findings by operational staff.
Emerging results architecture
Management welcomes this AROE, the third in as many years to focus on the Bank's progress in articulating a results architecture and procedures for sharpening its focus on results. This year's report confirms the relevance of the emerging results architecture. That architecture is grounded in the country business model that envisions a results management system focused on the articulation of an RBCAS as the overarching framework for prioritizing the Bank's program of lending and knowledge support to client countries. Key supporting elements are the CAS mid-term progress report and self-evaluation through the preparation of a CASCR. Last, the CASCR ratings of program achievements and Bank performance are validated by IEG.
The AROE's analysis confirms management's own assessment of recent progress (World Bank 2006b) and contributes more in-depth analyses, particularly of RBCASs and feedback from staff focus groups on the opportunities and challenges of strengthening the results focus of operational activities. The analysis is a useful complement to management's work to deepen its understanding of the key factors that will determine the success of its results initiative, notably how best to encourage staff efforts and improve the quality and utility of the key instruments in the results architecture.
AROE Analysis
The AROE catalogues the measures taken to strengthen the Bank's results orientation in M&E at the country, sector, and product levels, updating the overview in the 2003 and 2004 AROEs and focusing particularly on the current status of the M&E tools. Like management's recent update on the results agenda (World Bank 2006b), it notes progress but also areas for improvement.
Challenges to developing results-based monitoring
The AROE's compilation of the challenges to developing results-based monitoring systems, derived from focus group discussions within the Bank, yields a familiar listing of key issues: (a) accelerating work to help staff overcome gaps in their understanding of the practical dimensions of constructing useful results frameworks, (b) ensuring client capacity and buy-in, and (c) helping country and task teams integrate the demands of the results agenda into already overloaded work programs.
Overcoming internal challenges
Work to overcome many of these conceptual constraints to more active implementation (moving from seeing M&E as primarily a tool of relevance to project completion to viewing it as a tool to facilitate responsive management during implementation) is under way. Notably, management is developing a Bank-wide learning program that is supported by the Knowledge and Learning Board.
A critical element of creating a results culture in the Bank is to align the incentive and rewards structure with the results agenda. To improve incentives, management has made portfolio quality oversight a required performance evaluation item for sector managers. Staff have also been recognized Bank-wide in a variety of ways for their contributions to results, notably through the President's Awards for Excellence, IEG's Good Practice Awards, and the Results Showcase, which brought competitively selected good practice cases to senior management attention.
In addition, management has moved to increase the focus on results in its learning and accountability processes around operational products (RBCASs, operational product reviews during preparation and implementation, and the QAG process). More needs to be done, and management is committed to continuing to focus on incentives and expanding programs to communicate to staff the importance of this agenda. Two key elements going forward are management attention to the IDA 14 results measurement system and the results monitoring and learning system under preparation (see the results framework for the agenda on managing for results in the recent update) (World Bank 2006b, p. 33).
The global element in the focus on results
The AROE reviews progress internally in improving the results focus and externally in supporting countries in strengthening their focus on results. Management notes, however, that its work on the results agenda includes a third element, at the “global” level. The Bank has put considerable effort into harmonizing its approach with that of other development partners and working with them to align with country results efforts, a topic that this year's AROE analysis does not explore. Management's recent update, discussed at the Board's Committee on Development Effectiveness on March 8, 2006, documents the progress on this front (World Bank 2006b, p. 24).
Challenges at the country level
The AROE rightly cites these challenges. As noted in the recent update, in countries preparing PRSs, the Bank has continued to encourage countries to improve their PRS monitoring systems through ESW, lending for public sector management, training programs, and knowledge sharing. The Bank has given attention to helping partner countries gather national and subnational data for monitoring progress toward their PRS goals. Analytic work and policy advice focus on aligning allocations with the PRS priorities and on achieving operational efficiency, service delivery, and outcomes.
However, more needs to be done to stimulate greater demand for results within and across the line agencies that are most directly responsible for the delivery of public goods and services, as well as from citizens. One implication is that what is needed is a shift in incentives-from donor-driven results reporting to building the countries' capacity to provide credible evidence of effectiveness to their own citizens. Increasingly, management seeks to signal to staff that they are accountable for helping their country partners build systems and procedures that enhance accountability and transparency. One element in this process is the Development Impact Evaluation Initiative. This strategic scaling up of impact evaluations will provide countries with additional information that they can use in evidence-based decision making.
Self-evaluation tools and the bank's results agenda
Management agrees with the AROE analysis that there is a need to bring a results focus to the methods and tools that are used to improve performance during project implementation. The Rapid Results Approach cited by the AROE seems to hold promise, and management is assessing this experience. Management envisions this as the first of a series of collaborative learning activities across Regions, networks, and central service units designed to develop tools and examples that task and country teams can use in "real time" as they struggle to meet- and to help countries meet-the challenge of managing for results.
AROE Recommendations
The AROE 2006 proposes two core recommendations for management consideration:
Building on progress in advancing the results agenda, agree on a three-year action program with a corresponding budget to fund implementation of the next stage, differentiating new allocations and redeployed resources. Critical action items would be as follows:
Support country directors and country teams in their efforts to refine and use results frameworks at country and sectoral levels to manage country programs. Similar efforts need to be undertaken to strengthen the results frameworks of thematic and global programs.
Assess the effectiveness of (self-) evaluation approaches during project and country program implementation and provide guidelines to staff on their use.
Strengthen incentives and accelerate a results-oriented training and communications program for management and staff to encourage use of M&E information.
- To enhance countries' capacity and demand to manage for results by strengthening the collection and use of performance information, provide budgetary support to task teams and technical advice to countries that intend to institutionalize M&E systems.
Management views
Management broadly agrees with the thrust of these recommendations, with one major exception. A results focus is the responsibility of country teams and task teams and needs to be included in the formulation of RBCASs and in task preparation and task supervision budgets. To move forward on agreed priorities, a Results Steering Group has been formed to facilitate communication and coordination among Regional and network results teams, which are now charged with responsibility for leadership in identifying the priorities for implementing the action plan proposed in the recent progress report. A response to the AROE recommendations is included in the Summary of Management Response to IEG Recommendations (p. xv). |
RSS