|
CODE Chairman's Summary
|
|
|
|
On July 19, 2006, the Committee on Development Effectiveness (CODE) considered the 2006 Annual Report on Operations Evaluation prepared by IEG and the draft management response.
Background
The 2005 AROE assessed the use and usefulness of monitoring and evaluation information to improve results and enhance Bank effectiveness at the country level. In the Management Action Record (MAR), one of the recommendations for management was to establish clear lines of accountability for the follow-up and oversight of adopted recommendations and to make management responses more action specific to include timelines for implementation. For IEG, the recommendation was to institute a formal system for tracking and managing its recommendations. The Committee supported the proposed changes to the MAR, including a pilot three-year expiration criterion for MAR recommendations from fiscal 2006 onward. On March 8, 2006, CODE discussed two draft documents prepared by management that are relevant to the AROE: Accelerating the Results Agenda: Progress and Next Steps (World Bank 2006b) and Updating the Bank's Operational Policy on Monitoring and Evaluation (World Bank 2005d).
Report Findings
The 2006 AROE found that progress had been made on the 2003 action plan. Procedures were in place and results frameworks were being used in projects and RBCASs, but their effectiveness as a monitoring tool remained in doubt, and more work was needed at the sector and global program levels. Bank managers and staff faced challenges in managing for results such as conceptual difficulties in developing results chains, lack of client capacity and interest, competing priorities in the Bank, and lack of appreciation for the use of performance information and evaluations for management decision making. Moreover, disconnects in outcome ratings between country and sector policy outcomes and project outcomes pointed to a gap in understanding contributing factors to results.The AROE recommended establishing fully funded priority actions. Assessing IEG's effectiveness, the report recommended that IEG's results framework should continue, and even enhance, its focus on the use of evaluation findings.
Management Response
Management broadly agreed with the main thrust of IEG recommendations, which are in line with its own findings and the priorities for action set out in its February 2006 update to CODE, with one major exception. It noted that a results focus was the responsibility of country teams and task teams and needed to be included in the formulation of RBCASs and in task preparation and task supervision budgets.
Overall Conclusions
Speakers welcomed the opportunity to review management's and IEG's actions to strengthen Managing for Development Results. They noted the findings that the Bank has strengthened the results focus of its operations, while major challenges lie ahead. The Committee was gratified to observe a broad agreement between IEG and management on the main findings and recommendations, which it also generally endorsed. It also accepted IEG's clarification that it was not suggesting additional budgetary allocations but rather stronger prioritization in deployment of resources, a position closer to what was enunciated by management in its response. There were also a few questions regarding specific statistics reported, including some from the client survey and the MAR.
Several members concurred with IEG that accountability was as important as incentives and resources are to promoting the results agenda but also to achieve better results. Members also endorsed greater IEG efforts to enhance the usefulness and application of IEG products, particularly by operational staff. In this regard, the timing of Country Assistance Evaluations (CAEs) vis-à-vis the CAS was cited as an area where some improvement might be possible. A number of speakers raised questions about the effectiveness of the results architecture being put in place, including prioritization of areas and statistical capacity. The role of the Board in raising results issues when discussing Bank's operations was highlighted.
Next Steps
There was agreement about a two-year cycle for AROE; the report will be considered by the Committee, followed by a Board discussion. Given the likely Board agenda in August, the Committee recommended that the Board consider this year the AROE and the draft management response on a streamlined procedure (that is, without discussion). (Initially a Board meeting was scheduled for August 22).
The following main issues were raised during the meeting:
Results architecture and results framework. The Committee welcomed the progress of the Bank in implementing the results agenda. Yet several speakers had questions about the effectiveness of the results architecture being established. A member expressed concerns about possible distortions favoring more measurable activities and against more complex priority areas such as governance or rural development. The robustness of the system of indicators, given weak statistical capacity in many countries, was also questioned. A member stressed that countries were at the center of the effort to strengthen the focus on results and that a substantive demand from governments in this respect was instrumental. Management noted that there was limited demand from governments due in part to data gaps at the country level (yet these gaps would not be so expensive to fix) and lack of relevance in the decision-making process. Management found that there was a risk of distortions to only wanting to do things that are measurable.
Some speakers commented on “disconnects” between country and sector policy outcomes and project outcomes. In this regard, efforts to focus on results were perceived as uneven and inconsistent among and within countries, Regions and networks, rendering it difficult to assess the Bank's overall contribution to development results. Two members noted that frameworks appeared to be better developed in International Development Association countries than in International Bank for Reconstruction and Development countries, although issues of quantity and quality of data were raised. IEG noted that some reasons for "disconnects" and perception of inconsistency were the difficulty of indicators and measurement due to the political economy of reform or the problem of attribution of a single straightforward intervention on a range of outcomes. In response to comments about the importance of harmonization, management noted that the multilateral development banks were entering the second year of doing a comparative assessment of their respective works on results (COMPAS). One member urged management to redouble their efforts to monitor and evaluate results of trust fund activities. Management stated that they are working with trust fund and global programs to increase their results focus.
Members stressed the need for staff to better understand the results framework as a tool to link Bank operations with outcomes in country programs and projects, and to strengthen client capacity to manage for results, particularly low-income countries. Further analyses on project outcomes, baselines, and targets, especially in RBCASs and Poverty Reduction Strategies, were suggested. In this regard, consideration should also be given to time (time needed to implement the results framework and the time lag between project completion and outcome). Staff from the Africa Region provided information about efforts in promoting the results agenda and building capacities on results.
Organizational issues and incentives. Members commented on the need to change the organizational culture, align incentives and reward structure, and disseminate successful experiences for learning purposes. Management replied that the current culture, in which staff perceive greater incentive to report up than across, will be addressed by the development of the Results Monitoring and Learning System. The need for a better communication strategy was also mentioned. More analysis was sought about overload, time pressures, availability of resources, and difficulties in developing the results chain. Speakers and management shared the Director-General, Evaluation's views regarding the key role of accountability in implementing the results agenda. One member sought information on recent activities of the RSG. Management responded that the group has started to function and is expected to play a key role in increasing attention Bank-wide to results and in communicating and consolidating gains across the Bank.
Funding priorities. IEG proposed designated funding for implementing the results agenda differentiating new allocations and redeployed resources. Some members commented on special budgetary support to task teams. The Committee took note of IEG's clarification that it was not asking for additional budgetary allocation but rather stronger prioritization in deployment of resources.
Recommendations for IEG. The Committee welcomed the AROE's recommendation on the role of IEG in promoting the results agenda. There were other comments on the need to improve timeliness of IEG evaluations, including timing of CAE and CASCRs; to analyze the relative strength of results frameworks by region and sector, particularly to help operationalize the findings; and to track and publish the results of sectoral, Regional, or project analysis as well as their trends to facilitate learning. One member felt IEG needed to focus on new and ongoing operations. Another member commended IEG for the efforts to develop its own results framework and noting the difficulties of constructing measures that link final outcomes to IEG interventions. Regarding the client survey, one member expressed concern about the perceived decline in the quality and depth of IEG's analysis and recommended that IEG analyze the trend. IEG clarified that surveys did not show such trends in regard to external clients but that this was an internal issue that had to be sorted out.
Regarding the content of the AROE itself, a member made some suggestions to add some flesh to the report's substance and stressed the need to use the AROE to help promote “out-ofthe-box” thinking on the results focus by exploring such topics as how to better link the evaluation findings to administrative budget and human resources management and how the increased use of innovative work and experimentation- for example, conditional cash transfer-can foster the results agenda. IEG promised a more specific AROE in two years and took note of the suggestions.
Management Action Record. A member expressed concern about the statistic that the adoption of six percent of IEG's recommendations was negligible. This member also warned against the three-year expiration criterion acting as an incentive for delaying implementation of the recommendations. IEG explained that this figure reflected the relatively high number of new recommendations entered in the MAR in 2006. Members took note that a summary of the MAR will be posted in the Bank's external Web site for disclosure. |
|
|
|
RSS