Management Comments

The risk, of course, is that the Bank’s effectiveness as measured by IEG in its evaluations may be taken out of the context of the challenges inherent in development assistance or compared across agencies with less strong evaluation functions. The ARDE differs from most IEG outputs in that it draws on a wide range of evaluative findings and provides a summary overview. Because of the nature of the ARDE, it does not make specific recommendations for management and does not elicit specific management responses to issues raised in the document. However, management finds it important to provide its views on ARDE findings and to give an overview of a set of key management actions that are directly relevant to those findings.

Organization of the Comments. This note first provides comments on the findings and suggestions on the Bank’s overall performance in part I of the ARDE. It then responds to what IEG puts forward in part II on the special topic for this year, sustainable development. The final section of the note provides a brief set of conclusions.

Tracking Bank Performance
Management is encouraged to note the rebound in the quality of Bank- supported operations reported by IEG, with 81 percent of exiting projects in fiscal year 2008 rated as satisfactory (moderately satisfactory, satisfactory, or highly satisfactory) in terms of meeting development objectives, exceeding management’s target of 80 percent. Management also notes that the ARDE confirms the continuing underlying positive trend in the quality of the operations supported by the Bank. The ARDE goes back to 1993 to look at the evolution of Bank performance; clearly, the Bank of today is very different from the Bank in 1993. The focus on results is much more central now. As an example, monitoring and evaluation (M&E) is a key element for monitoring projects to achieve results, and the ARDE cites a weakness in Bank M&E for projects exiting the portfolio in fiscal years 2007 and 2008, projects that on average entered the portfolio early in this decade. More recent experience is different: 59 percent of new IDA projects that filed supervision reports for the first time in fiscal 2009 had baseline data for all project development outcome indicators and 85 percent had baseline data for at least one. Management expects a significant change in IEG findings on M&E when these operations exit the portfolio in the coming years. However, it is important that the Bank do more to adapt its work to enable it to be more agile and innovative in supporting the needs of borrowing countries, in particular paying more attention to implementation and to adapting support quickly in response to new information or changing circumstances and to improve performance in lagging Regions and sectors.

Addressing Long- Standing Issues— the Bank's Results- Focused Reform Efforts
To address the challenges of a changing Bank and to take on long- standing issues, including those identified in the ARDE, management has an extensive results- focused response effort under way across virtually the whole spectrum of Bank operational support— from Country Assistance Strategies (CASs) to investment lending, development policy operations, and analytic and advisory activities (AAA).

CASs. During fiscal 2010 the Bank will implement a set of actions based on a recent review of experience with results- based CASs. These actions include guidance on how to rate Bank performance under the preceding CAS and on preparing results frameworks for new CASs. The CAS Progress Report process will be strengthened, providing an early opportunity for gauging progress and the potential need for adjustments. Guidance has already been issued on assessing country statistical capacity— an issue cited by IEG as important for the results agenda.

The Investment Lending Reform Program. Several of the issues raised by the ARDE are ones that management is addressing through investment lending reform (World Bank 2009). Specifically, these reforms are aimed at adapting and differentiating the investment lending model to better fit the kind of support that the Bank provides in sectors such as health and public sector management, where there is no "engineering blueprint" or way to calculate a credible expected economic rate of return, but a need for a more flexible design and more emphasis on support to partners during implementation. The central tenet of the reforms is a stronger focus on a clearly defined set of results the project is designed to achieve (with well- defined monitoring indicators), combined with the identification of the key risks to achieving those objectives and a regular reevaluation of these risks throughout implementation. Those changes, which will focus management and staff attention more on implementation, are designed to overcome the problem of overoptimism cited by IEG.

Project Restructuring. IEG cites the issue of an apparent reluctance to restructure projects that are not meeting development objectives. A key part of investment lending reform is a reworking of the confusing guidelines for project restructuring, which staff cite as a major factor in delayed restructurings. Bank executive directors will be asked in the coming months to approve a simplified restructuring process.

Development Policy Operations. The framework for development policy operations was modernized in fiscal 2005. To date, results from self- and independent evaluation have been robust (as reconfirmed in the ARDE). Recent self- evaluation has identified ways to further strengthen performance. In fiscal 2010, management will provide stronger guidance to staff on supporting borrowers on the design of development policy objectives (including in the context of joint budget support with other donors) for development policy operations and in consolidating good practice on poverty and social impact analysis.

Cost-Benefit Analysis and Impact Evaluation. The ARDE cites the decreased use of cost- benefit analysis and the potential of impact evaluation. Management looks forward to working with IEG on its upcoming in- depth review, notably on when cost- benefit analysis is most appropriate and how useful it is relative to or along with alternative tools for appraisal, monitoring, and evaluation across the spectrum of investment projects supported by the Bank. Management agrees with IEG that impact evaluation can be a valuable tool and has recently expanded its use across sectors and Regions in operations where it is most appropriate and cost- effective and owned by the client country. Again, management looks forward to working with IEG in 2010 on its impact evaluation analysis.

AAA. Management is taking on reforms to its results framework for AAA in the context of its work on a new knowledge agenda. The approach is to look at AAA in a more programmatic fashion, as opposed to individual products. Management is actively working with IEG on the new AAA results framework, because a common approach and understanding is central to strong self- and independent evaluation. These are sequenced for later in the reform program timetable. (In general, management would note the importance of sequencing these reforms to keep them within staffing and budget constraints.)

The Results Agenda Going Forward. To consolidate all of the above, management is working to establish a “results monitoring platform” to strengthen monitoring and reporting on the outcomes of Bank- supported activities. One key step is the adoption of core sector indicators— a few standardized indicators for each sector that are comparable and aggregable at the institutional, country, or sector level, which will facilitate reporting on activity outputs and outcomes, especially under the IDA Results Measurement System (IDA RMS). The core indicators for the road transport, water supply, education, and health sectors were launched in July, together with an improved data system for collecting and tracking quantitative data. Core indicators for other sectors (urban, environment, micro, small, and medium enterprises, and so on) are also under preparation. These indicators (mainly output and outcome indicators appropriate for tracking at the project level), together with the country- or sector- level indicators available from other sources (for example, Tier 1 indicators), will improve our ability to focus on results. In addition, the new results platform offers staff and managers tools for monitoring country portfolio results and for making more strategic and effective use of impact evaluations where these are appropriate. IDA at Work narratives are also being expanded, with an emphasis on thematic and country level briefs.

Country Capacity. Management agrees with IEG on the importance of building country M&E capacity, both as a goal in itself and to enable us to better understand our own development effectiveness. For example, in partnership with other donors, the Bank is launching a new Statistics for Results Facility to help the poorest countries build capacity to increase the availability of development statistics, starting with those that are most urgently needed for the implementation of strategies for poverty reduction and national development. More broadly, country capacity building has always been part of Bank activities, especially since Accra; for example, the Bank-wide emphasis on greater use of country systems also includes a greater reliance on and development of country M&E systems. (IEG is also active in this area through activities to build evaluation capacity). However, we recognize that this is a longer- term effort.

Lagging Regions and Sectors
IEG, while commending the Bank on the increase in operational quality, notes a history of poor performance in certain Regions and sectors. Work is ongoing to address this issue. In particular, management has taken on two, the Africa Region and the health sector, as a priority.

The Africa Region’s Quality Enhancement Program. To improve outcomes of the operations it supports, the Africa Region has taken the following actions:

1. Restructuring the weakest- performing portfolios (HIV/AIDS, regional integration, fragile states)
   
2. Providing just- in- time operations and M&E support to country and task teams to improve the results frameworks and ensure better compliance with IDA15 monitoring requirements
   
3. Training task team leaders in the development of results- focused operations and programs and results- based M&E systems
   
4. Developing sector results chains to help improve the underlying causal logic in designing operations
   
5. Developing the Africa Results Monitoring System (ARMS), which is now being rolled out as a Bankwide system to track results for IDA credits. The ARMS entails the development of clear and realistic outcome- based project development objectives as well as standard ized and measurable indicators. It will harmonize and facilitate results reporting.

The ARDE provides an early indicator of progress. It cites fiscal 2008 data showing 77 percent of projects in the Africa Region as satisfactory (moderately satisfactory, satisfactory, or highly satisfactory) in terms of meeting their development objectives, up sharply from fiscal 2007.

The Health Sector. The Bank’s Health Network has taken action to improve the outcomes of country health sector operations it supports. Based on the findings of an IEG evaluation of the health sector, it developed a portfolio- quality improvement plan and launched it in fiscal 2009. It is regularly monitored by senior management to ensure that it stays on track.

The Management Action Record (MAR) 
As noted above, management values independent evaluation and takes it very seriously. As a result, it takes seriously the recommendations that come out of IEG evaluations. All Draft Management Responses are discussed by executive directors (normally at the Board’s Committee on Development Effectiveness— CODE). These documents include an initial MAR that sets out what management commits to do as a result of the IEG recommendations. Given the importance of these commitments, which often have resource implications, they are vetted and cleared by senior management. Management then revises these commitments in light of the CODE discussion. Management reports annually to IEG on progress against its commitments. While this system is valuable and necessary as an accountability tool, management agrees with IEG that it is time for a reevaluation.

Possible Improvements to the MAR. The biggest issue is that in its annual monitoring of progress, IEG starts with its recommendations, rather than with the part of the matrix that sets out management’s commitments— the actions management agreed to undertake and for which it can be held accountable. That approach often leads to disagreements over ratings on progress, notably when management agreed only partially with a recommendation, or when, as in the case of the IEG evaluation of analytic and advisory services, management noted that it would need to sequence many of the actions later in the overall reform effort. Another key issue is the sheer number of recommendations and subrecommendations. The current MAR contains 63 separate recommendations, often with a significant number of subrecommendations under each recommendation. (In addition to IEG, management commits to take action in response to the findings of other entities, including the Inspection Panel, the Internal Audit Department, and the Integrity Vice Presidency.) Management would like to work with IEG on a system that reduces and prioritizes IEG recommendations and monitors progress directly against management commitments.

Achieving Sustainable Development 
Management welcomes IEG’s attention to the issue of sustainable development. The ARDE draws on a range of findings, including two recent IEG evaluations. As management noted in its responses to those evaluations, there are several areas of agreement: the centrality of environmental sustainability in the Bank’s work, the importance of country ownership, the key role for analytic support to countries, and the need for strong collaboration across sectors and across World Bank Group institutions on environmental sustainability issues. At the same time, management expressed concerns about some of the analysis and recommendations (IEG 2008c, 2009a). However, it is more important to note how management is using the lessons summarized in the ARDE going forward.

Bank Progress on Environmental Mainstreaming. The ARDE emphasizes that environmental improvement requires mainstreaming environmental awareness into investment decisions and technology choices across sectors. Management believes that the criterion on which Bank performance for environmental mainstreaming is now measured— confluence of projects with environmental themes with sector board mapping— is a secondbest indicator. Background work for the Environment Strategy 2010 will address how to improve environmental mainstreaming results and will also assess to what extent, after only two years, the merger of the two Bank networks into the Sustainable Development Network is leading to stronger mainstreaming outcomes. Management is continuing to strengthen the ability of staff in the areas of forestry, natural resource management, and energy to deliver integrated climate change– based advice to clients by supporting learning modules on climate change–related tools and mechanisms, such as ecosystem- based adaptation and Reducing Emissions from Deforestation and Degradation (REDD).

Measuring and Achieving Project, Country, and Global Results. The ARDE emphasizes that in the new Environment Strategy, the Bank should include a focus on better monitoring and reporting of environmental outcomes at the project and country levels. In the IDA15 RMS exercise, management is making progress in creating environmental outcome indicators for each environmental subtheme, all of which will be aggregated across sectors. Country- level environmental reporting is under development in a “green” data sets reporting format. Management agrees on the importance of expanding environmental partnerships, especially for strengthening environmental governance, across all relevant institutional actors, especially at the country level. To this end, the preparation of the Environment Strategy 2010 will involve extensive consultations with such external stakeholders as governments, the private sector, civil society, and development partners. The consultations will seek to identify key areas in which the World Bank Group should engage to support its clients in more effectively addressing environmental priorities and moving toward greater environmental sustainability and will help inform the development of a results framework for measuring progress in priority areas. In addition, the strategy will address how the World Bank Group can better support client countries in their efforts to strengthen the environmental institutions and governance mechanisms that are crucial for sustainable development.

Conclusions
Management appreciates that submitting the Bank to the learning and accountability discipline of independent evaluation is a strength that sets it apart from many other development institutions, notably given IEG’s acknowledged standing as best among its peers. It is confident that the benefits are worth the costs of being held to this high standard, with the resulting misunderstandings that can arise. Even in a balanced report like this one, the positive outcomes risk being overlooked relative to the less positive findings. Management accepts the overall message from the ARDE— a finding of generally good and improving quality in the operations the Bank supports, but also a strong challenge to do even better, notably in supporting a results culture in partner countries and in the Bank and strengthening Bank support for sustainable development. These are priorities that management has taken on in its fiscal 2010 work program and beyond, notably in a set of actions that address some longstanding issues found by both self- and independent evaluation.