NEW: Monitoring and Evaluation in the United States Government: An Overview
The U.S. government’s long history of concern with accountability and oversight continues and evolves in new directions. The last two decades have seen an increased interest in outcomes-based performance monitoring and evaluation. Important new legislation, special attention by the administrations of Presidents Bill Clinton and George W. Bush, greater emphasis on transparency and results, and the introduction of initiatives such as the Program Assessment Rating Tool all have brought performance measurement more prominence. Furthermore, President Obama’s administration has introduced additional initiatives addressing performance-based management and promoting program evaluation, so that more advances may be seen. Still, there are a number of ways in which the U.S. government can improve its use of M&E to increase the effectiveness of its programs.
The Australian Government's Performance Framework
There has long been a keen interest from countries around the world in Australia’s experience in creating an evaluation system to support evidence-based decision making and performance-based budgeting. Australia’s evaluation system lasted from 1987 to 1997, and during that time it was used to systematically evaluate all government programs every three to five years; these evaluation findings were used heavily by officials, ministers and the Cabinet in the annual budget process. This paper updates two previous World Bank papers that reviewed the Australian experience with
M&E and other performance-related initiatives.
Writing Terms of Reference for an Evalulation: A How-To Guide
The terms of reference (ToR) document defines all aspects of how a consultant or a team will conduct an evaluation. It defines the objectives and the scope of the evaluation, outlines the responsibilities of the consultant or team, and provides a clear description of the resources available to conduct the study. Developing an accurate and wellspecified ToR is a critical step in managing a high-quality evaluation. This publication provides a step-by-step guide to preparing effective TORs for evaluation.
Performance in Government: The Evolving System of Performance and Evaluation Measurement, Monitoring, and Management in the United Kingdom
Performance measuring and management in the British public service has become almost ubiquitous over the past three decades. Virtually every part of the UK public services produces publicly available performance data which has progressively started to play a role in central decision making. Considerable capacity to formulate, monitor and analyse performance information has evolved. The UK has been successful in developing a fairly comprehensive performance and evaluation measurement, monitoring, and management system which, over time, has become increasingly "outcome" focussed, although at various levels it still includes large elements of output, process and even input monitoring. This paper offers a brief review of the evolution of the UK governments' performance measurement, monitoring and management systems since 1997, when the performance measuring practice became almost universal across public activities.
Improving the Quality of Public Expenditures through the Use of Performance Information in Mexico (also in Español)
This compilation reflects efforts to institutionalize a performance-based culture in the Mexican public sector through the introduction of various reforms, including performance budgeting. It also addresses the achievements, challenges, and opportunities to improve the quality of public spending. Steps to make such changes have come through monitoring and evaluation approaches that can be replicated or expanded; sectoral efforts to improve the performance of priority programs; Congress’s use of information on the results of public spending; the implementation of performance budgeting at subnational levels; and the harmonization of accounting between the three levels of the federal government. All these aspects are key elements of comprehensive reform.
The Canadian M&E System: Lessons Learned from 30 Years of Development
The Canadian M&E system is one that has invested heavily in both evaluation and performance monitoring as key tools to support accountability and results-based management. The M&E system has evolved and changed over the past thirty-plus years, as officials have gained more experience with the use of M&E and/or new demands for public sector reforms have placed a new emphasis on the use of M&E in public sector management. This paper provides an overview of the model for monitoring and evaluation employed by the government of Canada, examining how the concepts have evolved and identifying key lessons learned from more than thirty years of public sector experience.
Implementing a Government-wide Monitoring and Evaluation System in South Africa
This paper analyses the process of designing and implementing a government-wide monitoring and evaluation (GWM&E) system in South Africa. The case study provides a number of interesting lessons relevant to other developing countries as well as to developed countries. South Africa has followed a "big bang" approach in its efforts to build a national M&E system, starting as recently as 2005. The country has pursued capacity building and system building with an initial focus on monitoring; a conscious decision was made to pursue evaluation later. Interestingly, M&E development is being allowed to evolve, rather than follow a completed and detailed blueprint.A number of major M&E initiatives have been pursued over the past three years, and considerable success has already been achieved. This paper provides an honest assessment of the remaining challenges, including the barriers to achieving more of a performance culture in the government and civil service.
Evaluation of Government Performance and Public Policies in Spain
This paper covers selective aspects of Spain’s experience in evaluating government performance and public policies. Rather than a cohesive "evaluation system", there is instead a constellation of organizations, with evaluation mandates and/or practices, which are not interrelated. These organizations and their respective practices have been evolving without coordination over the past three decades. An evaluation culture is slowly emerging, amid different conceptual approaches used by different organizations that are managing and/or conducting evaluations.
Mexico's M&E System: Scaling Up from the Sectoral to the National Level - This paper provides a brief review of Mexico’s progressive movement from a sectoral to a governmentwide M&E system. It highlights the critical institutional reforms introduced, the policy decisions, and the most important operational steps that were taken, offering an account of the political context within which such changes and decisions were possible. The paper emphasizes the specific role of the National Council for the Evaluation of Social Policy (CONEVAL) as an innovative development, which furthered institutionalization of evaluation and a results focus at the federal government level through the implementation of the System for Performance Evaluation (SED). A quick review of the governmentwide system’s strengths and main challenges for the future is offered. Some lessons for other countries and conclusions follow. (also in Español)
Implementing a Subnational Results-Oriented Management and Budgeting System: Lessons from Medellín, Colombia - The purpose of this paper is to describe the budget process reform implemented in Medellín, and to analyze its actual performance and evaluate its success. The reform is changing the way public resources are allocated and executed, while gradually institutionalizing supply and demand-side practices beyond the government’s political cycles. This paper describes and analyzes how the RoB was designed and implemented, and the achievements of the system to date, in terms of resource allocation and the policy-making process. The following section describes the context of Medellín: political and economic trends, and the objectives of the budget reform. The third section presents the state of the budget process in Medellín as it was diagnosed before implementation. Section four describes the implementation process of the system, and section five addresses its major achievements. A final section includes some policy recommendations, which can be divided into two parts: (i) recommendations for consolidating the system as a main support tool of a RoM approach, and (ii) recommendations for replicating the system in other municipalities and subnational governments.
Insider Insights: Building a Results-Based Management and Evaluation System in Colombia - This paper examines the experience of institutionalizing Colombia’s M&E system, from the perspective of a former SINERGIA manager: Manuel Fernando Castro, Director of Public Policy Evaluation, Department of National Planning. It identifies the main differences between Colombia’s monitoring and evaluation (M&E) system and other experiences of its kind and illustrates the extent to which an M&E system goes hand-in-hand with systemic public sector reform. Accordingly, it complements previous studies by other authors with an insider’s account, description, and analysis of the obstacles encountered and the results obtained. Likewise, it presents recommendations on how to ensure further institutionalization and identifies several lessons that might be of interest to professionals who are developing evaluation systems in other countries.
Instituting Impact Evaluation Within the Framework of a Monitoring and Evaluation System - This booklet describes the progress being made in the transition from individual impact evaluation (IE) studies to building a systematic approach to identifying, implementing, and using evaluations at sector and national levels whereby IE is seen as an important budgetary planning, policy formulation, management, and accountability tool. The institutionalization of IE has been achieved in a relatively small number of developing countries, mainly in Latin America, but many countries have already started or expressed interest in the process of institutionalization. This paper reviews this experience in order to draw lessons on the benefits of an institutionalized approach to IE, the conditions that favor it, the challenges limiting progress, and some of the important steps in the process of developing such an approach.
How to Build M&E Systems to Support Better Government Governments in a number of developing countries are devoting considerable efforts to strengthen their monitoring and evaluation (M&E) systems and capacities. They are doing this to improve their performance ― by establishing evidence-based policy-making and budget decision-making, evidence-based management, and evidence-based accountability. This publication is a synthesis of IEG's considerable experience on this topic. Spanish Version | French Version | Bahasa Version
A Diagnosis of Colombia’s National M&E System, SINERGIA provides in-depth information on the genesis, development and current status ― strengths, weaknesses and options for the future ― of one of the most advanced M&E systems in Latin America. Spanish Version | French Version
Impact Evaluation -- The Experience of the Independent Evaluation Group of the World Bank - This paper discusses various methodological issues to consider when conducting an impact evaluation, and it argues for a mixed-methods approach which combines quantitative rigor with well-contextualized studies -- in particular using a theory-based evaluation design. A number of examples of impact evaluations conducted by the Independent Evaluation Group are presented to illustrate these issues.
Conducting Quality Impact Evaluations Under Budget, Time and Data Constraints - This booklet has been written for project and program managers who wish to conduct a rigorous impact evaluation, to help them select the most rigorous methods available within the constraints they face. Spanish version
Experience with Institutionalizing Monitoring and Evaluation Systems In Five Latin American Countries: Argentina, Chile, Colombia, Costa Rica and Uruguay - This paper provides a comparative analysis and taxonomy of five countries which have sought to institutionalize government-wide M&E systems. Spanish version
Towards the Institutionalization of M&E Systems in Latin America and the Caribbean - These conference proceedings focus on the experience of good-practice or promising countries, including Chile, Colombia, Mexico and Brazil. Spanish version
Institutionalization of Monitoring and Evaluation Systems to Improve Public Sector Management
This paper reviews the growing body of experience in this topic in a range of countries, and identifies various success factors and mistakes to avoid. Spanish version
Evaluation Capacity Development in the Republic of Ireland - This paper analyses the key drivers of, and barriers to, the institutionalization of M&E in the Irish government.
Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs - This OED study presents examples of 8 evaluations that had a significant impact. For many of them, it has been possible to compare their costs with the benefits which they provided, and to show that the evaluation was a highly cost-effective management tool.
Influential Evaluations: Detailed Case Studies - This companion volume discusses 8 case studies in depth. It presents key lessons about how to design evaluations which are influential.
Building Country Capacity for M&E in the Public Sector: Selected Lessons of International Experience - This paper presents the findings from two in-depth case studies (Uganda, Egypt), and from the growing literature on country experiences.
Evaluation Capacity Development: OED Self-Evaluation - This report describes OED's work to help countries strengthen their monitoring and evaluation systems. It assesses the scope, results, quality, strengths and weaknesses of this ECD work.
Second Edition. A thumbnail sketch of various types of M&E is provided in Monitoring and Evaluation: Some Tools, Methods and Approaches. This includes: their purpose and use; advantages and limitations; costs, skills and time required; and key references. Arabic version; French version; Portuguese version; Russian version; Spanish version
An Assessment of the Impact of Bangalore Citizen Report Cards on the Performance of Public Agencies - This OED review finds that report cards have had a significant impact on the quality of public services in Bangalore. The review identifies success factors and considers the reliability of the report card approach elsewhere.
Two Generations of Performance Evaluation and Management Systems in Australia contrasts a successful, centrally-driven system of performance evaluation with the later less-successful, devolved system which focuses on the collection and reporting of performance information.
The Development of Monitoring and Evaluation Capacities to Improve Government Performance in Uganda focuses on the large number of M&E systems which currently exist, and the need both to simplify them and to focus more on issues of development effectiveness. This paper provides an update on the detailed diagnosis prepared for Uganda in 2001. French version; Spanish version
The history of OED's involvement in ECD and the nature of its catalytic support for ECD is discussed in chapter 11 of OED: The First 30 Years.
OED's 2002 Annual Report on ECD (AREC) clarifies the meaning and relevance of Bank work on ECD, lays the methodological foundations for subsequent evaluations of the Bank’s ECD work, and presents an inventory of ECD activities under way across the Bank, with special attention to country-level activities. A Precis of this report is also available. (French version; Spanish version).
The DAC Working Party on Aid Evaluation has prepared a Glossary of Key Terms in Evaluation and Results Based Management to clarify concepts and help reduce terminological confusion. The terms are presented in English, Chinese, French, Italian, Japanese, Portuguese, Russian, and Spanish.
Developing African Capacity for Monitoring and Evaluation reports on a seminar and workshop held in Johannesburg in September 2000. It outlines recent progress and the latest thinking about M&E in the African and donor evaluation communities. A Precis of this seminar is also available. (French version) (Spanish version)
Guide For Reviewing Organizations Providing M&E Training provides a list of key issues to consider when assessing the ability of national bodies such as universities and civil service colleges to provide M&E training.
Assessment of Ghanaian Organizations Which Provide M&E Training is the first application of the new Guide. It includes an assessment of five principal Ghanaian training organizations.
Strengthening Capacity for Monitoring and Evaluation in Uganda: A Results Based Management Perspective is an example of the application of the ECD diagnostic guide. It provides a detailed diagnosis of M&E capacities and discusses options to strengthen them.(French version; Spanish version)
A Diagnostic Guide and Action Framework presents a detailed list of issues to consider when assessing the readiness of a country to develop its M&E functions. A menu of ECD options is presented, together with the key demand and supply factors to consider when designing an action plan (French version) (Spanish version)
The Role of Civil Society in Assessing Public Sector Performance in Ghana reports on the proceedings of a one-day workshop to examine the role that civil society wants to play in assessing public sector performance in Ghana, the main impediments to playing this role, and priority steps for strengthening civil society engagement in this area.
Comparative Insights from Colombia, China, and Indonesia identifies key factors which have influenced the institutionalization of M&E in these countries (French version).
Public Sector Performance--the Critical Role of Evaluation is the proceedings of an April 1998 World Bank seminar which identified success factors, the experience of several developing and developed countries--Chile, Indonesia, Zimbabwe, Australia and Canada--and of donors providing support for ECD.
Evaluation Capacity Development in Asia reports on a conference held in Beijing in October 1999. The links between M&E and results-based management are one of the many topics addressed.
Sub-Saharan Africa: Lessons from Experience in Supporting Sound Governance outlines the links and synergies between ECD and sound governance, including transparency and accountability--via provision of information on the efficiency, effectiveness and quality of government services. Pressures from civil society, the media, parliament and donors for a greater focus on government performance can help foster more of a results orientation with government.
Evaluation Capacity Development in Africa is the proceedings of a joint World Bank/African Development Bank seminar held in Abidjan in November 1998. Donor experience and country challenges in Africa are presented. A Precis of this seminar is also available (French version)
Lessons from National Experience discusses the factors underlying successful efforts to build M&E functions in governments. Implications for donor support are identified.
Zimbabwe: Issues and Opportunities looks at the constraints and potential for ECD
The Development of Australia´s Evaluation System looks at the success factors and impediments in that country, which is often regarded as having had the most systematic and comprehensive approach to evaluation of any developed country.