Our Impact Evaluation Toolkit helps practitioners design and implement impact evaluations, with a focus on Results-Based Financing in maternal and child health programs. The Toolkit consists of 1) guidelines for best practices and 2) tools to implement guideline recommendations such as:
terms of reference for team members/survey firms
maternal and child health indicators of interest
training manuals and curricula
fieldwork supervision materials
Video: Public Randomization Ceremony in Cameroon
Cameroon, February 2012: a public ceremony was held to randomly allocate health facilities to the treatment and comparison groups of the impact evaluation of the Results-Based Financing program financed by the World Bank. This real-life randomization increased local buy-in for the fairness of allocation of the intervention across geographic areas. Click to watch.
The Impact Evaluation in Practice guide is a companion piece of the Impact Evaluation Toolkit. It helps policymakers and practitioners understand available impact evaluation methods, and make decisions based on evidence of what works.
Experience from the Field: Country Spotlights (PDFs)
Our Country Spotlights give you real-life experience from the field in designing and implementing impact evaluations. Scroll over the world map and learn from what other countries have been doing.
In this module, you will learn why it's important to evaluate Results-Based Financing interventions. You will find out how to establish a theory of change and a results chain, and how these help develop hypotheses that an impact evaluation can test through evaluation questions. You will gather food for thought by browsing through first-generation questions on whether Results-Based Financing works, and second-generation questions on how Results-Based Financing can work better. At the end of this module, you should know how to develop impact evaluation questions relevant to the context you work in. These questions should respond to key aspects of interventions and fill knowledge gaps on your topic of interest.
In this module, you will find out how to set up an impact evaluation team, as well as define the roles and time commitments of each team member. At the end of this module, you will be ready to start building and contracting a team that can achieve the tasks you need to accomplish.
Module 2: Building the Impact Evaluation Team - Guidelines
In this module, you will dig into data collection activities and data quality safeguards. You will see how to define the sample and sampling frame in the field and create unique identifiers for geographical areas and study arms. You will understand the benefits of data collection pre-test and proper data entry management plans. You will learn to plan for fieldwork supervision, recruit and train field teams, conduct a pilot test and, finally, how to manage fieldwork and ensure regular reporting on fieldwork activities. At the end of this module, you should know how to provide sufficient training, support, supervision and communication channels to ensure quality fieldwork activities and data.
Module 5: Implementing the Data Collection - Guidelines
In this module, you will find out why properly storing and documenting data matters, how this can be done and who is responsible for it. You will discover the benefits of the World Bank data catalog and its key features. You will also be able to distinguish which types of data should be stored, where to store it and who should access it. At the end of this module, you will know how to ensure that survey respondents are not put at risk, how to keep the outcome of the survey safe with minimal risk of damage or loss, and how to provide easy access for the impact evaluation research team.
In this module, you will learn the functions of each impact evaluation report (baseline, midline, endline), get useful hints on how to clean the data, create variables including indicators of interest, and how to validate the design of the impact evaluation at baseline. You will receive guidance on impact analysis to supplement the impact analysis methods presented in Impact Evaluation in Practice (Gertler et al. 2011). Finally, you will learn how to build your analysis in the spirit of efficient dissemination to key stakeholders. At the end of this module, you should be ready to start analyzing the data while both preserving its integrity and planning for sharing results.
Module 7: Analyzing Data and Disseminating Results - Guidelines
In this module, you will learn how monitoring and documenting program implementation can help inform the relevance of impact evaluation design and analysis. You will discover examples of program implementation adjustments that can affect the impact evaluation, and why it is important to monitor program implementation. You will also learn how complementary data sources can help save monitoring costs and time. At the end of this module, you will know what to monitor, what can be used to monitor interventions, and how documenting results can help ensure the impact evaluation stays relevant over time.
Module 8: Monitoring and Documenting RBF Programs - Guidelines
Since 2010, over a dozen country teams have embarked upon evaluating the impact of various RBF interventions with support from the Health Results Innovation Trust Fund. Interestingly, teams often didn't need much convincing that an evaluation would be valuable in generating evidence on the impact of RBF on health status and health systems. At the same time, many teams were asking for concrete tools to help them implement these evaluations on the ground. Numerous emails went around asking for terms of reference for a survey firm or examples of health facility questionnaires, and it soon became clear that something needed to be done to share knowledge more effectively. This toolkit is the result of three years of work with and technical assistance to those teams, and brings together the tools and guidelines that were developed by and for the different teams.
The toolkit was assembled by three authors: Christel Vermeersch (World Bank), Elisa Rothenbuhler (World Bank) and Jennifer Sturdy (Millennium Challenge Corporation, formerly World Bank). However, this could not have been done without the contributions of country teams, who shared their own tools, tested the newly developed tools, helped refine them, and provided valuable feedback to ensure their relevance in the field.
The toolkit was developed with generous funding from the multi-donor Health Results Innovation Trust Fund (HRITF). The authors' views expressed in this publication do not necessarily reflect the views of the World Bank.