Click here for search results

Site Tools

Impact Evaluation Toolkit

  • Home
  • 1. Evaluation Questions
  • 2. Team Building
  • 3. Design
  • 4. Data Collection Preps
  • 5. Data Collection
  • 6. Data Storage
  • 7. Analysis
  • 8. Monitoring
  • 1 of 3
During the impact evaluation of a Maternal and Child health intervention in Rwanda, the enumerator faces the common challenge of getting accurate information on child vaccination from a damaged vaccination card.
featured

Toolkit Provides Hands-On Guidance

Our Impact Evaluation Toolkit helps practitioners design and implement impact evaluations, with a focus on Results-Based Financing in maternal and child health programs. The Toolkit consists of 1) guidelines for best practices and 2) tools to implement guideline recommendations such as:

  • terms of reference for team members/survey firms
  • maternal and child health indicators of interest
  • research protocols
  • questionnaires
  • training manuals and curricula
  • fieldwork supervision materials
Public randomization ceremony in Cameroon, 2012
featured

Video: Public Randomization Ceremony in Cameroon

Cameroon, February 2012: a public ceremony was held to randomly allocate health facilities to the treatment and comparison groups of the impact evaluation of the Results-Based Financing program financed by the World Bank. This real-life randomization increased local buy-in for the fairness of allocation of the intervention across geographic areas.
Click to watch.

A woman is being interviewed in Bangladesh with due care to protecting respondent's confidentiality.
featured

Need More Information on Impact Evaluation Methods?

The Impact Evaluation in Practice guide is a companion piece of the Impact Evaluation Toolkit. It helps policymakers and practitioners understand available impact evaluation methods, and make decisions based on evidence of what works.

Experience from the Field: Country Spotlights (PDFs)

Our Country Spotlights give you real-life experience from the field in designing and implementing impact evaluations. Scroll over the world map and learn from what other countries have been doing.

World Map with country spotlights

In this module, you will learn why it's important to evaluate Results-Based Financing interventions. You will find out how to establish a theory of change and a results chain, and how these help develop hypotheses that an impact evaluation can test through evaluation questions. You will gather food for thought by browsing through first-generation questions on whether Results-Based Financing works, and second-generation questions on how Results-Based Financing can work better. At the end of this module, you should know how to develop impact evaluation questions relevant to the context you work in. These questions should respond to key aspects of interventions and fill knowledge gaps on your topic of interest.

Module 1: Choosing Evaluation Questions - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

In this module, you will find out how to set up an impact evaluation team, as well as define the roles and time commitments of each team member. At the end of this module, you will be ready to start building and contracting a team that can achieve the tasks you need to accomplish.

Module 2: Building the Impact Evaluation Team - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

In this module, we will teach you how to design your impact evaluation. Steps include

  1. defining output and outcome indicators of interest that translate research questions into observable variables;
  2. determining an identification strategy, including the "arms" of a study that respond to research questions and guarantee a valid comparison group;
  3. making sure the sample has sufficient power to detect changes across a representative population;
  4. knowing which data will be needed to measure indicators of interest;
  5. how the timeline, team composition and budget must align and adjust simultaneously; and
  6. how to plan for dissemination activities. At the end of this module, you will know how to design an impact evaluation.

Module 3: Designing the Impact Evaluation - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

In this module, you will learn essential steps that lay the groundwork for data collection. Steps include

  1. scheduling data collection,
  2. reaching formal agreements ex-ante with stakeholders on data ownership,
  3. making sure participants in the survey are protected by defining a research protocol and getting the study ethics right,
  4. determining a data entry strategy and what's needed from survey firm(s),
  5. hiring one or more survey firm(s), and, finally,
  6. adapting and preparing data collection instruments and entry tools. At the end of this module, you will be ready to go into the field.

Module 4: Preparing the Data Collection - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

In this module, you will dig into data collection activities and data quality safeguards. You will see how to define the sample and sampling frame in the field and create unique identifiers for geographical areas and study arms. You will understand the benefits of data collection pre-test and proper data entry management plans. You will learn to plan for fieldwork supervision, recruit and train field teams, conduct a pilot test and, finally, how to manage fieldwork and ensure regular reporting on fieldwork activities. At the end of this module, you should know how to provide sufficient training, support, supervision and communication channels to ensure quality fieldwork activities and data.

Module 5: Implementing the Data Collection - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

In this module, you will find out why properly storing and documenting data matters, how this can be done and who is responsible for it. You will discover the benefits of the World Bank data catalog and its key features. You will also be able to distinguish which types of data should be stored, where to store it and who should access it. At the end of this module, you will know how to ensure that survey respondents are not put at risk, how to keep the outcome of the survey safe with minimal risk of damage or loss, and how to provide easy access for the impact evaluation research team.

Module 6: Storing and Accessing Data - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

In this module, you will learn the functions of each impact evaluation report (baseline, midline, endline), get useful hints on how to clean the data, create variables including indicators of interest, and how to validate the design of the impact evaluation at baseline. You will receive guidance on impact analysis to supplement the impact analysis methods presented in Impact Evaluation in Practice (Gertler et al. 2011). Finally, you will learn how to build your analysis in the spirit of efficient dissemination to key stakeholders. At the end of this module, you should be ready to start analyzing the data while both preserving its integrity and planning for sharing results.

Module 7: Analyzing Data and Disseminating Results - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

In this module, you will learn how monitoring and documenting program implementation can help inform the relevance of impact evaluation design and analysis. You will discover examples of program implementation adjustments that can affect the impact evaluation, and why it is important to monitor program implementation. You will also learn how complementary data sources can help save monitoring costs and time. At the end of this module, you will know what to monitor, what can be used to monitor interventions, and how documenting results can help ensure the impact evaluation stays relevant over time.

Module 8: Monitoring and Documenting RBF Programs - Guidelines

Download full document: English [PDF, 3.83MB] / Español [PDF, 3.47MB] / Francais [PDF, 3.97MB]

The Toolbox

Since 2010, over a dozen country teams have embarked upon evaluating the impact of various RBF interventions with support from the Health Results Innovation Trust Fund. Interestingly, teams often didn't need much convincing that an evaluation would be valuable in generating evidence on the impact of RBF on health status and health systems. At the same time, many teams were asking for concrete tools to help them implement these evaluations on the ground. Numerous emails went around asking for terms of reference for a survey firm or examples of health facility questionnaires, and it soon became clear that something needed to be done to share knowledge more effectively. This toolkit is the result of three years of work with and technical assistance to those teams, and brings together the tools and guidelines that were developed by and for the different teams.

The toolkit was assembled by three authors: Christel Vermeersch (World Bank), Elisa Rothenbuhler (World Bank) and Jennifer Sturdy (Millennium Challenge Corporation, formerly World Bank). However, this could not have been done without the contributions of country teams, who shared their own tools, tested the newly developed tools, helped refine them, and provided valuable feedback to ensure their relevance in the field.

The toolkit was developed with generous funding from the multi-donor Health Results Innovation Trust Fund (HRITF). The authors' views expressed in this publication do not necessarily reflect the views of the World Bank.

Afghanistan Argentina Benin Cameroon CAR DR Congo Nigeria Rwanda Zambia





Permanent URL for this page: http://go.worldbank.org/IT69C5OGL0