Methods & Techniques

In determining which techniques to use when conducting an impact evaluation, one must consider evaluation designs, estimation methods, and implementation issues.

• Evaluation designs hinge on the fundamental question, "What would the situation have been if the intervention had not taken place?" Although it is impossible to observe such a situation, it is possible to approximate it by constructing an appropriate counterfactual, or group of non-participants in a program. Evaluation designs are determined by the choice of methods used to identify the counterfactual and can be broadly classified into three categories that vary in feasibility, cost, and degree of selection bias:
  • Experimental
  • Quasi-experimental
  • Non-experimental

• Estimation methods broadly follow evaluation designs. Different designs require different estimation methods to measure the welfare of the counterfactual. The four categories of estimation methods are:
  • Comparison of means
  • Multi-variate regression
  • Instrumental variables methods
  • Double difference or difference-in-difference

• Implementation issues  may compromise the results of an impact evaluation. They generally fall under two categories:
  • Operational issues - compromise the objectivity of the program and thus might make it difficult to conduct the study or draw any inference from it. They deal with the complexity of attempting to net out the program impact from the counterfactual conditions that are likely to be influenced by contemporaneous events, selection bias, and contamination.
  • Threats to validity - general problems with drawing economic conclusions from empirical evaluation studies. They may prevent drawing valid inferences within the context of the study (internal threats), or from generalizing the effects found in an experiment to different individuals, contexts, and outcomes (external threats).

Related Section:

• See Data & Data Sources for a collection of data initiatives collected for evaluation purposes and for a guide to qualitative and quantitative evaluation instruments.
• See Training Events and Materials for presentations on how to employ the methods and techniques introduced here.