Click here for search results

Subscribe to Mailing List

Estimation Methods

 

Contents

Comparison of means

Multi-variate regression analysis 

Instrumental variable method 

Double difference or difference-in-differences methods 

Estimation methods broadly follow evaluation designs, with the determination of the counterfactual being the core of the evaluation design. Different ways to establish controls under different evaluation designs determine the methods to be used to measure the counterfactual:

  • Randomized controls allow individuals to be placed randomly into two groups—those that receive the intervention and those that do not. This allows the researcher to determine program impact by comparing means of the outcome variable between the treatment and the control group.
  • Quasi-experimental (non-random) methods can be used to construct controls when it is not possible to obtain treatment and comparison groups through experimental design. With constructed controls, individuals to whom the intervention is applied (the treatment group) are matched with an “equivalent” group from which the intervention is withheld and the average value of the outcome indicator for the target population is compared with the average of that for the constructed control.
  • Another nonrandom method of obtaining control involves reflexive comparisons when participants who receive the intervention are compared to themselves before and after receiving the intervention.
  • Statistical controls or instrumental variables are used in cases when participants and non-participants are compared controlling for other characteristics which may be statistically different between the two groups.

Estimation methods can be classified into four categories: comparison of means, multi-variate regression, instrumental variables, and difference in difference.

Comparison of means
An estimation method to be used with experimental design

Comparison of means is a method of estimating impact using experimental design that involves comparing means of treatment and control groups. A random allocation of the intervention among eligible beneficiaries creates comparable treatment and control groups. The program impact on the outcome being evaluated can be measured by the difference between the means of the samples of the treatment group and the control group.

Multi-variate regression analysis
An estimation method to be used with non-experimental design

Multi-variate regression analysis is used to control for possible observable characteristics that distinguish participants and non-participants. Selection bias is similar to an omitted variable bias in regression analysis. Thus, if it is possible to control for all possible reasons why outcomes might differ, then this method is valid to estimate the program or treatment effect. The differences in the mean outcomes of the two groups, participants and non-participants, conditional on the set of variables that cause outcome and participation, constitutes the program or treatment effect.

Instrumental variable method
An estimation method to be used with non-experimental design

Instrumental variable method is used in statistical analysis to control for selection bias due to unobservables. These variables are such that they determine program participation, but do not affect outcomes. In the IV method, one uses one or more variables that matter to participation but not to outcomes given participation. This identifies the exogenous variation in outcomes attributable to the program, recognizing for example, that its placement may not be random but purposive. The “instrumental variables” are first used to predict program participation; then one sees how the outcome indicator varies with the predicted values. Often, one can use geographic variation in program availability and program characteristics as instruments especially when endogenous program placement seems to be a source of bias.

Double difference or difference-in-differences methods
An estimation method to be used with both experimental and non-experimental design

Double difference or difference-in-differences methods compare a treatment and a comparison group (first difference) before and after the intervention (second difference). This method can be applied in both experimental and quasi-experimental designs and requires baseline and follow-up data from the same treatment and control group.

A baseline survey is conducted for the outcome indicators for an untreated comparison group as well as the treatment group before the intervention followed by a follow-up survey of the same sampled observations as the baseline survey after the intervention. If the sampled observations tend to differ in the follow-up survey from the baseline survey, then they should be from the same geographic clusters or strata in terms of some other variable.

The mean difference between the “after” and “before” values of the outcome indicators for each of the treatment and comparison groups is calculated followed by the difference between these two mean differences. The second difference (that is, the difference in difference) is the estimate of the impact of the program (A special case of double differences is “reflexive comparison” that only compares the treatment group before and after the intervention).


Related Sections:

  • See Data & Data Sources for a collection of data initiatives collected for evaluation purposes and for a guide to qualitative and quantitative impact evaluation instruments.
  • See Training Events and Materials for presentations on how to employ the methods and techniques introduced here



Permanent URL for this page: http://go.worldbank.org/T5NWV3AV20