|Impact Evaluation (IE) rigorously measures the impact that a project has on beneficiaries. It typically does this by comparing outcomes between beneficiaries and a control group, both before and after a project has been implemented. |
It is important to do IE rigorously and accurately in order to arrive at a true picture of the best way of achieving desired results for clients. These desired results can include meeting the Millennium Development Goals, or achieving the targets under the Africa Action Plan.
With very few exceptions, rigorous and accurate IE uses one or more of the methods listed below. Each of the methods seeks to find the true causal impact of a program by not only observing the beneficiaries’outcomes before and after the project, but also answering the question “what would have been the beneficiaries’ outcomes in the absence of the project?”, both before and after the project.
“What would have happened to these beneficiaries in the absence of the project” is called the counterfactual. The counterfactual is never observed for the beneficiaries themselves at the same time as they are in receipt of the project. Instead, the counterfactual is typically approximated by control groups or comparison groups, as in all the methods below.
Which of the methods is used, typically depends on a combination of the way in which program roll-out has been designed, what is operationally feasible, and what is technically most rigorous from an IE point of view.
This technique randomly assigns a pre-determined fraction of the eligible beneficiaries to the project, creating what's called the treatment group. The remaining eligible beneficiaries make up the control group. The difference in outcomes between the treatment and control group is the impact of the project.
Randomization is often the fairest way to allocate beneficiaries to a project if there is insufficient resource to roll out the project to all eligible beneficiaries at once. That is, if the project needs to be phased in to eligible beneficiaries over a number of years due to operational or budgetary constraints, the fairest way to decide who receives the project in the first year is often random assignment. By this method, each beneficiary has an equal chance of receiving the project in the first year.
Instrumental Variables is a technique that identifies a factor that determines receipt of a project, but which does not influence outcomes of interest. This factor is then used to simulate who would have been in the treatment group, and who would have been in the control group if receipt of the project was based on that factor. The difference in outcomes between these simulated treatment and control groups is then the impact of the project.
Regression Discontinuity Design
This is used when a cutoff point on a continuous variable such as a poverty index is used to determine who receives a given project. The impact of the project can then be estimated by comparing outcomes for beneficiaries who just qualify for the project on this score, with outcomes for individuals who just fail to qualify for the project given their score.
The difference in a given outcome between recipients of the project (the treatment group) and a comparison or control group is computed before the project is implemented. This difference is called the “first difference”. The difference in outcomes between treatment and control groups is again computed some time after the project is implemented, and this is called the “second difference”. Under the difference-in-difference technique, the impact of the project is the second difference less the first difference. The logic is that the impact of the project is the difference in outcomes for treatment and control groups after the project is implemented, net of any pre-existing differences in outcomes between treatment and control groups that pre-date the project.
Propensity Score Matching
Propensity score matching is a tool for identifying a suitable comparison group to compare to the recipients of the project (the treatment group). Essentially, propensity score matching finds a comparison group comprising individuals who did not in fact receive the project, but who, given their observable characteristics, had the same probability of receiving the project as individuals in the treatment group. The project’s impact is then the difference in outcomes between the treatment and comparison group.
This method compares outcomes of beneficiaries who have already received the project, with those who have not yet received the project but are about to. This method relies on the assumption that the beneficiaries who have already received the project, are similar to those who are about to receive the project.