Determining the counterfactual is at the core of evaluation design. It is, however, quite tricky to net out the program impact from the counterfactual conditions that are likely to be influenced by contemporaneous events, selection bias, and contamination. Details of a program can and should affect the choice of the evaluation method and how it is carried out. In particular, the evaluation should be designed to minimize any risk that the study itself might compromise the program by altering the program’s delivery in some fundamental way, changing the type of individual who would be served, or changing the behavior of the members of the control group. The following operational issues of an evaluation study might compromise the objective of the program and thus might make it difficult to conduct the study or to draw any inference once it is conducted. - Altering the services delivered: Sometimes the evaluation study can provide new information that might alter the program’s delivery in some fundamental way thus affecting outcomes. Also, the study can overburden program staff with research requirements, reducing their time and energy to deliver services.
- Changing Target Population: Research requirements can skew selection of participants so that the population served by the pilot program is different from the intended recipients of the program itself. For example, individuals who are willing to participate in the evaluation study may differ from individuals who are simply willing to receive services. See selection bias.
- Contamination of the control group due to (a) attrition, (b) migration between treated and untreated groups and (c) information leakage between the two groups and (d) unintended receipt of program services by the group.
- Constraints on time and resources: Random assignment and prospective comparison group designs (including prospective reflexive studies) need a baseline survey. So, they are more expensive than retrospective comparison group designs which do not have to allow time for building up a sample for treatment and for assessing post program effects. This is an important issue to take into account when designing an evaluation. Also, given the constraints, there is always a tension between conflicting objectives- investigating short run effects versus long-term effects of the intervention.
Besides the operational problems cited above, there are some general problems with drawing economic conclusions from empirical evaluation studies. These problems pose threats to internal validity as well as external validity. Internal validity refers to whether one can validly draw the inference that within the context of the study the differences in the dependent variables were caused by differences in the relevant explanatory variables. External validity deals with whether effects found in an experiment can be generalized to different individuals, contexts, and outcomes.
The threats to internal validity could be due to several reasons: - Omitted variables, for example, events, other than the experimental treatment, occurring between pre-intervention and post intervention observations that provide alternative explanations for the results.
- Trends in outcomes, when processes within the units of observations produce changes as a function of the passage of time per se, such as inflation, aging, and wage growth.
- Misspecified variances, causing overstatement of the significance of statistical tests due to effects such as the omission of group error terms that indicate that outcomes for individual units are correlated.
- Mismeasurement, that is, changes in definitions or survey methods that may produce changes in the measured variables.
- Political economy, for example, endogeneity of policy changes due to governmental responses to variables associated with past or expected future outcomes.
- Simultaneity, that is, endogeneity of explanatory variables due to their joint determination with outcomes.
- Selection bias, that is, assignment of observations to treatment groups in a manner that leads to correlation between assignment and outcomes in the absence of treatment.
- Attrition, which is the differential loss of respondents from treatment and comparison groups.
- Dropouts of some of the experimental treatment group members from the social program under study prior to receiving some or all of the treatment. This is different from the attrition problem as persons drop out of the program, but not out of the experimental data.
- Omitted interactions, for example, exclusion of interactions such as differential trends in treatment and control groups or omitted variables that change in different ways for treatment and control groups.
The threats to external validity are just the possibility that there are important interactions between the treatment and individual characteristics, location, or time such that results from an evaluation study cannot be generalized to different individuals, contexts, and outcomes. These interactions are as follows: - Interaction of selection and treatment: Unrepresentative responsiveness of the treated population. The treatment group may not be representative of certain population, or the treatment may be different from that which one would like to examine.
- Interaction of setting and treatment: The effect of the treatment may differ across geographic or institutional settings.
- Finally, interaction of history and treatment: The effect of the treatment may differ across time periods.
Related Section:
See Data & Data Sources for a collection of data initiatives collected for evaluation purposes and for a guide to qualitative and quantitative evaluation instruments.
|
RSS