|This section briefly describes various types of data and surveys that can be used for an impact evaluation. |
In many countries, administrative data are the most accessible data source. Usually provided by line ministries and specialized agencies, these data describe specific activities and programs such as school enrollment, disease prevalence, malnutrition information, hospital expenses, road network information, and income and expenditure for decentralized units. This information is important in assessing levels of public and private inputs, outputs, and outcomes, as well as their distribution within the country. For example, it is possible to compare how the distribution of enrollment rates matches spending on primary schools; how the structure of health spending-- primary versus tertiary care --reflects disease patterns; or how agricultural productivity of main crops varies with land tenure patterns.
Administrative data can often provide an important entry into poverty analysis, especially if it they are used to compare need and demand for services. However, administrative data do not allow for cross-tabulating or analyzing poverty across different dimensions. For example, it is generally not possible to look at enrollment rates of children by the income group of their parents. (Multi-topic household surveys, which are discussed below, differ from administrative systems in that they allow the analyst to relate indicators with each other.)
A population census contains basic information on all citizens of a country. The census is carried out for all households to obtain basic information on the population, its demographic structure, and its location. The census is typically carried out by the national statistics institute, which then provides data to lower levels of government tailored to local information needs. Since the census covers the whole population, it is costly, and most countries conduct a census only once a decade. The census can provide policymakers with important data for planning in the years directly following its implementation, but its usefulness diminishes after that.
Since the census is carried out across millions of households, the information gathered is, by necessity, limited. Information on household income, consumption, disease patterns, and poverty perceptions are generally not included. However, the census usually contains descriptive statistics of the housing stock, access to basic services such as water, electricity, and sanitation; information on education and employment patterns, and population statistics. The census has the advantage of being able to provide information at low levels of aggregation, such as the municipality level. Census data are also an important tool to check how representative other surveys are. The usefulness of sample surveys can be increased substantially if they are combined with census information, for example for providing poverty maps.
Household surveys are essential for the analysis at the household level. The census covers the whole population in the country, surveys interviews only a subset, generally a small fraction, of all households. This sample of households must be carefully chosen so that the results of the survey accurately describe living conditions in the country, and different parts of the country.
Sampling should be based on mapping of actual settlements, including newly-formed informal urban settlements. Sampling is most often informed by a recent population census. The sample size -- the number of households interviewed -- will vary with several factors, including:
- the indicator that is to be measured (a survey that aims to measure countrywide averages of income may require a larger sample than a survey designed to measure the percentage of the population with water connection, in part because the latter is easier to measure);
- the level at which the policymaker needs the information (a national electricity connection rate will require fewer households to be interviewed than regional or district rates).
Different types of household surveys are presented in Table A, directly below.
Table A: Household Survey Types
Measurement and analysis of different poverty dimensions, their inter-relationships, and correlates
Time-intensive (collection and evaluation)
Demographic and health surveys
Health-poverty measurement, health behavior analyses, basic poverty diagnostics
Measurement of other dimensions of welfare limited
Analysis of employment patterns, wage income analysis (link to education)
Limited use for poverty and welfare measurement
Income-poverty measurement (or one other dimension)
Limited set of variables
Rapid monitoring surveys and service satisfaction surveys
Quick and cost-effective monitoring of key welfare indicators
Income-poverty measurement not possible, limited inputs
Measure actual rather than planned quality of service delivery by frontline providers
Time and cost-intensive
Living Standard Measurement Study (LSMS) surveys and other multi-topic surveys
Multi-topic welfare surveys, like the LSMS, are geared towards measuring and analyzing poverty and are important instruments for poverty diagnostics. LSMS surveys collect information on household expenditures and income, health, education, employment, agriculture, the ownership of assets such as housing or land, access to services, and social programs. Dozens of countries have implemented multi-topic surveys and many now have several rounds of surveys that allow rich comparisons across time.
Demographic and Health Surveys
These are special household surveys geared to exploring the incidence of diseases and use of health facilities. They collect anthropometric data—height, weight, and age of children, that can be used to calculate malnutrition rates--and many other health and health behavior-related variables that enable such factors as survival rates, birth histories, and disease incidences to be computed. The surveys also contain basic data about housing conditions, educational attainments, employment patterns, and sometimes household assets.
Labor ministries use employment surveys to gather information on employment and wages. These surveys include questions about household income, demographics, and housing features. They can be good sources for employment statistics, income-based poverty indicators—if the income module is good—and input indicators such as access to basic services. Employment surveys tend to be more important information sources for heavily urbanized countries.
Rapid Monitoring and Satisfaction Surveys
These surveys are generally large, contain relatively short questionnaires, and include predetermined data entry packages. They are easy to implement and have a rapid turnaround time. The Core Welfare Indicator Questionnaire (CWIQ)—widely applied in Africa—is one example. Satisfaction surveys are best viewed as complements to multi-topic household surveys and have been used in many countries to monitor access to and quality of basic services.
Many other, specialized surveys exist that can be used for poverty diagnostics. These can range from violence surveys--for example, in Lima, Peru--to opinion surveys--for example, those conducted by the Social Weather Station in the Philippines. Several countries also have surveys of health centers, schools, or other public institutions. Firm surveys can be essential to understanding the impact of crisis on employment and specific groups at risks and were used extensively in understanding the impact of the East Asian crisis. Food security assessments identify high-risk groups and are often used by relief organizations. Typically, the websites of national statistical institutes and international organizations will provide information about the availability of such data.
A. Facility Survey
Facility surveys are conducted to collect data at the level of service provision, e.g. a school or a health clinic. These are not as common as the three sources of data discussed above, and may be conducted as stand-alone surveys, or in conjunction with other surveys, such as a household survey or a specialized survey. Data from facility surveys is often most useful in analysis for providing control variables on infrastructure characteristics, e.g. student-teacher ratio for each school.
Facility surveys are used when service delivery is one of the key components of a project. Facility surveys can not only provide information on the demand and supply of services by households, but the influence of facility characteristics on health and educational outcomes of the population. Facility surveys can also be used to measure changes in productivity and efficiency of a facility as a result of interventions. There are many types of facility surveys that try to address different dimensions of service delivery, some of the examples from the health sector are listed below:
Public expenditure tracking survey (PETS) tracks the flow of resources to determine how much of the originally allocated resources reach each level, from the central government to spending at the facility level. It is useful to quantify leakage of funds at each level, and understand the processes to find solutions to problems, such as adequate staffing, textbooks, hospital equipment and drugs etc.
Quantitative service delivery survey (QSDS) evaluates efficiency of public spending and incentives by collecting data on inputs, outputs, quality, pricing, and oversight etc.
Both PETS/QSDS provide new information on “the complex transformation from public budgets to services. They typically consist of questionnaires for interviewing facility managers (and staff) as well as separate data sheets to collect quantitative data from facility records. A beneficiary survey can also be added.1” In addition, they can be used in conjunction with household surveys to look at the demand and supply side of service delivery.
Clinical Vignettes: These are typically used in health clinics to measure the quality of patient care. The doctor is presented with a hypothetical patient scenario and asked how he/she would respond. Or in other cases, an enumerator gets trained as a sick person and the characteristics of the illness are predetermined and the practitioner then asks questions and performs physical examination for diagnoses. These are useful in providing a standardized test for measuring service across facilities and provides additional information that explains differences in outcomes and impact of an intervention or project.
Exit Polls: can be done for patients alone or a sample of households if non-users are included. Focus group discussions and report cards are some of the few methods that are also used to gauge users’ satisfaction with the services. However, there are problems with interpreting the subjective perceptions of quality of service provided and it is difficult to interpret if there are systematic differences between clients. Additionally, there could be “courtesy bias", where individuals may provide responses that they are socially acceptable.
Qualitative research tools range from participatory assessments to ethnographic and sociological case studies, and institutional to political investigations. Some of these tools are described below in Table C. These tools help in gathering information that household surveys are not able to capture, and can complement the work done through other survey techniques.
Table C: Data Collection Methods for Qualitative and Participatory Assessments
Data Collection in:
Participant observation and more systematic data collection methods like structured interviews over a limited time span.
Anthropological research techniques, especially direct observation, to analyze the influence of ethnicity, gender, and village stratification on the household and group well-being and behavior.
Longitudinal Village Studies
Wide variety of methods ranging from direct observation and recording (tabulation), periodic semi-structured interviews with key informants (for example, health center staff) and village population, to survey interviews in several different observation periods.
Ranking, mapping, diagramming, and scoring methods are prominent besides open interviews and participant observation. The time horizon of participatory assessments is often short. They sometimes build on local populations describing and analyzing their own reality surrounding well-being.
Geographic Information Systems data, and Global Positioning System data provide fairly precise indicators on the locations of various infrastructure and geographic terrain features. For instance, GIS data may have indicators on the locations of schools, hospitals, public telephones, roads, rivers, etc. as well as data on elevation, contour, terrain type, etc. In evaluations where spatial relationships are important, GIS data can supplement other data to enrich the analysis. For instance, GPS readings could be recorded for each household in a survey, and this data could be used with existing or new GPS data to estimate travel times to the nearest market, police station, school, health post, etc. Recording GPS coordinates for each dwelling in household surveys also greatly facilitates the construction of panel data sets, by making it much easier to locate the dwellings to include them in a later round sample (see the section on panel data below).
1 Dehn J., R. Reinikka, and J. Svensson, “Source: Survey Tools for Assessing Performance in Service Delivery”, In Francois Bourguignon and Luiz Pereira da Silva (eds.). The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools. A copublication of the World Bank and Oxford University Press.
Sources: Coudouel et al. (2002), Poverty Measurement and Analysis, in the PRSP Sourcebook, World Bank, Washington D.C. and “Data for Impact evaluations”, World Bank, Washington D.C.