On Day 1 we rampaged through 17 evaluations from 14 countries, all done since the Policy Research Report on CCTs was published in 2009. We saw evaluations of huge national programs and tiny comparative treatment arm experiments and debated the pros and cons of impeccable control groups versus questions about external validity. On day 2, we paused to reflect on the new frontiers of CCTs, when several more evaluations were interjected into the discussion. For a change the big majority of evaluations were from countries outside of Latin America -- 5 from Africa, 1 from the Middle East, 2 from East Asia and 1 from South Asia.
The clearest and most unanimously agreed message from the workshop is that there is an increasingly long vector of outcomes shown to be positively affected somewhere by some CCT program - many concerned with health and education, but also with work, livelihoods and investments in productive or housing investments, and another vector associated with youth risky behaviors and mental health. This should not perhaps be surprising, as we know that poverty brings with it a dizzying array of deprivations and risks. But it has implications for thinking about the role and design of transfer programs, and for ensuring that when evaluations are done, they are comprehensive. We may have been underestimating total benefits or not captured unintended consequences because we weren’t measuring them all.
Opening the Black Box – Which Design Features Matter in Producing Human Capital Outcomes?
We improved our understanding of the black box, but there are still not very firm answers on some key questions, partly because the findings vary, and partly because CCTs involve multiple outcomes and different countries (or analysts) give them different values.
Do Conditions Matter?
The role of conditionalities took a central stage at the opening of the workshop. The issue is important, as conditions can have big implications for who will and won’t benefit from the transfers, may impose costs on households, and monitoring their compliance requires some serious administrative systems. These tradeoffs are especially critical in low-income settings. The first careful and innovative study going at the heart of the question is the Malawi scholarship experiment for adolescent girls. As Berk Ozler pointed out in his presentation (and here) conditional transfers were more effective than unconditional transfers in raising school enrollment and attendance, which is the outcome made salient by the conditionality. Yet, unconditional transfers brought about improvements on a larger set of outcomes, ranging from the choice of less risky sexual partners, to delayed fertility and marriage. In Burkina Faso, conditions did matter to enrollment of younger children (late school start is common in Burkina) and for the use of preventive services. In the Morocco multi-arm experiment, the conditionality did not really make a difference, possibly because many households did not understand how it was supposed to work.
The Puzzle of the Low Elasticity to the Transfer Amount
The evidence from four scholarship programs with comparative treatment arms (in Cambodia, Morocco, Malawi and Nigeria) and from a pilot in Pakistan seems to indicate that the small benefit amounts, on the order of 3-5% of household welfare, yield large increases in enrolment.
The implications of these findings are potentially powerful, but we aren’t sure the question of how much to pay is fully settled.
First, these results stand in contrast with evidence from Latin America where the direct/opportunity cost of schooling may be binding. New research from Progresa/Oportunidades shows that the program raises secondary school enrolment only for those households whose family network includes households with primary school children. As Francisco Ferreira pointed out in his discussion, we should learn more about the range of transfers amounts where the elasticity is still high, and that may vary by context and program objectives.
Moreover, though the potential to use small(er) transfers to produce the desired behavioral changes in enrolment, retention or graduation, is undeniably attractive, raising human capital is often NOT the main or only objective of CCTs, and low transfers are much less effective for the short run poverty objectives of these programs.
The Power of Framing of Conditions
The idea that it is neither cash nor conditions that drive results, but ‘something else” got a lot of discussion. Ben Olken suggested that behavior will depend on the way the message of the conditionality is framed and internalized by the households. Ester Duflo suggested that in the Morocco experiment, the increases in schooling resulting from a relatively small cash transfer not understood to be conditional are likely to be driven by the increased awareness and salience of education. The qualitative work from Malawi also emphasized the importance of the ‘culture of school’ as a key factor in explaining the gains. How expectations, aspirations and projections into the future get changed by these programs is still an unexplored but critically important area of research in the future. Larry Aber convincingly argued for the need of a theory of change, where mechanisms, mediators and moderators are clearly outlined, as well as and for the need to broaden the scope of inquiry, fostering interdisciplinary work with psychology and communication science. Economists are now making the same call for ‘mechanisms’ experiments to complement policy evaluations and to directly test those parts of the causal chain (mediators) where the knowledge gap is largest (see here).
The New Nuanced Evidence on the intra-household Decision Making:
The experiments for Malawi and Opportunity NYC helped us learn that on the surface there is no conflict of interest between the parents and the children in terms of educational outcomes. However, enrollment and school attendance aren’t the only outcomes that matter. In Malawi, the mental health of the girl deteriorates where large transfers to her parents depend on her school attendance. In contrast, in Opportunity NYC, there was no decline at all in mental health among beneficiary children. On the other dimension of intra-household decision making, the Burkina Faso and Moroccan experiments imply that issuing payments to mothers rather fauthres seems to be marginally more effective for education outcomes, but not for health. Indeed, Richard Akresh ended his presentation on the Burkina Faso experiment with the words “fathers aren’t so bad”. The explanation for the different results is likely to lie in how income pooling and separate spheres of command vary across cultures, which begs for more situation specific and qualitative work to really understand the underlying pathways.
Going from Conditions on Use of Services to Those on Outcomes
As Norbert Schady suggested in his final remarks: maybe “you get what you pay for” that is, since CCTs are targeted on enrolment or attendance, we should not expect impacts on learning. After all, prior evidence showed no additional learning despite increased schooling in Mexico, Colombia, and Cambodia. New evidence presented in the workshop for Morocco, Nigeria and Malawi was more optimistic, showing learning gains in all three countries.
We need to understand more fully the interplay of different factors in different settings. Lack of final outcomes could be due to the fact that the program brought or kept less advantaged children in school (selection effect) or due to the lack of response of the supply side (crowding). Perhaps, if maximizing learning is the main objective, this could be achieve by either incentivizing graduation and tertiary enrolment (as in a Colombian experiment analyzed by Barrera-Osorio et al 2011), or explicitly targeting test scores (as in a Mexican experiment Berhman, Parker, Todd, Wolpin 2011), where providing incentives to both the demand (students) and the supply side (teachers) yields larger gains in tests score than giving incentivize to each separately. In a broader human capital policy perspective, Costas Meghir pointed out that investment in early childhood development would increase the returns to later schooling investments. While transfers may play an important role in this domain (see here and here), it is not at all clear whether conditionalities beyond those already present for health or health education are needed or desirable.
And of course equity and efficiency need to be balanced. If poorer students are less academically prepared, basing conditions on performance may make it difficult for the poor to comply and benefit, a consideration which has been given heavy weight by Latin American policymakers.
We are grateful to the Spanish Impact Evaluation Fund for the support of the workshop and of six of the evaluations presented.
A Reminder that Reducing Poverty is Often the Key Objective
Most of the research presented at the workshop focused on the effectiveness of CCTs in achieving one of their twin goals, namely to increase human capital. But as Jere Berhman, Fabio Veras, Norbert Schady and Margaret Grosh pointed out, most CCT programs, especially in LAC, were driven by redistribution as their primary objective. Weak final results on learning or health weaken the belief in 'killing two birds with one stone', but for reasons of equity governments may still want redistributive programs than provide transfers to poor families, even if final human capital outcomes aren't much improved.
In this short run poverty reduction agenda, largish transfers and transfers that increase with family size would be more effective, and their targeting to the poorest important. The workshop was quite short of evidence on this. Few authors presented distributional analysis of their programs. None of the studies brought new evidence to the table on adult work effort. There was new evidence of delays in marriage or fertility for female adolescent beneficiaries for Malawi and Pakistan, but not for total family size among adult women/mothers in the program. Thus the politically powerful fears of labor disincentives and increases in fertility that greatly affect policy design remain, with little new evidence to put them to test or to rest.
CCTs are hoped to reduce poverty in the long run because the children who benefit from CCTs are expected to get more human capital and thus to earn more as adults and escape the intergenerational transmission of poverty. On that front, the evidence to date is thin. Evidence from Brazil shows that Bolsa Familia was successful in delaying the age of entry into the labor market by 1.5 years for boys and in increasing school progression for girls. The long term impact of Oportunidades was shown to be increasing linearly with the duration of exposure to the program, with on average 0.2 additional grades per year into the program. Yet, the long lasting gains on completed schooling do not seem to translate into improved labor market and labor income outcomes: graduates from Oportunidades have not shown improved labor market outcomes at first entry (see here and here). It is possible that it takes time for those gains to materialize. It is also possible that the pathway to long term poverty alleviation might come from gains in health, nutrition or non-cognitive skills. More long term follow-up is needed on these important outcomes.
Chile Solidario, a program that works with socially excluded adults, is one of the second generation programs which combines the short term objective of social assistance with the promotional objective of increased skills and endowments for the current poor. The program was successful in activating the demand for services and the labor supply for those individuals previously detached from the social welfare system, with larger results for women. The Chilean example shows that good employment and earnings depend not just on the individual, but also on active labor market policies tailored to needs of the target population as well as the surrounding labor market.
More hopeful evidence suggests other pathways to longer term poverty reduction. Paul Gertler showed evidence from Mexico that social assistance promotes productive investment through households' investment in durables and investment in micro enterprises. He and co-authors estimated that households save and invest a quarter of their transfers, thereby raising their autonomous income by 22 percent after five years. He also suggests that the investments in housing and durables may open a pathway to better health that hasn't been explored yet in the CCT literature.
How Well do CCTs Work in Lower Income Settings?
The results so far are mixed and from young and relatively small (Cambodia, Kenya, Indonesia), pilot (Nigeria, Tanzania) or experimental programs (Burkina Faso, Malawi, Pakistan). Results are impressive in Cambodia and Pakistan. Kenya and Indonesia are the first countries where no increase in primary enrollment is observed, and there wasn't much in Nigeria at secondary levels either, where the constraints on the supply side are binding. In Indonesia, primary enrolment rates are already high, payments low and not timed to when school fees are due. In Kenya, there weren't results in primary, but there were in secondary. It seems to us that the lessons from these cases aren't so much about whether CCTs can work in low income settings as they are about truisms in development: that careful diagnostics should precede policy design; that imperfections in policy design or in implementation can preclude impact; that it doesn't make sense to evaluate a program until it is up and running well enough to be given a fair trial. Happily, several of these "plain vanilla" evaluations seem to feeding into policy design and implementations in their respective countries.
New Frontiers
As though there weren't enough on the plate for evaluators and policy makers sorting out the 'old' issues during day one, the discussions of day two wetted our appetite for more. Some highlights:
- Doing proper cost-effectiveness analysis of CCTs programs is left aside in most evaluations. Jere Berhman led us through some of the difficulties, and how to avoid facile and misleading comparisons, especially for programs like CCTs that have multiple objectives or effects. He made important recommendations (moving towards internal rates of returns, relate benefits to resource costs) that can make the analysis more informative for policy.
- Behavioral perspectives: Larry Aber made a compelling case for broadening the scope of inquiry and establishing multidisciplinary collaborations with psychologists, sociologists and experts in communication science. Our understanding of the black box needs to be grounded in psychological theories. Self-efficacy beliefs, tradeoffs between intrinsic and extrinsic motivation and child engagement are going to be crucial pathways behind the short and long term gains from CCTs.
- The critical role of the supply side: CCTs are demand side interventions, and evaluations usually take the supply side as given, or background context. Veronica Silva highlighted the need to set a new research agenda on measurement of access and quality. We need to better understand and document how the supply side responds to the induced new demand for services, as well the set of motivations and incentives and constraints that drive services providers.
- Governance and political economy affect CCT effectiveness, especially in large scale programs where implementation is done at a very deconcentrated and decentralized level. Claudio Ferraz highlighted that politicians are important actors alongside beneficiary households and service providers: they have electoral incentives to increase program caseload, as well as to minimize the cost of getting rid of beneficiaries that do not comply with the conditionalities. He also warned about the risk of political gridlock that may arise from large and established redistributory programs.
- CCTs in the broad context of social policy and social protection: Helena Ribe, Antonio Claret and Agustin Escobar reminded us pioneering CCT programs revolutionized the delivery of social assistance making it more systematic, better targeted, more accountable; Now countries with mature programs are moving on in their social protection agendas, especially considering layering on services, or making linkages among them. However, evaluations of Chile Solidario and Guanajuato (in Mexico) state's Contigo Vamos Por Mas program showed how hard that can be – both in terms of their implementation as well as their evaluation. The research community should expand evaluation methods that evaluate comprehensive strategies and systems beyond the focus on single programs.
We are grateful to the Spanish Impact Evaluation Fund for the support of the workshop and of six of the evaluations presented.
RSS