Summary

The conference was organized by the World Bank, in partnership with Eurostat, the PARIS21 consortium, and the United Nations Development Program, to bring together senior practitioners from national and international agencies who share a common concern for improving the quality of development statistics. The purpose was to review recent progress made by a Task Team of PARIS21, discuss new initiatives in measuring and monitoring development outcomes, and identify steps leading to a plan of action that addresses needed improvements in national and international data systems.

The conference was held exactly one year after the June 2002 Roundtable on “Measuring, Monitoring and Managing for Development Results.” Over 100 participants from UN agencies, statistical offices, and donor agencies attended. The conference provided a valuable forum to discuss elements of a new strategy to improve the quality, consistency and availability of key development indicators. 

Details of the conference, including the agenda, a list of speakers and participants, presentations and background papers are available at data website under conference.

The Need to Improve Statistics for Measuring Development Results 

Shaida Badiee, Director, DECDG, told participants at the opening of the conference that the time is right to establish a global work program to improve statistics. Joanne Salop, Vice President of the Bank’s Global Monitoring Secretariat, spoke of the need for a clearly prioritized, costed and time-bound plan of action, so that users can have a better idea of the challenge ahead.

Ian Kinniburgh, Director of the Development Policy and Planning Office at the UN, stressed the challenging opportunities for statisticians presented by the need to monitor progress toward the Millennium Development Goals and the move towards results-based management. He announced a major review of progress towards the MDGs in 2005.

Roger Edmunds, Acting Chief Statistician at the UK Department for International Development, demonstrated how improved statistics are necessary for countries to monitor their own development, for the global community to measure progress towards the MDGs, and for results-based management of development. A clear understanding of the problems, and an agreed plan of action, would be important conference outcomes. Jürgen Heimann, Eurostat’s head of Statistical Cooperation for Africa, the Caribbean and the Pacific, explained how the European Commission is able to support countries through technical assistance and training, and through “twinning” arrangements with EU Member States.

Speakers warned that a “business as usual” approach will not deliver the better quality statistics needed to monitor development outcomes - the need is pressing.

Nick Stern, Senior Vice President and Chief Economist, in an address to the conference on the second day, stressed that this is the time to push forward to improve statistics to inform policy making in developing countries and to strengthen the international statistical system. He reflected on his own experience in collecting data at the village level in India and on the importance of good data to researchers and policymakers. He said that improvements in development statistics must be built on national efforts, but there is an important supporting role for the international agencies. This is the moment when statisticians need to set their own targets for improving the quality and availability of statistics, and then implement the programs needed to reach them.  

The Demand for Statistics

Speakers from international agencies and developing countries reviewed the way that the demand for statistical data has changed as a result of national and international initiatives, and highlighted existing gaps and problems.

Diana Alarcon of the United Nations Development Program presented progress on the country MDG reporting process, with 30 reports completed and a further 30 in the pipeline. Several issues have arisen as a result of this work, including the need for central harmonization of indicator estimates at national level, the need for resources for statistical capacity building, the need for improved analytical capacity among data users, better coordination in the selection of PRSP indicators, and joint assessments of data gaps.

Ellen Goldstein of the World Bank discussed how results measurement is being used within the International Development Association to focus attention on the effectiveness of IDA support, by monitoring aggregate outcomes through a core set of key development indicators. These have been aligned as far as possible with PRSPs, the MDGs, and initiatives of other donors. Borrowers have generally welcomed the proposals, but have asked for better consistency with reporting requirements by the international community, and support for statistical capacity building.

The United Nations Fund for Population Activities have studied the demand for population based data in PRSPs. Richard Leete of UNFPA mentioned several problems: measurability issues related to the use of Maternal Mortality Ratios for monitoring maternal health; the lack of focus of many indicators on the poor; the need for greater indicator disaggregation, for example, by wealth group, sex and geography; and the need to use statistics better to set realistic national targets. Agencies need to work more with national statistical institutes, pool resources, and avoid the temptation to “dart in and out” to get data they require for global monitoring purposes.

Three representatives from national statistical offices opened the discussion. San Sy Than of the National Institute of Statistics in Cambodia presented the increasing demands on the statistical system as a result of the PRSP and other processes. José Carvajal, Head of Bolivia’s National Statistics Institute, emphasized the need for coordinated donor support for the development of sustainable statistical systems, and for high-level political support for the process. Oleg Cara, Deputy Director of the Department for Statistics and Sociology in Moldova, discussed the role of centralized standards and classifications, the importance of the General Data Dissemination System and the Special Data Dissemination Standard of the IMF, and the need for a Statistical Master Plan.

The conference agreed that there is a need to pool resources and balance national and international data requirements, in order to respond to the high demand and coordinate efforts. The tension between international demands and national needs and ownership should be resolved; short term needs for data must be carefully balanced with longer term systematic improvements in capacity to avoid placing excessive demands on national systems. The implications of international demands on national statistical systems need to be more clearly articulated, including the need for increased resources. Coordination within statistical systems at national level needs more attention - both to better meet international data demands, and to improve effectiveness of statistical capacity building.

Mapping the International Statistical System 

Simon Hunt of Oxford Policy Management presented initial findings of a study commissioned by the PARIS21 Task Team to review the process of compiling the indicators produced by the international statistical system (PARIS21 is an international consortium that promotes a culture of evidence-based policymaking and monitoring in all countries and especially in developing countries.)The study highlighted five key areas where improvements are possible and will result in better and more coherent statistics: 

1. More systematic useof survey data, for example to supplement administrative data, and to ensure that international databases fully capture all available surveys;

2. Review of the instruments and processes through which agencies collect data from countries;

3. Population estimates, particularly to improve the coherence between estimates used by countries and those used by international agencies;

4. Definitions and indicator construction methodologies, which need to be clear and consistent;

5. Improved and more widely available metadata, particularly on websites.

Robert Johnston of the United Nations Statistics Division agreed with the illustrations of statistical problems presented in the report and with the main issues raised; there is a long-recognized need to improve the international statistical processes in a number of areas, such as harmonization and documentation of sources and methods, but the system in general is not sufficiently well resourced to deal with all problem areas. The inter-agency relationships are complex, and although there are effective UN mechanisms for coordination and improvement at the international level, coordination in multilateral and bilateral technical cooperation is not effective. The current levels of interest in monitoring the MDGs present an important opportunity, perhaps for greater involvement of the UN Statistical Commission in technical assistance issues. He identified a number of key issues, including the need for clearer standards and recommendations on the compilation of poverty statistics; the need for better coordination of household survey programs; and problems in estimating environment indicators.

Eric Swanson of the World Bank highlighted the issue of data quality, and proposed a framework for data “accountability” which would lead to improved statistics. An agency would be designated for each indicator, and would be responsible for standards and data collection. It would manage the relationship with data providers, and make candid and transparent assessments of quality and adherence to standards. Suppliers would be held accountable for the data provided. The agency would monitor discrepancies, provide advice to users, and mobilize assistance to help suppliers overcome technical difficulties. Agencies would take responsibility for active collaboration and for timely resources.

Conference participants were generally in agreement with the findings of the studies and the comments of panelists, but felt that the study should be completed to produce a thorough set of documentation and evidence of the problems. The meeting agreed that a key issue is to clarify and improve data accountability, with clearly defined responsibilities and supplier agreements, quality standards and audit trails. Resources at the level of one to two person years per indicator would be required for this. The conference also agreed that the internationally sponsored survey program should be better coordinated, and balanced with country priorities. Goals include better accessibility and use of a wider range of surveys than at present, for example through shared household survey databanks and inventories.  

Assessment of Country Capacity

Ludovico Carraro of Oxford Policy Management presented the second part of the Task Team study, which was based on desk studies of the capacity of two countries (Malawi and Pakistan) to produce estimates of MDG and PRSP indicators. It was agreed that follow-up field visits will be required to complete this work, but initial conclusions focused on the reliance on household survey programs, and weaknesses of estimates based on administrative data sources.

In a presentation on national capacities to monitor health related MDGs, Chris Murray of the World Health Organization also noted the importance of household surveys, but also stressed the role of registration systems – even where they are incomplete. He outlined a system of data accountability for key indicators developed by WHO in collaboration with data suppliers.

Charles Machinjili, Malawi’s Commissioner for Statistics, Mushtaq Malik, Economic Minister of the Embassy of Pakistan in Washington DC, and Zia Abbasi, IMF Statistical Adviser in Cambodia, responded to the report and gave an overview of statistical capacity in their respective countries. The common themes were the need for a “holistic” rather than a sectoral approach,  better coordination of donor activities at the country level, and involvement of all stakeholders at an early stage in the preparation of Statistical Master Plans. The responsibility on national governments to provide sufficient resources for the sustainable functioning of the national statistical system was also stressed. Ellen Starbird of USAID mentioned that USAID sponsorship of Demographic and Health Surveys, which provide estimates for a large number of MDG indicators, will continue. Improvements in the use of data and in capacity to execute surveys are constantly sought. One country has implemented a continuous demographic survey, a new approach to ensuring a permanent household survey capability.

Lucie LaLiberté on the IMF presented the PARIS21 indicators of statistical capacity, which give an opportunity to rationalize the process of statistical capacity building, and to set a basis for the assessment process during the next phase of the work of the PARIS 21 Task Team . The indicators, the first systematic attempt to develop a monitoring system that can be applied internationally, comprise 16 quantitative indicators and 18 qualitative indicators, the latter based on the Data Quality Assessment Framework methodology introduced by the IMF.

Agreed practical solutions include pursuing the joint assessment of data gaps and statistical capacity to develop coordinated, pragmatic master plans, focusing on the needs for development statistics. Various potential components of these plans were suggested, including the better utilization of existing data sets, focusing the efforts of statistical suppliers and strengthening the role of NSOs (through, for example, the  creation of  central statistical database), and implementing continuous or annual household survey programs. Statistical capacity building programs need to include users - bringing them closer to producers, helping them develop better skills to use and analyze statistical data., and creating a culture that values high quality data.

New Initiatives

Sessions on the second day reviewed a number of important new sectoral initiatives aimed at getting better statistics and making better use of existing data sets.

Health Metrics Network (presented by Chris Murray, WHO)

The Health Metrics Network is a proposed partnership of organizations interested in strengthening national health information systems, focusing on a core set of health indicators and the health MDGs and linked to global funds. The work program consists of developing improved norms, standards and tools to monitor the core indicators; developing a globally recognized minimum information system required to produce them; and technical and fund raising support to develop the information systems required.

The potential of this initiative to improve health information and increase donor coordination and harmonization in the health sector was recognized and welcomed. Jill Mathis of USAID and Carol Medlin of the Institute for Global Health discussed the measurement of health issues, and stressed the importance of ensuring sustainable outcomes, which will require an emphasis on building long-term capacity of national governments and ensuring widespread national ownership and support. Improving the use of information, particularly by national governments, will require the engagement of users as well as producers of information. The development of minimum standards for statistics and statistical systems needs to be sensitive to the varying stages of development of statistical systems, and health systems more generally, and realistic about what can be expected from the least developed countries.

The conference suggested considering how similar models can be applied in other sectors, or how the network might be widened to include other areas of statistics. The important and potentially difficult role of the governing board was also recognized, and the suggestion of a small autonomous panel of advisers may help to provide the board with independent expert support.

Education Statistics (presented by Doug Lynd, UNESCO Institute of Statistics)

Two new initiatives, the Literacy Assessment and Monitoring Programme (LAMP) and the Survey of Primary Schools (SPS), are aimed at directly improving the provision of education statistics. LAMP aims at improving access to accurate, timely and comparable estimates of literacy for monitoring progress towards the MDG and Education For All targets. The current proposal is for “stand-alone” literacy surveys, but possibilities of attaching a module to existing household surveys are also being explored. The SPS aims to obtain information from school administrators and teachers on teaching and learning processes to satisfy data needs related to equity and education quality. Participation in the survey programme is voluntary for countries already part of the World Education Indicators program, and the survey will be fielded in 2004.

UIS will also work with countries at different levels of statistical development to help them move further towards self sustaining systems producing better data. A key tool will be the Data Quality Assessment Framework for Education, to help countries evaluate their statistical systems and processes, identify data gaps and plan technical assistance support.

UIS has undertaken a review of the methodology and data required to estimate primary completion rates. The current focus is on obtaining information from countries on graduates from the last year of primary school, and on assessing the quality of rates generated from this information. UIS is also reviewing ways of improving data on teachers.

Barbara Bruns of the World Bank acknowledged the challenge facing UIS. The Education For All and Fast Track Initiatives have gone further in focusing attention on the need to link inputs to outcomes, and highlighting the need to track fiscal and human resource inputs and donor efforts.  A “3-P” model might be appropriate for UIS: Priority to data work and new initiatives; Partnerships with agencies that can complement the work of UIS; Pragmatism by meeting data needs with adequate rather than perfect information. UIS needs to play an important role in capacity building, and to provide a forum for harmonization among donors regarding indicators and capacity building resources.

Audrey-Marie Schuh Moore and David Guemes of the Academy for Educational Development discussed the problems of working with inadequate data that are published too late to inform policy, and the lack of harmonization among different indicator sets used by different agencies/donors.  AED and USAID are collaborating on improving data and their use, and countries will be encouraged to outsource advanced data analysis. In an effort to fill data gaps, USAID is collaborating with DHS to introduce an education module in their surveys. There are also plans to create a forum to engage all donors to refine their methods of data collection.

Taking a Coordinated Approach to Household Surveys (presented by Kinnon Scot and Haeduck Lee, World Bank)

Two initiatives were presented, the Living Standards Measurement Study (LSMS) and the Regional Program of Technical Assistance for Improving the Surveys of Living Conditions
Project in Latin America and the Caribbean (MECOVI – “Mejoramiento de la Encuesta de Condiciones de Vida”.)

LSMS surveys have become an important source of information for PRSPs and the MDGs. The program is dynamic, in order to address new data needs and improve the existing program. An important new project is the Comparative Living Standards Project (CLSP), which will extend current activities to facilitate and increases data use by documenting and archiving the 60+ LSMS survey databases in electronic format and hardcopy, improving data access policies and agreements, and providing data and documentation to researchers.

MECOVI can serve as a model for other regions wanting to improve the capacity to conduct household surveys. The objectives of the program include improving quality, relevance and timeliness of household survey and poverty data, promoting use of data for research and policy analysis and open accessibility to household survey and poverty data, and improving survey methodologies and poverty analysis. The resulting regional data bank includes micro-data sets from over 200 household surveys.

Key lessons from both initiatives include the importance of recognizing the central role of central statistical offices; the emphasis ion capacity building rather than data collection; taking a phased approach, building on existing systems; promoting a culture of the use of statistical data; the importance of “horizontal” cooperation at the country level; and the focus on delivery, and accessibility to data.

Indicator Aggregation Methods (presented by Sylvester Young and Jeff Johnson, ILO, and Tim Holt, Southampton University)

The ILO is collecting information from agencies responsible for MDG indicators, to review methods used by each to estimate missing data, regional and global levels, and trends. Key areas of focus include data availability and patterns of “missingness”; methods of imputation; methods of aggregation, and the difference between aggregation to represent a region or the countries in that region; the estimation of change and trends, for groups and countries and for countries within groups; and the documentation of existing practices. The objective is to draw lessons to improve what is done both at international and national levels, and report to the next Committee for Coordination of Statistical Activities in September.

Sylvester Young and Jeff Johnson also discussed efforts to collect statistics on employment and work. To improve global and regional estimates, some work is being done to regard the problem as a missing value problem with small samples.

Infrastructure Data Initiative (presented by Antonio Estache, World Bank)

Renewed emphasis on infrastructure, detailed in the World Bank’s Infrastructure Action Plan, has resulted in a new need for better data to underpin sector management and global monitoring tools. The strategy is to make a full assessment of existing data sources, availability, reliability, and policy relevance in each sector (energy, transportation, water and sanitation etc.), and identify gaps and strategies for filling those gaps. A discussion will be launched to identify who needs to do what, how, how often, and what resources are needed. An important element will be statistical capacity in client countries.

Conclusions and Next Steps

Three teams discussed conclusions and next steps over lunch, and reported back to the afternoon plenary session. Gilles Hervio and Martin Dyble of the European Commission reported on next steps for the country case studies of the PARIS21 Task Team, which should concentrate on assessing the current data available to monitor the MDGs, further actions and associated costs required to produce the information, conflicts between PRS and MDG data needs, the effectiveness of donor support and the efficiency and impact of donor and international agency data demands and collection. Antoine Simonpietri of PARIS21 reported on actions to improve the international coordination of household surveys. Important proposals include maximizing the use and accessibility to existing data through an international household survey databank, the development of core household survey programs, improved documentation of best practice, and the development of a process to assess household survey data quality. Roger Edmunds of DFID reported on activities to improve the accountability for indicator estimates. Although agencies need to discharge their mandate through existing mechanisms, the PARIS21 Task Team work is an important opportunity. The international study of the Task Team should concentrate on providing more comprehensive evidence and analysis to support the key findings of the initial work, obtaining information from agencies on plans for improvement and their resource needs, to respond to the growing demand for higher quality international statistical databases, and exploring current accountability and data quality assessment systems. The resulting assessment of the problem needs to be presented to Chief Executives of the various agencies, so that they are aware of the potential resource implications.

 At its conclusion, the conference recognized the need for action and change within the international statistical community and in national statistical systems, and that the international community needs to gather behind a global work program for statistical development. Agreed proposals included better documentation of the methods used to produce key indicators, and identifying levels of accountability of agencies and countries, and the promotion of national indicator databases. The conference called for a pragmatic approach to improvement in national statistics based upon realistic assessments of national capacity. Better use should be made of existing household surveys to provide key monitoring indicators, and there should be clear documentation and accountability within the international system to ensure that a complete and harmonized set of monitoring indicators is available for monitoring the MDGs. 

The PARIS21 task team on improving statistics for monitoring development goals will continue its work, with the aim of presenting a costed “roadmap” spelling out the various alternatives for making sustained improvements in both the international and national statistical systems to a proposed Roundtable meeting on Results-Based Management later in the year. The original terms of reference for the studies commissioned by the team will be reviewed to reflect conference discussions, so that they provide a systematic analysis of current data availability and quality, the constraints facing national and international data systems, and the actions and resources required to make improvements