Frequently Asked Questions
In June 2006 the World Bank publicly disclosed for the first time the numerical scores of its 2005 Country Policy and Institutional Assessment (CPIA). The results of the 2006, 2007, 2008, and 2009 exercises were disclosed in June 2007, June 2008, June 2009, and June 2010 respectively. This Q&A note has been prepared to facilitate greater understanding of the CPIA and the process and methodology used to determine the scores. The questions and answers are divided into six sections: Overview, technical details, process, 2009 scores, disclosure, and harmonization.
Q1. What is the CPIA?
The CPIA is a diagnostic tool that is intended to capture the quality of a country’s policies and institutional arrangements—i.e., its focus is on the key elements that are within the country’s control, rather than on outcomes (such as growth rates) that are influenced by elements outside the country’s control. More specifically, the CPIA measures the extent to which a country’s policy and institutional framework supports sustainable growth and poverty reduction, and consequently the effective use of development assistance. The outcome of the exercise yields both an overall score and scores for all of the sixteen criteria that compose the CPIA (see Q6 under Technical Details for a fuller explanation.) The CPIA tool was developed and first employed in the mid-1970s and over the years the World Bank has periodically updated and improved it to reflect the lessons of experience and the evolution of thinking about development.
Q2. Who makes the judgments about the quality of a country’s policies and institutional arrangements and how often does the process happen?
The ratings are the product of staff judgment and do not necessarily reflect the views of the World Bank’s Board of Executive Directors or the governments they represent. The CPIA is an annual exercise.
Q3. Why does the Bank undertake the exercise?
The answer is twofold. At the broadest level, experience has taught the development community that good policies and institutions lead, over time, to favorable growth and poverty reduction outcomes, notwithstanding possible yearly fluctuations arising from internal and external factors. In part, this conclusion derives from another conclusion of experience: development cannot be imported to a country from the outside; rather, it is largely a function of a country’s own efforts. The second reason the Bank undertakes the CPIA exercise is functional: the ratings help determine the relative sizes of the Bank’s concessional lending and grants to low-income countries (see Q4 for further details). By concessional lending, we mean lending by the World Bank Group’s International Development Association (IDA) on terms with significant grace periods, long repayments periods, and very low interest rates.
Q4. What is the significance of the CPIA—what is it used for, why does it matter?
IDA resources are allocated in per capita terms on the basis of a country’s IDA country performance rating (CPR) and, to a limited extent, per capita gross national income (GNI). Use of the CPR ensures that good performers receive, in per capita terms, a higher IDA allocation —i.e., allocations are performance based. A country’s overall score is the main element of the CPR. (A more detailed explanation is provided under Technical Details—see Q10.) To fully underscore this role, the overall CPIA country score is referred to as the IDA Resource Allocation Index (IRAI). The scores are also used internally by the Bank, among other things to help guide its interventions, its assessments of risk, and its research (see Q11 under Technical Details for further information).
B. TECHNICAL DETAILS
Q5. What criteria make up the CPIA and what do they measure?
The CPIA consists of 16 criteria grouped in four equally weighted clusters: Economic Management, Structural Policies, Policies for Social Inclusion and Equity, and Public Sector Management and Institutions (see Box below). For each of the 16 criteria, countries are rated on a scale of 1 (low) to 6 (high). The scores depend on the level of performance in a given year assessed against the criteria, rather than on changes in performance compared to the previous year. The ratings depend on actual policies and performance, rather than on promises or intentions. In some cases, measures such as the passage of specific legislation can represent an important action that deserves consideration. However, the manner in which such actions should be factored into the ratings is carefully assessed, because in the end it is the implementation of legislation that determines the extent of its impact. For further information on the criteria, click here.
A. Economic Management
1. Macroeconomic Management
2. Fiscal Policy
3. Debt Policy
B. Structural Policies
5. Financial Sector
6. Business Regulatory Environment
C. Policies for Social Inclusion/Equity
7. Gender Equality
8. Equity of Public Resource Use
9. Building Human Resources
10. Social Protection and Labor
11. Policies and Institutions for Environmental Sustainability
D. Public Sector Management and Institutions
12. Property Rights and Rule-based Governance
13. Quality of Budgetary and Financial Management
14. Efficiency of Revenue Mobilization
15. Quality of Public Administration
16. Transparency, Accountability, and Corruption in the Public Sector
Q6. How is a country rated against these criteria?
For each of the criteria, the Bank has prepared guidance to help staff assess the country’s performance, by providing a definition of each criterion and a detailed description of each rating level. Bank staff assesses the country’s actual performance on each of the criteria, and assign a rating. These scores are averaged—first to yield the cluster score, and then to determine a composite country rating as the average of the four clusters. The ratings reflect a variety of indicators, observations, and judgments based on country knowledge, originated in the Bank or elsewhere, and on relevant publicly available indicators.
Q7. Have the criteria remained the same over the years?
No. The Bank has periodically reexamined the criteria and revised them to reflect the lessons of experience and the evolution of thinking about development. Over time, the assessment criteria have shifted from a largely macroeconomic focus to include governance aspects and a broader coverage of social and structural dimensions. Some of these changes refined some of the criteria, or added new ones; others modified the process. In 1997, for example, criteria covering governance-related issues were added, and in 1998 the CPIA process was revised to add a benchmarking phase (see Q12). In 2001, several changes were introduced that included establishing a written record, providing detailed guidance for criteria with several subcomponents, broadening the set of benchmark countries, revising the content of the criteria, and defining the different rating levels (previously only the 2 and 5 rating levels were fully defined). In 2004, the Bank commissioned an external panel to review the CPIA ratings and methodology.
Q8. What were the main conclusions of the external panel review?
On request of the Bank’s Executive Board, an external panel of experts, academics and policymakers from developing and developed countries reviewed in 2004 the CPIA methodology and process. The Panel concluded that the CPIA criteria focus on the right set of issues and produce robust results. However, the Panel also found unnecessary overlap in some of the criteria, and outlined actions to address some methodological and process issues. The Panel broadly supported the practice of determining the ratings on implemented policy actions rather than intended actions and it also underscored that although available outside indicators could help guide the ratings, the Bank staff’s professional judgment of country performance against the criteria should continue to play the key role in the final assessments. The Panel strongly favored disclosure of the ratings for IDA-eligible countries and suggested some steps to be taken prior to disclosure to enhance the ratings’ robustness. The Panel report is available at this link. The criteria were revised in 2004 to take into account the recommendations made by the panel.
Q9. What were the major conclusions of the recent IEG evaluation of the CPIA?
In the fall of 2009 the Committee on Development Effectiveness (CODE) of the World Bank Board of Executive Directors discussed an evaluation of the CPIA prepared by the Independent Evaluation Group (IEG). The evaluation concluded that the content of the CPIA is largely relevant for growth and poverty reduction in the sense that its components map well with the determinants found in the literature; that the ratings are generally reliable and correlate well with similar indicators and there is no evidence of upward bias. The evaluation also found that there is room to streamline and revise some of the criteria and makes some specific suggestions such as broadening the trade criterion, revisit the financial sector criterion taking into account the lessons of the recent financial crisis; adding an assessment on disadvantaged socio economic groups, and strengthening the linkages among criteria. The IEG report is available at this link. The Bank is currently undertaking a review of the CPIA criteria that will be informed by the recommendations of the IEG evaluation. The review will be completed early next year and the 2011 CPIA exercise that will be launched in September 2011 will use the revised criteria.
Q10. How does the CPIA affect the allocation of IDA resources?
As mentioned earlier (Q4), the overall CPIA country rating, or IRAI, has a significant influence on the allocation of IDA resources, through its role in the IDA country performance rating (CPR). The CPR is obtained by calculating a weighted average of the IRAI Clusters A-C average (24 percent), IRAI Cluster D average (68 percent) and the portfolio rating in the Bank’s Annual Report on Portfolio Performance (ARPP) (8 percent ). Details of the IDA allocation process are described in the IDA15 replenishment report— Additions to IDA Resources: Fifteenth Replenishment, February 2008. particularly Annex 1. For more information, see IDA’s Performance-Based Allocation System: Simplification of the Formula and Other Outstanding Issues, September 2007.
Q11. Earlier it was mentioned that the Bank uses the CPIA findings for internal purposes—what are these purposes?
In addition to their major role in IDA’s performance-based allocations, the scores are used to inform the Bank’s Country Assistance Strategy program and country policy dialogue; to assist in the assessment by the Bank’s staff of the degree of portfolio risk; to help identify countries for extra attention on fiduciary standards and governance; to provide background to the Bank’s Independent Evaluation Group’s Country Assistance Evaluations; to help assess a given country debt sustainability, and to offer input to research on the determinants of growth and poverty reduction. More recently, the scores have been used as inputs in the Bank’s Global Monitoring Reports.
Q12. How are the ratings prepared?
The process of preparing the ratings involves two phases: (a) the benchmarking phase, in which a small, representative, sample of countries is rated in an intensive Bankwide process; and (b) a second phase, in which the remaining countries are rated using the derived benchmark ratings as guideposts. The process is managed by the Bank’s Operations Policy and Country Services Vice Presidency.
a. Benchmarking Phase. The objective of the benchmarking phase is to help ensure that, on the basis of the criteria, the ratings are set at the right level and are consistent across countries and Regions. Benchmark countries include countries from the World Bank’s six operational Regions, and are chosen to include both good and poor performers. To maintain some continuity in the sample but also refresh it, each year’s set of benchmark countries contains some countries from previous years as well as some new ones. For each of the benchmark countries, country teams submit rating proposals for each criterion, accompanied by written justifications. The proposals are initially vetted by the respective Regional Chief Economist and are then reviewed by the sector experts outside the Region—i.e., staff in the Bank’s Networks and central departments. The final ratings are determined at a two-day meeting in which representatives from the Regions, Networks, and central departments review the proposed ratings for all the criteria. The review is informed not only by the written submissions and Bankwide comments, but also by available external indicators and other relevant supporting documentation. At the conclusion of the benchmarking phase, these countries ratings are “frozen” and the second phase of the exercise begins.
b. Second Phase. In this phase, a similar process as in the benchmarking phase is followed by staff to prepare the ratings for the remaining countries. Country teams prepare ratings proposals that are accompanied by a written justification. These proposals are reviewed within the each of the Bank’s operational Regions by the respective Chief Economist and are then submitted to a Bankwide review by the Networks and central departments. Throughout this process, the benchmark country ratings are used as guideposts to help ensure consistency of the ratings of countries within a Region and across Regions.
Q13. What information does Bank staff draw on in preparing the ratings?
The country teams that prepare the rating proposals are very familiar with the country, and they draw on their own knowledge of its situation. More importantly, however, they also use relevant diagnostic studies—for example, country economic reports, a Public Expenditure Review, or a poverty assessment—that the Bank, the country itself, or other parties may have conducted. In addition, over several years the Bank has assembled sources of data from both inside and outside the Bank on which staff can draw in making their judgments. These are listed in the criteria under guideposts.
Q14. How is the objectivity of scores ensured?
The starting point of the process is the set of country team proposals that are based on their informed professional judgment. To ensure that scores are consistent across countries and Regions, the country team proposals undergo a series of check and balances in which they are reviewed first within each operational Bank Region by the respective Chief Economist, and then by sector specialists outside the Regional units and by staff from central departments (see the preceding description of the process used to obtain the final scores). A key goal of the benchmarking phase is to provide a set of robust scores to guide the second stage of the ratings process. The exercise is managed at the center to ensure a clear separation between the resource allocation function and the operations for which the allocations are used. In the context of the review by the non-Regional staff, the country team proposals are adjusted to ensure the consistency of scores across countries. In cases where differences of view persist, internal mechanisms have been put in place to reach closure. Before all scores become final, they are reviewed and vetted by the Regional Chief Economists.
Q15. Does the Bank consult with country authorities during the process?
Yes. During the CPIA process, country authorities are involved at two stages. The first stage occurs early in the process when the Bank’s country teams prepare the rating proposals and before these are submitted for Bankwide review. World Bank staff meet with country authorities to discuss progress made in addressing issues identified in the previous year’s exercise. This discussion helps identify areas in which the Bank’s assessments might differ from those of the country authorities and provides them with an opportunity to bring additional information to the attention of Bank staff. The objective of this phase is to ensure that country teams have taken into account all the relevant available information when preparing their write-ups and the associated rating proposals. The guidance provided to the teams makes clear that these interactions with country authorities are part of a process of consultation, not a negotiation over the ratings. The second stage of client involvement occurs at the end of the process when, after the completion of the Bank wide review process, country teams communicate to the authorities the results of the Bank’s assessment and the implications for the Bank’s program, and explore ways to address identified weaknesses.
D. 2009 SCORES
Q16. Did the criteria change between the 2008 and 2009 exercises?
No. The CPIA criteria used in the 2009 assessments is the same as of 2005-2008 exercises. The last revision of the criteria took place in 2004 (see Q8 under technical details for further information).
Q17. Is the country coverage the same as in 2008?
No. In addition to 75 countries included in 2008 exercise, 2009 exercise also covers Kosovo and Liberia resulting in a total of 77 countries.
Q18. When were the ratings finalized and what period do they cover?
The ratings process typically starts in the fall and it is concluded in the spring of the following year. The scores disclosed in June 2010 (the 2009 CPIA exercise) cover 2009 country performance.
Q19. Do the ratings change much from one year to another?
Given the criteria’s focus on the quality of policies and institutions, little change from one year to the next is to be expected in the ratings. Typically, at a country level changes can be more pronounced in the Economic Management cluster than in the other clusters, reflecting the fact that changing institutions (e.g., judicial systems) is a lengthier and more complex process than changing macroeconomic policies (e.g., taking steps to reduce the fiscal deficit). Between 2008 and 2009, the change of the overall score for 67 out of the 75 IDA countries covered during both years (89 percent) was between -0.1 and +0.1, with the scores of 35 countries (47 percent) unchanged relative to 2008. These results are similar to the pattern observed in recent years. It is important also to note that as with all indicators, there is a degree of imprecision in CPIA measurement of the quality of the policies and institutions. The scores can be interpreted as estimates with some margin of error. The 2005 Global Monitoring Report cites estimates of a standard error of 0.24 for the overall scores on the 1-6 scale. This means that small annual changes from one year to the next may not be necessarily significant. High-, middle-, and low-performing countries can be distinguished, but attempts to position countries on a fine scale will likely result in some of them being misclassified. Small ranking differences on a fine ranking scale should not be seen as necessarily significant.
Q20. What information has the Bank disclosed to the public?
In 2000, the Bank began disclosing the CPIA relative ratings in a quintile format for IDA-eligible countries. The Bank posted the quintile-based rating results for the CPIA, its four clusters, and the country portfolio and IDA country performance ratings on its external website. The criteria and the methodology of the performance-based allocation system were also disclosed. For 2004 quintile groups click here.
Q21. What new is being disclosed?
In September 2004, the World Bank’s Board of Executive Directors approved the disclosure of the numerical CPIA scores for IDA-eligible countries, starting with the results of the 2005 exercise. Specifically, the Bank discloses for all IDA-eligible countries, including blend countries (i.e., countries that are currently eligible for funding from IDA and IBRD): (i) the scores for the 16 criteria, the cluster averages, and the overall score (IRAI); (ii) the portfolio rating; and (iii) the overall IDA country performance rating.
F. HARMONIZATION OF PERFORMANCE ASSESSMENTS AND RATINGS
Q22. Do other institutions use the Bank’s CPIA criteria?
In 2005, both the Asian Development Bank (AsDB) and African Development Bank (AfDB) adopted the World Bank’s criteria as a starting point for their respective performance-based allocation processes. World Bank staff maintains periodic communications with their counterparts in other multilateral development banks (MDBs) to discuss the criteria and the respective performance-based allocation processes. Several workshops have been held, most recently in April 2009, where staff of the three MDBs and other interested agencies shared their experiences with their respective performance-based allocation systems.
Q23. How do the World Bank’s scores compare with AsDB and the AfDB scores?
The scores are highly correlated. For 2009, for example, the correlation coefficients between the World Bank CPIA scores and those prepared by AsDB and the AfDB are 0.8 and 0.9 respectively. Although both the AsDB and the AfDB use the World Bank’s CPIA questionnaire in its assessment of country performance, country scores may not necessarily be the same. Several factors may explain such differences. First, different cut-off dates for the information that underpins the assessments, which may result in differences in ratings. Second, portfolio scores are likely to differ between the two institutions, as they measure performance of different set of projects. Finally, the questionnaire allows room for differences in professional judgment. Although elaborate review mechanisms used by both institutions tend to reduce the room for differences, some differences are to be expected.
Q24. Why do the MDBs prepare separate country performance assessments?
Given that each multilateral institution is accountable for the use of its concessional resources, full institutional ownership of country ratings that underpin the allocation of these resources is critical. Each institution follows an allocation cycle that parallels its respective fiscal year. The integrity of the policy and of the allocation processes require that each institution carries out an independent assessment and determines its own ratings. With disclosure of the ratings, the availability of independent assessments by different institutions using the same questionnaire is a useful means of cross-checking individual country assessments.