Quality Controls

The World Bank’s main business is development. Projects must be technically, economically and financially sound and produce results. Results start with quality controls. Every investment project receives a report card on its quality compared to all other projects financed by the Bank. The projects are rated by international experts at the initial design stage and at closing. During implementation, if a project is not performing well, it is considered “at risk” of not achieving results. If it is rated unsatisfactory, it is a “problem project.” Such projects are carefully monitored. They receive extra management attention. Ratings are posted inside the Bank and reported to Government. Once per year regions are compared in a report to the Board of Directors, who represent the government-shareholders of the World Bank.

Annually, the overall portfolio of projects is assessed independently by the Quality Unit in each Region. The ratings of projects in Russia have improved considerably from a low point in 1998, when the financial crisis brought many operations to a halt.

Currently, out of 18 projects, all but one are rated as fully or moderately satisfactory while 1 has a moderately unsatisfactory rating on implementation performance. For less than fully satisfactory projects, the Bank institutes a plan with the implementing agency to improve performance. Sometimes this involves restructuring the project to help it meet changing conditions more effectively.

Since 2005, projects in Russia have been improving on several dimensions. The number of projects and commitments at risk has declined and is lower than average. Spending on project activities (disbursement ratio) has increased above average. However, a specific issue in Russia compared to the rest of Europe and Central Asia is that many projects have been extended more than one year. Long implementation reduces the efficiency of project finance. Moreover, older projects were designed at a time when quality controls were less strict on quantifying results. Newer projects have better results frameworks.

Quality controls include close supervision to provide operational advice during project implementation. Performance is monitored using quantitative indicators developed under the project. World Bank investment advisory services include results monitoring.

Regional Pilots Provide Lessons for National Implementation

The E-Learning Project is but one example of the benefit of testing new approaches before undertaking nation-wide programs. In the Tax Administration Project, two different international models were tested before the most appropriate one for Russia was adopted nation-wide, incidentally earlier than planned in the original project design. In four other operations – Regional Fiscal Reform, Education Reform, Sustainable Forestry and Customs Development – new approaches tested in selected regions are now applied throughout the Russian Federation.

A similar approach may be considered by the Government following the evaluation of the Health Reform Implementation Project, now reaching completion in the Chuvash Republic and Voronezh. It included support for policy and institutional reform at the federal level. Health system challenges in these regions have unique features that depend on population characteristics, but also problems that are common across Russia. Capital and human resources devoted to health were unequally distributed. Only 40% of the need in rural areas was met because of low quality of care, poor transportation, few medical personnel and lack of appropriate equipment.

Budget funding of health institutions was based on inputs like number of hospital beds, not on how well the system cured patients. This led to more hospital beds than were needed. Likewise, there were no incentives for health workers to improve their productivity and the quality of services. In both regions, too much money was spent on hospital services and not enough on preventing disease in primary care. The system struggled to cope with rising cases of TB, HIV, and high rates of cardiovascular diseases and injuries.

Health Reform: Improving the efficiency of resource use (day surgery)

Health Reform: Improving the efficiency of resource use (hospital stay)

Health Reform: Improving the quality of health services (re-admissions)

Health Reform: Improving the quality of health services (polls results)

Health Reform: Infant Mortality

Faced with these challenges, Chuvash Republic and Voronezh region began restructuring their health systems in 2002. The goal of the reforms is to improve efficiency, equity, effectiveness and choice. Both regions have developed strategic plans with strong ownership by regional leadership and links to the federal level, enabling mutual knowledge transfer and learning.

A recent assessment conducted by the World Bank and the regional governments (Is Health Reform Possible in the Russian Federation? Emerging Evidence from the Chuvash Republic and Voronezh Oblast, forthcoming 2008) show promising results. For example, in both regions, access to quality basic health services through general practitioners has grown impressively, particularly in underserved areas. After new training the percentage of the population with access to general practitioners has risen from 13% to 41% in the Chuvash Republic and from 3 to 28% in Voronezh Oblast. Perhaps more important, spending on primary health care, as opposed to specialty care, has risen significantly. Average life expectancy is increasing and infant mortality is declining faster than in the rest of Russia. While it is not possible to link these gains in population health outcomes to improvements in health system functions in both regions, they provide a useful indication of the positive developments experienced by the regions over the 2002-2007 period.

This project will be formally evaluated by the Government in 2009 and will provide an opportunity to glean deeper lessons for application in other regions. Already, there is demand from many regions for advice on similar health reforms.

Monitoring and Evaluation Improves Accountability and Learning

The World Bank Group uses a monitoring and evaluation system with checks and balances to quantify and validate the results of all investment projects. Since the 1980s, all completed projects are assessed by the Bank and Government for efficiency and effectiveness, and then subjected to an independent evaluation by an audit group (IEG) that reports directly to the World Bank's Board of Directors. Each independent evaluation is disclosed to the public for accountability as well as for learning. Taxpayers see the results of public investments and governments learn how to improve the design of future projects. Since 2003, all projects must have baselines and measurable indicators so that results can be quantified.

For example, Housing and Communal Services Project has led to more efficient and cost-effective housing and social services in participating cities. Housing allowances are better targeted (20.3% versus 100% of population), and municipal heating subsidies have been eliminated. Fuel consumption was reduced by 15%, water consumption declined by 10%, heat loss in the networks decreased from 195,000 to 138 thousand gal/year, and about half of the cities have achieved 100% cost recovery. Good quality water is supplied 24 hours a day in 100% of the pilots. Compliance with quality standards is 99%.

Communal Services: Drinking water meets standards

Heating: Reduction of energy usage

Heating: Lower heat losses

Heating: Targeting low-income households

There are lessons from project implementation to achieve even better results. One major lesson is that municipalities need incentives to introduce difficult reforms in a timely fashion.

The new Housing and Communal Services (HCS) Project will help the Government get better results from the US$10 billion allocated to related housing and infrastructure investments. The key goal is to accelerate implementation at the local level so people see results. The World Bank is being proposed as a project advisor to help implement the operation.

The project will test a competitive grant mechanism to cities that increase market mechanisms in housing and communal services (heating, power, water, sewage and sanitation). It will also strengthen social protection of vulnerable consumer groups. Lessons from the project would be studied and adapted by the Russian Government for use in federally-funded infrastructure projects.

Performance Management

Cadastre: Property registrations increased

Cadastre: Registration time shortens

Cadastre: Services standards reached

Municipal: Rent ratio equilized

Municipal: Budget arrears reduced

Quantifying results is an increasingly important part of public sector management in the Russian Federation. "Performance management", as it is known internationally, became a Government priority in response to poor results on achievement of several national goals. To accelerate improvements in services for citizens and businesses, the Government introduced specific targets and monitoring and evaluation into the "Concept of Administrative Reform" adopted in October 2005.

Performance Management in Federal Agencies: Strategic planning and results monitoring are embedded in projects designed in partnership with the World Bank in the Customs and Tax Administration Agencies. The projects go farther by including institutional reforms and performance pay. They are modernizing every aspect of the agency, from IT and human resources to introduction of e-government practices. This will lead to faster service delivery, but will also curtail corruption since reducing physical interaction will minimize opportunities for rent-seeking.

Key results to date in Customs Administration project are:

• Increase in efficiency by reducing the average customs clearance time by over 25% from 2003 to 2006;
• Decrease in opportunity for bribes by reducing items selected for physical inspection from 30 to 16.2% on imports.

Citizen Feedback Improves Performance: Top-down declarations rarely produce change. Citizens and civil society groups have a strong stake in reforms. Systematically engaging them and monitoring their views increases the likelihood that results will be achieved and maintained. For instance, client satisfaction surveys, complaint hotlines, and other techniques in the Cadastre and Customs Agencies are leading to faster and better services:

• In the Cadastre project, citizen feedback led to reduced paperwork requirements and bureaucratic steps, enabling title registration to decline from 2-3 months to 20 days.
• In the Customs project, complaints from businesses led to better services and clearance of over 90% of inland cases in less than 1 day, up from 63% in 2004.

Performance Management at the Municipal Level: Public officials are closest to citizens at the municipal level. In partnership with the World Bank, St. Petersburg and Kazan provide good examples of how public officials can improve public finances and economic development through strategic planning and results orientation. These changes translate into better social services and jobs for citizens.

Procurement Standards Bring Savings and Higher Quality

Approximately two-thirds of the Russian federal budget funds the procurement of goods and services. In addition to careful monitoring and evaluation, procurement under Bank-financed projects has led to considerable savings for the Russian public as well as higher quality of services.

For example, based on experience in several contracts for ICT equipment and consulting services in the Treasury Development Project, savings against market estimates ranged from 4% to 54%. Given present trends the overall savings are expected to average 20% by the end of 2009.

In addition, astute monitoring was critical to ensure that quality of services remained at international standards. One team provided by a winning bidder was not as experienced as reported in bidding documents, leading to a slow-down in the project. The Bank intervened to ensure that the quality of service met high standards. There is now stronger performance and implementation has improved.

Another benefit to the Government from World Bank procurement methods is that various bidders are encouraged to participate. This provides experience to Russian companies to compete internationally. Based on an analysis of IT procurement, Russian firms are gradually improving in this regard.

Physical Audits Prevent Corruption

Unfortunately, projects are vulnerable to corruption in most countries. While rare, it is an important reminder of the importance of procurement monitoring. All World Bank staff receives additional training to detect and avoid corruption. Anti-corruption guidelines must be applied in all projects.

Health supplies are especially vulnerable to being diverted from patients who need them. Hence, anti-corruption practices in the TB-AIDs project in Russia provide a good example of the World Bank approach. First, it is important to recall that the TB mortality rate in Russia was the highest in Europe and 20 times that of Spain. More than 275,000 people died of the disease in 1995–2004. Containing TB in Russia was a massive undertaking. It required the delivery of drugs, equipment, and training on internationally-accepted treatment approaches across the Russian Federation.

In partnership with the Federal Ministry of Health, the World Bank financed and is supervising a $150 million project that compliments the national effort to contain TB. To ensure the delivery and quality of health supplies:

• A special project implementation unit managed procurement and payments for 380 million dosages of drugs and 35,335 sets of equipment for approximately 3000 healthcare institutions at the federal to municipal level in 85 regions.
• The final 10% payment was only made to suppliers after originals of acceptance reports were received from end users. Regions submitted inventory reports for delivered equipment and drugs annually. Data was verified in annual audits.
• Treatment effectiveness was verified during independent monitoring visits to 60 regions.

As of 2008, the number of new TB cases in Russia has leveled off and the mortality rate has declined. While a significant effort remains, this is a major success for public health in such a large country. Systematic project management and surveillance were key factors.

Catalytic Effect of Project Finance

Small investments when well managed stimulate improvements in other public investments. Past financing provided by the World Bank to Russia is very modest in comparison with federal programs. A special Joint Group analyzed 22 projects either under implementation by the Russian Government or closed in the period from FY03-07, with total financing of approximately US$2 billion. Annual loan disbursements amounted to 0.87% of consolidated budget funding for fixed investment in CY2006. During this period, the World Bank provided technical advice on preparation of 12 projects, supervised implementation of 20 projects, and conducted economic and sector research, often in conjunction with Russian experts.

As a result, even this modest level of Bank financing catalyzed larger improvements in four areas of Government priority:

• Sustaining rapid growth;
• Improving public sector management and performance;
• Improving the delivery of social, environmental and communal services; and
• Ehancing Russia’s global role.