1. Summary The World Bank’s Justice Reform Practice Group recently launched the soon to be published Justice Sector Assessment Handbook. The handbook is the product of extensive, collaborative research and analytic work by Dory Reiling, Judge and Senior Judicial Reform expert in the World Bank’s Justice Reform Practice Group; Linn Hammergren, Senior Public Sector Management Specialist at the World Bank; and Adrian Di Giovanni, also with the World Bank’s Justice Reform Practice Group. It is a practical guide for carrying out justice sector assessments, meant primarily for Bank team leaders but also for development practitioners in a wider sense. The handbook is the first description of practices in justice sector assessments available. It provides lessons learned from past experience, an overview of good practices from the Bank and other institutions, and a step-by-step approach to organizing, conducting, writing up, and disseminating these diagnostic studies. Acknowledging that there is no one-size-fits-all model for a good assessment, the authors present a series of alternatives in terms of scope of coverage, data collection methodologies, and issues addressed. The emphasis is on providing an assessment that will inform reform programs and thus on starting by defining the problems to be resolved. The authors were joined by fellow experts in the Justice Reform field, who gathered at the launch event to discuss and celebrate the handbook’s completion. The event’s aim was not so much to produce comments for a revision of the handbook, but rather to highlight and to discuss what it offers that is new and noteworthy, and to pull together some thoughts and insights of colleagues in the field on possible lines of research and issues that merit further study to feed into a later edition of the handbook. There was a brief presentation by the authors, who discussed the making of the handbook and highlighted some elements of the handbook’s approach focusing on user problems, engaging stakeholders, using multiple methodologies, and creating a baseline for development. The launch was also attended by discussants who were invited to contribute their views on different aspects of justice assessments and of justice reform. The guest speakers who contributed their views were Melissa Thomas, Associate Professor of International Development, Johns Hopkins University (SAIS); Olga Ruda, American Bar Association, Rule of Law Initiative; and Randi Ryterman, Sector Manager, PRMPS at the World Bank. The speakers’ presentations were followed by a general round table discussion by attendees, who included both employees from the World Bank and members and employees of other organizations engaged in the justice reform field.
2. Welcome note
Guests were welcomed by Christina Biebesheimer, Senior Justice Reform Expert and Head of the World Bank’s Justice Reform Practice Group. Christina highlighted the urgent need for the handbook and pointed out that one common explanation for the sluggish advances in reforming justice sectors is that we do not have enough of a knowledge base and systems from which we can learn about how to conduct justice assessments. Justice reform projects have become too much alike with very little focus on the specifics of each country’s justice sector. Diagnostics should be the key focus of all justice reform efforts and this handbook seeks to improve our ability to complete such diagnostics. Other highlights of the handbook include: • the definition of “Justice Sector”, which is revolutionary in that it changes the way in which we think about assessments by broadening the scope of the justice sector;
• the “problem centered” approach of the handbook;
• the methodology of data collection; and
• the recognition that short-medium to long-term approach to reform is good.
3. Presentation
3.1 Author: Dory Reiling Dory Reiling gave a brief overview of the contents of the handbook explaining each chapter’s function and purpose. Using a very insightful example of judicial reform that was attempted in Sri Lanka, Dory introduced the handbook’s “approach to assessments”, discussed in chapter one. In the example, the message sent is that channeling funds into reform without an adequate strategy would not yield appropriate reform. Without the knowledge about how and why the target justice institution functions inefficiently, any proposed course of action would be not be a test against reality. This chapter talks of the importance of using an assessment as a knowledge base for developing a strategy, and the importance of utilizing diagnostics before attempting any justice reform. Dory pointed out that this is one of the problems with older assessments; recommendations do not flow from accurate diagnostics. Dory also presented chapter 5 of the handbook, which gives an overview of what in the authors’ opinion are the most common complaints of the users of justice systems. These are delay, lack of access to justice, and corruption. In this chapter the authors talk about why each of these issues is important, how each is defined, and other problems relating to the conceptualization of each problem. For instance; what do we mean by corruption? The chapter also guides users on how to determine whether the problem exists, the underlying causes of the problem, as well as examples of remedies that have been found to be useful in dealing with the problem. 3.2 Author: Adrian Di Giovanni Adrian Di Giovanni introduced the chapter on organizing and planning of justice assessments. A very practical chapter, here users are walked through the steps of how to structure their assessments. This chapter’s focus is on planning, organizing and sequencing the required work. For new users, this chapter is vital in that it provides a framework around which users can formulate a plan. For old users, the chapter is a helpful reminder of the various components and layers of the assessment process, and it also provides one with a view of the final product. Equally important is that this step-by-step approach provides users with some predictability of the whole assessment exercise. Adrian also presented chapter 6 – on report writing. The key lesson of this chapter is -after the assessment is complete- how to get the messages of the assessment across, how to phrase recommendations, and how to set intervention priorities. The chapter presents report writing as a three dimensional task; organizational, analytical, and political. The chapter gives users basic advice on how to present the content of assessments in a way that effectively represents the findings, how to structure the main analysis, and finally, how to avoid ruffling feathers in the country’s political context. 3.3 Author: Linn Hammergren Two of the highlights of the handbook are its “problem centered” approach and its emphasis on institutional analysis, both of which are laid out in detail in the third chapter on conducting the assessment. Linn Hammergren gave a brief overview on how the problem-driven approach will assist teams in organizing their work. The chapter suggests problems be identified through an iterative process of document review, observation, and interviews with a wide variety of users. ”Problems” it was stressed, need to be defined from the perspective of the system users and thus their inputs are critical. One of the requisites for identifying a problem is thus to obtain information from many sources because different sources – judges, lawyers, governments, civil society, bar association – will each present their own variation of the problems/challenges of the justice sector. However first defined, the problem and its causes should be treated as the hypothesis which the rest of the assessment will explore. While the problem focus helps define which organizations and processes will be studied, the accompanying emphasis on institutional analysis guides the user in collecting information on their operations and impacts. This chapter guides the user in determining how the structures, mandates, real and actual practices, and informal rules of an institution shape outputs, and/or contribute to the weak performance/backlog/delay in the institution. An assessment needs to diagnose and evaluate. Therefore, at the end of an assessment one must be able to identify what could be better about the system. Linn also talked about Chapter 4, which expands the discussion with a review of the various methodologies for information collection. The key message of this chapter is that the sources of data or information are going to vary, and there is no one correct way of gathering information. It stresses the need for multiple methods inasmuch as no single methodology is likely to serve all purposes, and for combining soft (non-quantitative, more subjective) approaches with hard (quantitative, more objective) techniques.
4. Discussant: Olga Ruda, American Bar Association Rule of Law Initiative
Olga Ruda added her support to the handbook and pointed out that the handbook will be very useful not just to World Bank teams but to other organizations as well. She pointed out that there are a number of institutions that have developed their own methods/books/practices/approaches to justice assessments, but that this handbook is the first to attempt to put together the different processes. Ruda spoke on two issues particularly important in getting the handbook utilized: • How to build stakeholder support for the handbook and its processes;
• How to use the handbook to implement follow-up improvement of justice sectors. The relevant questions relating to how one builds stakeholder support, and on which Olga presented very insightful comments are the following: Whose support should one generate: The justice sector leadership would be the first port of call, because without their ownership, reform efforts could be severely frustrated. Other relevant key actors whose support would be equally important are chief justices, chief prosecutors, bar associations, ministry of justice, potential funders, and if possible and depending on the political context, the legislative branch of the government. When should you seek support: The best time to seek stakeholder support is throughout the entire assessment process, or at the very least, when one obtains funding for the assessment. As soon as possible at the beginning of the assessment process, one should start consultations with the ministry and the leadership in the justice sector, and continue these consultations throughout the process, reporting periodically. Why should you seek stakeholder support: Stakeholder support is important because it encourages the interest of potential funders, it gives relevance and legitimacy to the process, and it ensures the greatest receptivity and willingness to build upon the findings of the end product. Legitimacy can be demonstrated by an act as small as the chief justice’s endorsement in a letter. Such small symbolic approval goes a long way in encouraging participation by other parties. How do you get stakeholder support: Olga joked that this was the million dollar question! There is no one best way in which to obtain stakeholder support, but past experience teaches us that we could do the following: • Inform key stakeholders about the assessment, providing them with information on what, why, who, how long it will take, what the end product will look like, what issues are to be addressed etc. One should go as far as one can in providing information about the assessment.
• During the data collection phase, one can generate trust and support from officials providing the information; for example, a good repertoire with judges whom one interviews can generate important influence, support and interest by other judges and officials;
• Share preliminary drafts with these persons;
• Keep stakeholders appraised of what is happening throughout the assessment process;
• Invite stakeholders as key speakers to launch events of the assessment product, and view them as participants in the reform process. This is particularly important because the stakeholders are in a position to influence the reform attitude in the country and to generate a receptiveness of the findings and recommendations. There will be a greater sense of ownership and the assessment would not be seen as imposed.
• Use the findings and recommendations of the assessment.
5. Discussant: Melissa Thomas Associate Professor of International Development, Johns Hopkins University (SAIS)
Melissa Thomas congratulated the team on the handbook and pointed out that it was very timely in view of the Law & Development course that she teaches at SAIS. She also spoke about the recurring theme in judicial reform literature of “how little we know”, and that we should be asking ourselves - how come, after 50 years of justice reform work, we still know so little. However, there are some things that we have learnt from past experiences and they are the following: • The written laws are not the same as the justice system, and may be a poor proxy for peoples’ behavior within the system;
• The important parts of the justice system are not necessarily court-centered or even state-centered; we need to also consider forums other than courts;
• There is a great deal of path-dependency and cultural-specificity in the operation of justice systems;
• The expertise of both lawyers and social scientists is needed to undertake justice reform work;
• Many of the things that we assumed about justice systems (what works and what doesn’t work) are either not supported by hard-evidence or even disproven by hard evidence. The above lessons or “knowns” seem to be discussed in article after article, but to say that we know these things does not really make a difference if we cannot find ways to put them into practice. This has been the missing gap. This new handbook closes that gap, by incorporating all the above lessons and guides us through the steps of using what we know; by:
• insisting that a justice assessment is a social science undertaking, and that it focuses on institutions and organizations and not just the laws;
• drawing attention to the extra-judicial and extra-governmental features of the justice system;
• emphasizing the need for interdisciplinary teams;
• encouraging reserving judgment while seeking hard evidence; and
• takes into account local specificity by adopting an iterative approach rather than using a checklist approach, even though checklist approach would be quicker and easier. Melissa pointed out that, looking forward to the next edition of the handbook, there are some things that may be useful and could be incorporated into the next edition. • It would be very useful to provide the user of the handbook - who has the initial task of organizing a justice sector assessment, or charged with hiring someone to undertake a justice assessment - with a sample of Terms of Reference in an annex to the handbook.
• Incorporating a few examples of longer case studies of justice sector assessments would help make some of the points of the handbook more concrete, even if these would be hypothetical.
• Although it might be too early, the handbook could also incorporate more guidance on conducting an assessment of non-court institutions, for example administrative tribunals or non-state institutions. There may not be nearly enough lessons learned to provide such guidance, but a few suggestions could also accomplish this goal.
• The absence of a central collection point for justice assessment reports has sometimes resulted in the same justice assessments being undertaken more than once, and without the benefit of the authors’ seeing what others had already done. Therefore, it would be very useful to have a central collection point for assessments undertaken. Melissa then closed her talk with a short overview of the following three empirical challenges that still remain: Measures of Quality: If the purpose of justice reform is to improve justice systems, then we need to have a clear idea for ourselves of what makes a good justice system and how we can measure this. So far, measures have been value-laden. The current measures are not accurate. For instance, the time it takes for a court case to reach judgment may not be the best way to measure efficiency or quality. Similarly, the use of “litigation numbers” is not necessarily accurate, since the question of “what are the ideal numbers” or “whose ideal numbers these are”, remains. Who gets to define what quality is? Justice systems usually balance a number of interests and we don’t know the ideal balance for every country. Comparable Measures: More work needs to be done here. The formal justice system operates differently and occupies a different place in the constellation of dispute resolution mechanisms and social mechanisms for change in different countries. For example, in Japan, the government decided to channel disputes away from the formal legal system so as to limit its size. Justice systems operate differently and do not necessarily serve the same functions. Therefore, how do compare them? What, if any, are the essential commonalities? Measures of Construct: Constructs are an abstract. Perhaps, instead of measuring institutional aspects – such as the number of cases and number of judges – we should measure the end result that we are truly interested in. But measuring abstract concepts like “justice” or “fairness” presents unique measurement issues and requires as a point of departure a well-articulated definition of what it is we seek to measure.
6. Discussant: Randi Ryterman, Sector Manager, PRMPS World
Randi made the following comments: The handbook is timely: Referring to the World Bank’s recent consultations with NGO’s and civil society groups on its Governance and Anti-Corruption strategy, she pointed out that one of the many questions asked during these consultations was: What can the Bank do better in the area of governance? The most common request and response was to have more support with legal and judicial reform. The handbook is unusual: The practical guidance provided by the handbook is innovative. The section on how one goes about defining the problem is excellent in its practicality. The book also encourages building teams and working with clients to build ownership of the end product, and provides triangulation. Some of the things mentioned in the practical steps are common sense, but are not always applied. Therefore it is very worthy to make this guidance explicit – as this handbook does. Encourages and build internal capacity: The handbook helps to answer the question: how do we organize ourselves to be able to carry out good diagnostics? Good diagnostic capacity is rare, especially in public sector governance and probably too in the justice sector. The World Bank is good at looking at a particular problem and coming close to a diagnosis, but the challenge is helping the client to prioritize interventions. A justice assessment manual cannot fix this capacity problem, but it can help our staff figure out what expertise they need and build a team that is capable of carrying out the diagnostic. For this purpose, we could increase complimentary resources; for example, we could provide more information on a website with examples of diagnostics, terms of reference, sources of funding, comparative institutional data basis, and other comparative institutional arrangements. Practical organization of the handbook is very relevant to the reality that our resources are declining. Therefore, we need think more creatively about how we’re going to build in learning in a more systematic way into operations. For instance, more TOR’s and examples of good products/diagnostics that can be shared on the website for use by staff of the Bank. Strengthen Local Capacity: More emphasis should be placed strengthen local capacity and expertise to carry out diagnostics and analytical work. Family of Diagnostics: The handbook seems to resist a checklist for understandable reasons. Although the handbook is not prescriptive in what area is covered, the book is talking about a family of diagnostics, in the sense that there are common areas for which we are being asked to provide advice. There is probably a family of diagnostic we may need to think about going forward. For instance, grouping problems where they most commonly occur – commercial, civil, administrative, or criminal law. Another suggestion going forward is for each of the common areas to have a stakeholder map. So that when a user starts on an assessment he/she has a view of the different stakeholders he/she needs to be involving and focusing on. Frontiers: The highlights of the handbook are its guidance on how to prioritize, and its methodology of seeking out the “binding constraints” in a justice system – focusing on the question: where does the system pinch? Going forward, we also need to think about the political economy and sequencing, by asking ourselves: what is feasible and desirable in which types of environments, and how to build a sustainable process of reform.
7. General Discussion and Comments
7.1 One of the scariest things for an assessor is receiving the feedback “you didn’t tell us anything new”. Part of this is due to the process of obtaining the information that is usually followed when conducting assessments, i.e., talking to key informants. Most of these key informants come from the legal communities in the capitals of the countries. While the information gathered during the process may be news to the assessors, it is not news to the informants/country. Therefore, it is more and more important to get statistics and to test these against what people are thinking. The challenge is that the statistics are not always available for various reasons. Does the handbook recommend the type of statistics that the assessor should try and obtain, and how these should be used? 7.2 One of the challenges that we face, is the impact of tenure on the implementation of assessments (especially in South America). The handbook does not raise this issue, but it does canvass all the possibilities and issues that may need to be red-flagged in relation to ownership of an assessment. 7.3 Ownership of the assessment by the country is difficult to achieve if we (the Bank) send a team of experts to conduct assessments, after which we have a “product” that we try to sell to the country. We should be doing more to empower the local capacity of the country to conduct assessments because the country’s local experts know better than us. Furthermore, assessments take time and resources and patience, and the dilemma faced is that we (the Bank) have timetables, rules and methodologies into which we expect the country to fit into as a stakeholder/partner. We don’t respond to the challenge of empowering the country to undertake the assessment. 7.4 Does the Bank anticipate conducting more than one assessment on the same country as a way to chart progress? 7.5 The Bank and governments sometimes need diagnostic information for different purposes. Therefore there may be a question of how well a Bank diagnostic is actually going to serve the people of the country and vice versa. A second problem is that both the Bank and the local judiciary/people responsible for conducting the diagnostic have different political constraints. For instance, a local diagnostic may leave out things that are “restricted” like the fact that there is torture in the criminal justice system. On the other hand a Bank diagnostic has the risk of telling people what they already know, or skipping things people know to be problems because of the Bank’s own political restrictions/ outside input, or putting local facts into whatever is accepted as being the current wisdom about what needs to be done with diagnostics. The challenge is getting the best out of these, and not substituting one for the other. |
RSS