Ontanu, Elena Alina and Velicogna, Marco and Contini, Francesco, How Many Cases? Assessing the Comparability of EU Judicial Datasets (June 17, 2017). Presented at the Conference Ius Dicere in a Globalized World XXIV Bi-Annual Colloquium of the Italian Association of Comparative Law (AIDC), Naples, 15-17 June 2017. Available at SSRN: https://ssrn.com/abstract=2990558

“Efficiency is often considered a key component of any effective justice system, and a crucial drive for economic growth. A growing body of comparative studies explores how judicial reforms leading to a greater efficiency or effectiveness are positively correlated with economic growth (e.g. Global Competitiveness Report of the World Economic Forum, Doing Business Report of the World Bank, Judicial Reforms in Europe Report of the ENCJ, The Economics of Civil Justice of the OECD). At EU level, the European Commission has launched tools like the EU Justice Scoreboard to help the Member States to improve the effectiveness of their justice systems. This instrument, in particular, has been created to help EU Member Stares upholding more effective justice and, in particular, to measure and compare the efficiency of EU justice systems. The belief is that more effective and efficient justice systems will drive stronger economic growth, since “effective justice systems are a prerequisite for an investment and business friendly environment” (EU Justice Scoreboard 2016, p. 1). Efficiency and effectiveness are just two, out of several, basic features of justice systems. An efficient (or effective) justice system could potentially suffer from a lack of an independent judiciary and/or miss fairness of procedures and quality of judicial service. This paper, though, does not want to challenge the efficiency approach on these grounds. The researchers aim to check to what extent the data on efficiency used in academic and political discourses that are provided by the Scoreboard is sound enough to make empirically grounded statements in a valid comparative format among the Member States. In a simplified (but not simplistic) way, efficiency can be defined as the ratio between inputs (resources) and outputs (decisions) of the system. While formally aiming to measure and compare efficiency of Member States’ justice systems, the EU-Justice Scoreboard does not link inputs and outputs indicators to for this purpose. Furthermore, we argue that any attempt to make cross-country comparisons is affected by the comparability of the data sets used for the purpose.  Another scholar presenting his proposal at this conference, Marco Fabri, explores the question of the comparability of human resources data (judges) that in labour intensive organisation like courts can be considered as the key production factor (“Too few judges” paper). This paper explores a different area, complementing Fabri’s work. The researchers choose to explore the case-flow indicators presented by the Scoreboard which bases its analysis on the number of incoming, pending and resolved cases. The number of cases a court system manages to handle in a year is often considered emblematic for its efficiency. In Europe, the Commission for the Efficiency of Justice of the Council of Europe is the primary collector of such data, which is published in the “CEPEJ Evaluation of European Judicial Systems Report”. The same data is also used by the EU Justice Scoreboard and by many other academic and policy documents. The analysis will deal with the comparability of these data and show that such comparability cannot be taken for granted. It will assess if the definition of “case”, of the different “case types” and of their status (incoming, pending and resolved) is consistent across the different Member States. It will check if national peculiarities make the comparison between apparently identical groups of cases unreliable or inconsistent.  Previous analysis suggests that the comparability of such data is critical in many areas, such as the consistency of the answers across time (at state level), and between countries within the same period. The paper will show how the data provided by the Member States to fill apparently simple categories of cases like small claim, and litigious or non-litigious cases vary, making a comparison at least problematic. This finding, together with similar problems associated with measuring the number of judges, suggest caution should be exercised in the use of such indicators for comparative purpose among justice systems in the academic and political debate.”