The End of Indicators

On 16 September 2021, the World Bank unceremonially buried one of its most controversial projects, the Doing Business ranking: in a tight-lipped statement, the Bank announced that “management has taken the decision to discontinue the Doing Business report”. Since 2004, Doing Business had measured and ranked the business-friendliness of member states’ legal systems. The decision to end the project came after internal investigations confirmed what critics of the project long suspected: ranking data and methodology had been manipulated to achieve politically desirable outcomes, in this case to boost China’s ranking in the 2018 report, and Saudi-Arabia’s in the 2020 edition.

As Doing Business goes down in shame and scandal, its demise not only ends 15 years of political controversy and scholarly criticism of the Bank’s most influential “knowledge product”. The end of Doing Business also marks a symbolic caesura in global governance more broadly: it concludes the era of governance by indicators; it highlights the shifting relations of expertise, politics and law in global governance; and it points to new forms of global data governance that replace indicators in the digital age of competitive multilateralism.

Gaming Over for the DB Indicators

When the Doing Business (DB) report was first published in 2005 by the Bank’s private sector arm, the International Finance Corporation, it epitomized key features of liberal global governance: authored by an autonomous expert bureaucracy, ranking member states’ performance based on “objective” evidence and expertise; not legal in nature, but an informal instrument generating reputational incentives for governments; and a system of indicators that prized deregulation, openness towards global markets, and states competing for foreign investments, e.g. by liberalizing labor markets and building permit processes.

The DB report quickly became the most cited World Bank publication and a lightning rod for controversy. The International Labor Organization and trade unions slammed it as attack on labor rights, and social scientists and comparative lawyers pointed out methodological flaws in the measurements. As some governments liberalized their business regulations to improve their rankings, others undertook less genuine “reforms”, adopting strategies known as “gaming the indicators”, such as tweaking business regulations only in the cities where DB measured them.

With hindsight, it seems indicators were gamed on all sides. Already in 2018, the head of the Bank’s research department, Nobel-prize winning economist Paul Romer, claimed that DB had cheated Chile, downgrading the country when a leftist administration was in power and upgrading it during a conservative presidency. Recent investigations now confirm manipulations of China’s and Saudi-Arabia’s rankings in the 2018 and 2020 reports, respectively. After members of the DB team had blown the whistle, an independent investigation by the US law firm Wilmer Hale found that Chinese officials and senior Bank management had pressured the DB project team to upgrade the country’s ranking in the 2018 report, which was to be published during the Bank’s ongoing capital increase campaign and amid fears that China would disinvest from the Bank.

When it transpired that China’s ranking would not improve, but actually worsen in the 2018 edition, senior Bank management halted the report, which had already gone out for printing, and discussed ways to boost China’s ranking with the DB team. They contemplated merging China’s data with Hongkong or Taiwan (ranked 5 and 15, respectively), or changing the methodology for all countries in a way that would be favourable to China. In the end, the DB team revised their original assessments of Chinese reform legislation so that individual sub-indicators improved, such as the one measuring “Legal Rights”. China thus ended up with the same rank than the year before – 78 out of 190. A similar story occurred with Saudi-Arabia, which was praised as one of the “top reformers” in the 2020 report, only to see this corrected in 2021 due to data irregularities.

The remarkably frank investigative report, based on 80.000 files and interviews with dozens former and current staff members, offers vivid insights into the politicization of global indicators and is so full of damaging detail that the Bank saw no other option than to end the gaming of the indicators altogether.

The Nail in the Coffin of Governance by Indicators

The demise of DB has not only done lasting damage to the credibility of the “knowledge bank” as a provider of development data and expertise but is also of wider significance for global governance and international law. Among the array of indicators that proliferated since the 2000s, DB was the prime example of what international lawyers termed governance by indicators – a new “technology of global governance” that amounted to an exercise of international public authority, functionally equivalent to, and at times even more powerful than, formal lawmaking. While other indicators are still around – the UNDP’s Human Development Index, the UN Sustainable Development Goals (SDG) indicators, the OECD PISA ranking, human rights indicators –, many of them have lost their initial appeal and impact. In that sense, the end of DB is just the nail in the coffin of governance by indicators.

The reasons for this decline are partly related to indicator design and (over)use: methodological flaws and limited usefulness of highly aggregated indicators; over-supply and over-diversification of indicators and counter-indicators (the SDGs have 169 sub-goals and even more indicators); dynamics of the attention economy and declining newsworthiness of repeat rankings (the PISA-shock comes only once); declining relevance of official development assistance (ODA) in general, and of indicator-based ODA allocation in particular (with the rise of private and blended development finance and earmarked trust funds); and availability of alternative tools, with improving digital technologies and big data techniques: why use aggregated indicators based on expert surveys when you can digitally collect and process actual raw data, disaggregated all the way down to the smallest unit of relevance?

Other reasons are related to the changing political environment. The increasing (re)politicization of global governance is reconfiguring the relationship of expertise, politics, and international law. The populist attack on expertise in general, and on international expert bodies in particular, has also challenged the appeal of indicators to objectivity and science. The rise of China and other emerging economies has ended the unipolar moment of the 1990s and 2000s and led to a system of competitive multilateralism: now the World Bank competes with the Asian Infrastructure and Investment Bank dominated by China and the New Development Bank controlled by the BRICS countries (Brazil, Russia, India, China, South Africa). If DB was meant to make member states compete for good rankings and foreign investment in global markets, the tables have now turned to the extent that the World Bank is competing for Chinese investments and sacrificing DB along the way.

Finally and importantly, there are legal reasons for the decline of governance by indicators. The more indicators amounted to an exercise of international public authority, the more they depended on an adequate legal framework to ensure their legitimacy and protect their integrity. Few international institutions managed to enact such legal frameworks for governance by information. DB never had a proper legal base in the Bank’s secondary law that would lay down organizational and procedural rules to protect its independence and integrity as a scientific product, to legitimize it politically as a policy instrument, or to provide adequate legal accountability mechanisms. The UN chose a different approach with the SDG indicators, which were grounded in secondary law and enacted in a more formal and political process. This led to higher degrees of political legitimacy and acceptance, but also to a proliferation of indicators catering to a variety of interests and constituencies. This affected their ability to act as instruments of governance: while their precursors, the Millennium Development Goals, were still able to focus political efforts and public attention to a limited number of goals, the SDGs function more as internal management tools in fragmented international administrative settings and expert discourses. In this, they share the fate and function of most human rights indicators, which never rose to public prominence in the first place. All said and done, the importance of legal frameworks for the legitimacy, design and impact of indicators thus remains a lasting insight of the past decade of legal research into indicators, and an important lesson for the future of global data governance.

Indicators Are Dead – Long Live Global Data Governance

The end of indicators does not signify the end of global information governance. To the contrary, if the new global power China cares enough about DB to seek to influence it, this indicates the continued relevance of international institutions’ informational activities, and the institutional law applicable to them. International institutions remain important as hubs for informational cooperation beyond individual states and as providers of reliable data, especially with regard to authoritarian and populist regimes.

As more sophisticated digital technologies become available as tools of global data governance, international institutions also need to recalibrate their instruments, roles and functions. They will exercise less public authority through indicators towards member states, and rather engage in more finely-grained global governance of, and by, data, collecting and publicising raw data, reaching more deeply into states and towards individual data subjects, connecting research institutions, public agencies and digital platforms, and providing norms and standards for data governance at large. The success of such activities will also depend on whether international institutions will learn the lesson of DB, and devise adequate legal frameworks that ensure the legitimacy and integrity of global data governance in the future.