Introduction

These are some thoughts and questions about the IHME study of excess mortality published on May 6th. Originally posted on Twitter, so wording is very straightforward, I modified it a bit for an article format.

The main result of IHME’s analysis is that global excess deaths are 2-2.5 times higher than reported COVID deaths is probably true, though this might a lower bound as well. What worries me is the details that they have shared, which show that their excess estimates are wildly out of bounds of anything we know, even though data should be identical, and that they have very little actual data to work with.

On May 9th, IHME incorporated their excess death estimate to their COVID-19 interactive trackers. This allows me to inspect their results directly without trying to “eye-ball” their maps and figures. Article has been updated accordingly.

Critique

Data

They state their data comes from “56 countries and 198 subnational units have reported either weekly or monthly deaths from all causes for parts of 2020 and for prior years.”
But where is that data? The references they list only include some National Statistics Offices (NSOs), hardly 56. Moreover, are the sub-national data from these same countries or separate? In one of the videos explaining these estimates, they state that sub-national data is “US states, Canadian Provinces, Mexican states, etc.” so the answer is prob. That there is significant overlap between the national and sub-national data.

On the left hand side there’s a CSV file of “Excess Mortality data sources”:

It lists 47 countries according to my count. Some of the sources listed are outdated. For example, the latest source for Japan is September 2020, while World Mortality already has data for February 2021.

Using these not transparent data they estimate expected deaths and then use the difference between observed deaths and the expected deaths to obtain Excess Deaths, they supposedly also correct this for obviously non-COVID related excess deaths (such as the increase in Opioid deaths in the US).

After which, they estimate some model where excess deaths are explained by number of COVID tests, positivity rate and other demographic variables. They then PROJECT excess deaths based on the available variables for other countries. It is completely unclear for which countries they have actual all-cause mortality data and for which they do not. I do not think they have any information for India or Africa (other than South Africa) for example.

The results are presented the same way for countries for which they have actual all-cause mortality data, and for those they have none.

Excess Deaths Estimates

Then, they contrast these excess death estimates with the reported COVID deaths from each country. They published the numbers for some countries in Table 1:

(it continues).

This is how I am able to check their excess death estimates (shown in the column “Total COVID-19 deaths”) for countries – and some of them are suspicious, to say the least.

United States

The US excess deaths at 900k is much higher than World Mortality’s estimate of ~605K and the CDC estimate, which is similar. How has IHME arrived at this figure? Unclear.

Japan

For Japan, IHME reports excess deaths of more than 108k, vs. reported COVID deaths of 10,390. This is in stark contrast to World Mortality’s estimate and various other papers on the subject, which have found, if anything, NEGATIVE excess (deficit) mortality in 2020-2021. Indeed, just from looking at the raw data for monthly deaths in Japan (which IHME cites and supposedly uses up to September 2020), it is unclear where such large excess could arise, unless the model fit is bad:

This is World Mortality prediction (in black) and the raw data from Japan’s NSO:

It should also be noted that the ratio they find between Excess and COVID deaths of ~10 would place Japan as one of the worst offenders in undercounting their COVID deaths – similar to Kazakhstan (9.8) and higher than Russia (4.8).

France, Germany, UK

For France, Germany, and the UK, they find Undercount ratios of 1.25, 1.45 and 1.4 respectively. This again is in contrast to existing excess deaths estimates, which in all these countries are at about COVID deaths and even lower than COVID reported.

This is since the gap between excess deaths and COVID deaths has narrowed after the first wave, with COVID deaths even eclipsing excess deaths, as can be seen in this figure, which plots the weekly excess vs. weekly COVID in several countries:

India

Another country that was prominently featured in the IHME report and media coverage is India, where IHME estimate 654,395 deaths compared to 221,181 reported.
However, data from India is very hard to come by. Indeed, their own “Excess Mortality Data Sources” indicate no source for all-cause mortality data from India.

I have been in contact with researchers and journalists from India and probably the best we have right now are incomplete data from Mumbai, Nagpur and Ahmadabad, collected from various sources and publicly available in Local Mortality (with sources listed):

In my opinion, excess deaths far exceed COVID deaths in India, and under-counting is probably extremely large, especially in 2021, as shown by the FT. However, it is one thing to posit that undercount exists and could be large and another thing to put what seems to be an arbitrary number (2.93) on it.

Does IHME honestly think that there is more undercounting of COVID in Japan (undercount ratio of 10) than India?

Literature and sources

The IHME report fails to cite (and does not use?) World Mortality, HMD’s Short Term-Mortality-Fluctuations database or any of the many media and academic studies of excess mortality such as FT, NYT, The Economist, WSJ and Our World in Data.

It does not contend with the excess death estimates from these and other sources such as the Imperial College’s reports on excess mortality, The CDC, Institute and Faculty of Actuaries, EuroStat and many others which a simple Google or medRxiv search would show.

Some more questions on data

Some of the countries prominently featured in the tables of the “detailed report” have no data listed: India, Iran, Egypt, Peru, Indonesia, North Macedonia, Belarus.

So these countries, and many more which are not listed in the sources, have NO DATA on all cause mortality, and the results there are purely driven by a model which ties together excess deaths with various COVID related and demographic variables.

This is evident in the case of Turkey, where IHME estimated undercount ratio is about 1.5. Turkey has not released any all-cause mortality data for 2020 (let alone 2021) and I doubt IHME has it (they do not list it). The closest thing we have to all-cause-mortality data for Turkey is Güçlü Yaman’s collection effort of all-cause mortality in 24 cities which account for about 50% of deaths in ordinary times. If this ratio holds during the pandemic, the undercount ratio in Turkey should be about 5, with over 100K excess deaths compared to 20K COVID deaths up to end of 2020.

Indeed, this projection could be wrong - excess deaths outside these cities in Turkey could be higher or lower so that the 50% coverage does not persist into 2020 and 2021. But I would bet that this figure is much closer to the truth than the 1.5 ratio estimated by IHME using NO actual mortality data from Turkey in 2020.

Concluding Remarks

The main point of the IHME report, that total COVID deaths, estimated by excess deaths, are much larger than reported COVID deaths, is most likely true and the fact that they have drawn attention to this issue is welcome. In a study of 94 countries and territories by Dmitry Kobak and myself - we estimate this ratio (based on actual all-cause mortality data) at 1.6. We believe this to be a lower bound since we lack data for much of the world, where more localized reports and studies demonstrate larger excess.

The issue with the IHME report is that it uses extremely partial data when much more encompassing (such as World Mortality) exists, the issue is that the country-level estimates they showed publicly are incredibly different than known ones (mostly higher) and that they purport to accurately estimate excess deaths where data simply does not exist - this undermines a tremendous effort currently underway to improve and collect vital data in many countries.

As Prof. Stéphane Helleringer put it: “I do worry a lot though about false impression of knowledge and confidence that is conveyed by their estimates; especially the detailed global maps like the ones they just produced for COVID death toll and MANY other health indicators for which few or no data are available. The risk is that IHME figures, with their apparent precision, will distract some funders & governments from goal of universal death registration in low to middle Incomes countries. From their standpoint, if IHME readily estimates mortality, why invest in complex systems to register each death?”.