Beware the certainty of numbers
Image: shixart1985 via PhotoPin (licence)
As uncertainty and confusion about COVID-19 continue in people’s lives, those hungry for information are finding themselves taking a crash course in epidemiological terms. A number of information platforms have responded by publishing their own COVID-19-related glossaries, including the Yale School of Medicine and our own ABC.
But what about COVID-19 numbers? The daily media report tallies of confirmed cases and deaths across Australia, broken down by states and territories. Often, these tallies are followed by those for other countries around the globe. However, while making country comparisons is tempting, we risk comparing apples and oranges.
Numbers can appear precise, certain. But this semblance of certainty is rarely deserved, because they are as much about the tools used to make the count as they are about the quantities being counted.
These tools differ in rigour, in their assumptions, and in how they were employed, and these differences mean that numbers for Country A may not be comparable to those for Country B.
Let’s take the case of the case fatality rate (CFR), which fluctuates wildly across the globe. This is a regularly reported indicator of the deadliness of COVID-19, or the risk of death once a person has been infected with SARS-CoV-2 (the virus responsible for the disease COVID-19). The CFR is calculated as the ratio: (number of deaths from COVID-19)/(number of confirmed COVID-19 infections).
As at 9 June, the CFR in Italy, US and Australia respectively was 14.45%, 5.66% and 1.40%. The denominator alone – the number of confirmed cases – is riddled with uncertainty because it depends on a number of factors.
First, it relies on the extent to which testing was carried out. When testing is minimal or inadequate, people who have caught the virus are likely to have missed detection. Undetected infections (i.e. a lower number of confirmed COVID-19 infections than is the case) would have inflated the CFR, making COVID-19 seem more deadly than it actually is – a probable explanation for the high death rates in China that were reported early in the outbreak.
The more extensive the testing, the more likely the detection of infections, the more accurate the CFR and the more effective the modelling that can be used to inform actions – reasons why countries like Australia and New Zealand tested early and extensively.
Countries have differed not only in how thoroughly they tested their residents, but also in how they’ve done those tests. For example, in the first 30 days of the outbreak in their respective countries, Iceland not only conducted more tests per head of population than the US, but also tested those who had no signs of infection or who had no contact with infected people. By comparison, in the first 30 days of the outbreak in the US, only residents with symptoms were targeted for testing.
Further, even if testing was carried out, its recording may have been inconsistent. Inconsistent record-keeping of tests in the US, for example, has contributed to a data gap that undermined efforts to control the disease.
And that’s just the numerator alone: the number of deaths from COVID-19 – the numerator for CFR – is similarly rubbery and difficult to establish.
These imprecisions do not mean numbers are inconsequential. Methods exist to account for this uncertainty. Importantly, an awareness of these imprecisions can make readers appreciate that numbers are no more certain than words when it comes to conveying information.
Check the context of the numbers reported: what is the definition of the thing measured? How, when and where was it measured? The answers to these questions affect what the numbers are saying.
