## Wednesday, 21 November 2012

### Statistics and a Big Pinch of Salt

The CF Trust (UK), Cystic Fibrosis Foundation (US) and Cystic Fibrosis Canada all publish annual reports based on the data collected in their patient registries. The statistic that people always focus on is “median predicted survival”.  People talk about it as if it means the same thing as “life expectancy”.

Based on the latest available data, the median predicted survival number in the US, UK and Canada is 37, 41 and 48 respectively. How can it be eleven years higher in Canada than in the US? Patients in both countries have essentially the same treatment options and the genotype profiles are very similar. Also, the number in the US went down slightly from 38 in the previous year. This is very unlikely to be because treatment options got worse year-on-year. It suggests something weird happened in the data and, more generally, that statistics like this should be taken with a hefty pinch of salt.

What exactly is “media predicted survival”?

Median predicted survival is the age beyond which half of the people currently in the relevant CF patient registry are expected to live and below which half are not expected to survive.

Median predicted age of survival is calculated using a standard actuarial method called life table analysis. This is best known for its use in the life insurance industry. It is calculated by looking at how many people are in each“cohort” or group of people born in a given year (say 1975) and finding how many of them survived the past 12 months. That gives an age-specific mortality-rate. Say, at the start of the year, there were 100 people in the database who were born in 1975 and that 90 of them were still alive at the end of the year. Of course, some of the ten that died may have died from something unrelated to CF. Since 90 of the 100 survived, the age-specific mortality rate for that year was 10%. Then the people at the registry will assume all the people born in 1976 (i.e. a year younger than the first cohort) have a 90% chance of surviving the next 12 months (on the basis they will be exactly like the people a year older) and they can work out a life expectancy for each cohort. This process is repeated for all the cohorts.

They then run the numbers and figure out the age above which half of the people currently in the CF patient registry are expected to live and below which half are not expected to live. This is the median predicted survival age. It is specific to the people whose data is currently in the database.

Is "median predicted survival" the same as "life expectancy"?

No - median predicted survival is NOT the same thing as life expectancy. Life expectancy is the expected average length of life for someone of a certain age based on currently available data in relation to people of that age. Obviously the average CF baby born today will have a longer life expectancy than the average person with CF alive today but born 30 years ago, as new treatments and earlier diagnosis have made a big difference, but both could be in the same patient registry population.

Today in Canada, life expectancy at birth is 77 years while the median predicted survival is 48 years.

Why is “median predicted survival” so different to “median age at death”?

Median age at death is calculated by listing the ages of all the people who have died in a given year in ascending order and finding the middle number (i.e. the number separating the oldest 50% from the youngest 50%). The median age at death is calculated only on the basis of those individuals who have died. In other words, of the ones who died, half of them died younger than the median age of death and half died older than the median age of death. Clearly, this calculation does not tell us anything about the individuals who have not died in that year. You need to know the ages of those still living to get a feel for life expectancy and median predicted survival.

If you think about this in terms of a Formula 1 grand prix, you could look at a particular race in a particular season and make a note of all the cars which crashed and failed to finish. Of those, you could look at how far into the race they each got before they crashed. You might find that more than 50% of the "crashers" crashed in the first 30 minutes of the race. That is fine, as far as it goes, but it would be big mistake to conclude from this that any car starting this race in future would probably crash out in the first 30 minutes. This kind of analysis does not tell the whole story because it fails to take into account the cars which did not crash i.e. the "finishers" and they are very important cars.

Lies,Damned Lies and Statistics

Another reason for caution is that not everyone with CF is on the registry. Most are now but some people do not consent to have their data included. Others are not on there because they don’t go to clinic often enough to enable the researchers to get the data.I would speculate that the people who are not on the registry are relatively healthy. If that is the case, the statistics based on the data in the registry maybe slightly skewed too much towards the “unhealthy”.

A more general problem with averages is they only take into account things that have happened in the past. By definition, all the data points used to calculate the average are backward-looking. By the time averages are published they are already out-of-date. Imagine trying to drive a car without being able to look at the road ahead; only being able to look back through your rear-view mirrors. This is essentially what you are doing if you place too much reliance on averages and other statistics. The financial markets have learned this lesson the hard way over the last few years. It is a particular problem with Cystic Fibrosis because the medical science and treatments are improving at a relatively rapid pace. For example, the current data take no account of Kalydeco since it came to market after the end of the period for which data was collected. Other new treatments are coming on-stream all the time. This means the statistics will always lag behind the reality.

Another problem is that there is often no such thing as the average case (i.e. the statistical average does not actually correspond with anything in the real world) or that it produces something misleading.

If you are 1.6 metres tall and unable to swim, is it safe to cross a river which is 1.2 metres deep on average? Not if the deepest point is actually 2 metres. The average is no use to you in this situation. In fact, it is dangerous.

Now imagine 10 people each holding a tin of paint. Six of them are holding tins of red paint, four of them green paint.  What is the average colour? You could find the“median” by writing them all down in a list (red, red, red, red, red, red,green, green, green, green) and finding the colour in the middle. Do this by starting at either end and crossing out the items until you get to the middle.  You end with four of each colour crossed out and two reds left in the middle. So the median colour is red.  Does this tell you anything interesting about the colours? Not really. To me, it just begs the question of why some are red and why some are green and why not any other colours.

If you wanted to calculate the “mean” average, you could empty all the tins into one big pot, mix it up and then divide it back into the 10 tins. You will get brown paint. But in reality none of them was holding a tin of brown paint.  It is much more interesting and meaningful to take a given tin and look at what is inside.

Of course, published statistics tell you about average outcomes based on past experience across the whole CF population (i.e. for all mutations). They do not even tell you the average specific to your mutation. Not yet, anyway. To date, over 1,600 mutations have been identified and they all have different implications. Even two people with exactly the same mutation, can be affected in very different ways. Every person with CF has their own unique hue and shade. Focus on the colour of the paint in your tin; don’t pay too much attention to these statistical snapshots.

One thing that is clear from the data is that longevity and quality of life for people with CF is increasing significantly and steadily over time as a result of earlier diagnosis (and now newborn screening); better drugs and medication; better transplant options and techniques; more sophisticated physio and airway clearance techniques; and perhaps higher levels of awareness. With little support from governments around the world, this is largely as a result of the work done by the CF Foundation in the US, the CF Trust in the UK and Cystic Fibrosis Canada and the data they present is probably the best available to tell a complicated story.

For anyone who does want to read the latest patient registry reports they are accessible here (and there is some interesting stuff in there if you read the discussions behind the headline numbers):

UK Data from CF Trust (2010):

USA Data from Cystic Fibrosis Foundation (2011)