Defying the Statistics

- What is the chance that you meet a dragon on the street?
- Fifty per cent.
- ???
- Well, either I meet him, or I don't!
A joke

Statistics plays a large part in psychiatry, as in the rest of medicine, because it helps clinicians to make decisions about what is likely to be beneficial or harmful to their patients. Most of the guidelines for physicians are based on the statistically significant conclusions from the clinical trials. The argument goes like this: It has been found that, say 70% of the patients with a certain diagnosis and a particular set of other characteristics (age, race, height, weight, etc.) get better on a medication. Therefore, if you have a patient Ivan Johnson who fits all these characteristics, his chance of benefiting from this medication is seventy per cent, which means he should take it.

Here at the American Medical Informatics Association conference I constantly witness conclusions like this being drawn. They have become basis of medical expert systems and much of medical informatics in general. Yet, I am going to show with one simple example, that this logic is erroneous. It is a case of scientific confusion between the probability of an event observed in many previous experiments, and the ability to correctly predict the result of one next experiment.

Let's not go as far as medical predictions. Let us take the simplest model of probability - a tossed coin. The probability of the heads is 50% - everybody knows that. Indeed, if we toss a coin a billion times, we will probably observe the heads in something very close to 50% of the cases. But that is all it means! You can't go any further in your conclusions! In your ability to predict what the coin will fall next, in your one-billion-and-first experiment, you are as helpless as you were when you'd thrown it for the first time. You simply don't know what it will be - all you can say is that it will be either one or the other.

It is not because the probability is 50% that you don't know. Take an experiment with the chance of outcome A being 90% and the outcome B being 10%. What is the probability of the outcome A in the next experiment? Nine out of ten, very well. But what outcome will the next experiment have? You can't ask this. You don't know. It will be either one or the other.

So what does this nuance mean for the system of psychiatry and health care in general? Not much, because the system operates on statistics, it deals with patients en masse, and therefore the results of multiple experiments correlate well with the predictions based on probability. But it means the world for an individual patient; for you and me.

Even if we forget for a minute that the experiments in which the probability is calculated were not actually done on you and me, but on some other people; even if we forget that the medical outcomes are not binary but multidimensional and unpredictable by their very nature - even then predicting an outcome for a particular patient will be impossible because of the effect I've just described. If 99.99% of the patients like you die in a month without an operation, it doesn't mean that you will also die in a month. The probability of it is high; whether it happens is not known.

I admit of course that with a chance like that you might want to strongly consider the operation; I would probably do the same. However, the majority of psychiatric statistics operates with numbers quite remote from the extremes. People talk about reducing risks from twenty-five to fifteen per cent, of improving the outcome in sixty per cent of the patients versus forty. For the hospital and the insurance company these numbers are big business; for you and me, they are simply meaningless.