Why Speed Demons Appear to Be Safer Drivers: Intention-to-Treat Error
You’ll find it hard to believe, but speed demons drive more safely than so-called careful drivers. Why? Well, consider this: The distance from Miami to West Palm Beach is around seventy-five miles. Drivers who cover the distance in an hour or less we’ll categorize as “reckless drivers” because they’re traveling at an average of 75 mph or more. All others we put into the group of careful drivers. Which group experiences fewer accidents? Without a doubt, it is the “reckless drivers.” They all completed the journey in less than an hour, so they could not have been involved in any accidents. This automatically puts all drivers who end up in accidents in the slower drivers’ category. This example illustrates a treacherous fallacy, the so-called intention-to-treat error. Unfortunately, there is no catchier term for it.
This might sound to you like the survivorship bias, but it’s different. In the survivorship bias you see only the survivors, not the failed projects or cars involved in accidents. In the intention-to-treat error, the failed projects or cars with accidents prominently show up, just in the wrong category.
A banker showed me (Rolf Dobelli) an interesting study recently. Its conclusion: Companies with debt on their balance sheets are significantly more profitable than firms with no debt (equity only). The banker vehemently insisted that every company should borrow at will, and, of course, his bank is the best place to do it. I examined the study more closely. How could that be? Indeed, from one thousand randomly selected firms, those with large loans displayed higher returns not only on their equity but also on their total capital. They were in every respect more successful than the independently financed firms. Then the penny dropped: Unprofitable companies don’t get corporate loans. Thus, they form part of the “equity-only” group. The other firms that make up this set have bigger cash cushions, stay afloat longer, and, no matter how sickly they are, remain part of the study. On the other side, firms that have borrowed a lot go bankrupt more quickly. Once they cannot pay back the interest, the bank takes over, and the companies are sold off—thus disappearing from the sample. The ones that remain in the “debt group” are relatively healthy, regardless of how much debt they have amassed on their balance sheets.
If you’re thinking, “Okay, got it,” watch out. The intention-to-treat error is not easy to recognize. A fictional example from medicine: A pharmaceutical company has developed a new drug to fight heart disease. A study “proves” that it significantly reduces patients’ mortality rates. The data speaks for itself: Among patients who have taken the drug regularly, the five-year mortality rate is 15 percent. For those who have swallowed placebo pills, it is about the same, indicating that the pill doesn’t work. However—and this is crucial—the mortality rate of patients who have taken the drug at irregular intervals is 30 percent—twice as high! A big difference between regular and irregular intake. So, the pill is a complete success. Or is it?
Here’s the snag: The pill is probably not the decisive factor; rather, it is the patients’ behavior. Perhaps patients discontinued the pill following severe side effects and thus landed in the “irregular intake” category. Maybe they were so ill that there was no way to continue it on a regular basis. Either way, only relatively healthy patients remain in the “regular” group, which makes the drug look a lot more effective than it really is. The really sick patients who, for this very reason, couldn’t take the drug on a regular basis ended up populating the “irregular intake” group.
In reputable studies, medical researchers evaluate the data of all patients whom they originally intend to treat (hence the title); it doesn’t matter if they take part in the trial or they drop out. Unfortunately, many studies flout this rule. Whether this is intentional or accidental remains to be seen. Therefore, be on your guard: Always check whether test subjects—drivers who end up in accidents, bankrupt companies, critically ill patients—have, for whatever reason, vanished from the sample. If so, you should file the study where it belongs: in the trash can.
* Source: The Art of Thinking Clearly by Rolf Dobelli