Never Judge a Decision by Its Outcome: Outcome Bias
A quick hypothesis: Say one million monkeys speculate on the stock market. They buy and sell stocks like crazy and, of course, completely at random. What happens? After one week, about half of the monkeys will have made a profit and the other half a loss. The ones that made a profit can stay; the ones that made a loss you send home. In the second week, one half of the monkeys will still be riding high, while the other half will have made a loss and are sent home. And so on. After ten weeks, about one thousand monkeys will be left—those who have always invested their money well. After twenty weeks, just one monkey will remain—this one always, without fail, chose the right stocks and is now a billionaire. Let’s call him the success monkey.
How does the media react? It will pounce on this animal to understand its “success principles.” And they will find some: Perhaps the monkey eats more bananas than the others. Perhaps he sits in another corner of the cage. Or maybe he swings headlong through the branches, or he takes long, reflective pauses while grooming. He must have some recipe for success, right? How else could he perform so brilliantly? Spot-on for two years—and that from a simple monkey? Impossible!
The monkey story illustrates the outcome bias: We tend to evaluate decisions based on the result rather than on the decision process. This fallacy is also known as the “historian error.” A classic example is the Japanese attack on Pearl Harbor. Should the military base have been evacuated or not? From today’s perspective: obviously, for there was plenty of evidence that an attack was imminent. However, only in retrospect do the signals appear so clear. At the time, in 1941, there was a plethora of contradictory signals. Some pointed to an attack; others did not. To assess the quality of the decision, we must use the information available at the time, filtering out everything we know about it postattack (particularly that it did indeed take place).
Another experiment: You must evaluate the performance of three heart surgeons. To do this, you ask each to carry out a difficult operation five times. Over the years, the probability of dying from these procedures has stabilized at 20 percent. With surgeon A, no one dies. With surgeon B, one patient dies. With surgeon C, two die. How do you rate the performances of A, B, and C? If you think like most people, you rate A the best, B the second best, and C the worst. And thus you’ve just fallen for the outcome bias. You can guess why: The samples are too small, rendering the results meaningless. You can only really judge a surgeon if you know something about the field, and then carefully monitor the preparation and execution of the operation. In other words, you assess the process and not the result. Alternatively, you could employ a larger sample: one hundred or one thousand operations if you have enough patients who need this particular operation. For now it is enough to know that, with an average surgeon, there is a 33 percent chance that no one will die, a 41 percent chance that one person will die, and a 20 percent chance that two people will die. That’s a simple probability calculation. What stands out: There is no huge difference between zero dead and two dead. To assess the three surgeons purely on the basis of the outcomes would be not only negligent, but also unethical.
In conclusion: Never judge a decision purely by its result, especially when randomness and “external factors” play a role. A bad result does not automatically indicate a bad decision and vice versa. So rather than tearing your hair out about a wrong decision, or applauding yourself for one that may have only coincidentally led to success, remember why you chose what you did. Were your reasons rational and understandable? Then you would do well to stick with that method, even if you didn’t strike it lucky last time.
* Source: The Art of Thinking Clearly by Rolf Dobelli