In 1989 Leonard Mlodinow was unexpectedly denied life insurance due to the results of a blood test. When he called up his doctor’s office, he received the very scary news that he had a 999 out of a 1000 chance of having HIV, based on on the blood test’s false positive rate of one in 1,000. Fortunately Mlodinow (a theoretical physicist) knew enough about probability theory to be skeptical. He looked up statistics about his demographic group and learned that only about one in 10,000 people were actually confirmed as having HIV after testing positive. If one in every 1,000 results was a false positive, then on average ten of the 10,000 were likely to get a false result. In his book The Drunkard’s Walk Mlodinow writes:
We end up with 10 people who are false positives and 1 truly positive. In other words, only 1 in 11 people who test positive are really infected with HIV. My doctor told me that the probability that the test was wrong — and I was in fact healthy — was 1 in 1,000. He should have said, “Don’t worry, the chances are better than 10 out of 11 that you are not infected.”
As his analysis might suggest, Mlodinow did not turn out to have HIV. In tree form the counts look something like this:
The doctor’s mistake is called the Prosecutor’s Fallacy because it often comes up in the context of forensic evidence. If there is a one in a thousand chance of the defendant’s DNA matching the DNA at the scene, the prosecutor may imply that there is a 99.9% chance that the defendant is guilty. However, if there are hundreds of thousands of samples in the database, there could be hundreds of people whose DNA matches the sample but are not guilty. The result of a DNA test has to be considered in context. If the defendant is the victim’s ex-husband, the DNA match is pretty damning. If the DNA match is in the system from a property crime ten years back there’s a decent chance he’s not your guy.
The Prosecutor’s Fallacy illustrates something counterintuitive about probabilities: when you are dealing with a large population or conducting many trials, unlikely things are likely to happen some of the time. The odds that you or I any other person in the world will be struck by lightning in the next year is infinitesimally small (~ 1 in 700,000). Yet every year people die from lightning strikes because the union of all those tiny probabilities is very large.
A key takeaway is that a good way to increase the likelihood of an improbable outcome is to conduct more trials. In the book How Not to be Wrong Jordan Ellenberg relates the parable of the “Baltimore stockbroker”. The stockbroker sends you a letter in the mail predicting that a particular stock is going to go up in the next week, which it does. He sends another letter the next week and again his prediction is correct. This goes on for five weeks, at the end of which he asks whether you would like to engage his services.
Here’s the catch -- that first letter went out to 1024 people. Half of the recipients got a letter saying the stock was going to go up and half were told that the stock would go down. Only the people who received the correct prediction got another letter the next week. If on average the predictions are correct 50% of the time, after five weeks only 32 people will be getting the letter. Those final recipients have been given good reason to be confident in getting a good return on their investment even though the stockbroker hasn’t demonstrated any more skill than someone guessing at random.
The Baltimore stockbroker is fictional, but brokers who sell shares of mutual funds use a similar strategy by “incubating” funds for a few years before making them available to the public. Funds that perform poorly are quietly cut while funds that do well are touted to investors with stats about their impressive returns. Maybe those funds were invested very cleverly. More likely they were just lucky for a few years, which doesn’t guarantee that they will remain so in the future.
The reality that remarkable things can and do happen at random is at odds with the way we intuitively see the world. Studies show that people are inclined to spot patterns and causal relationships where there are none. When stock prices begin to recover or a great team has a streak of losing games we could stand to be a bit more skeptical before assigning praise and blame. No coach or politician exerts as much influence as good old regression to the mean.
References:
The Drunkard’s Walk by Leonard Mlodinow
How Not to Be Wrong by Jordan Ellenberg
The Art of Statistics by David Spiegelhalter
Naked Statistics by Charles Wheelan