Let’s assume that you get a call from a relative. Said relative has enjoyed packing away the carbs for years on end, and is built along, shall we say, fairly generous proportions. But also – and this is the good news – said relative would now like to shed some kilos.
You, trained in statistics and fitness, have been enlisted as an important team member of Team Let’s Lose Some Weight.
And so off you go to do your research online, and after barely half an hour of ferreting around on Google, you come back and announce that there are two pills that will get the job done. While both have some unfortunate and unavoidable side-effects, both are also guaranteed to help you lose weight. You don’t need to do anything else, you tell your relative. No exercise, no diet, no lemon water in the morning, none of that jazz. Pop the pills, and you’re guaranteed to lose weight.
But, you go on to say, there’s also bad news.
What’s the bad news, asks the relative.
Well, you say. The first pill – Oomph is its name – it will help you lose nine kgs on average in a month.
Nine kgs, goes the old relative. And you call that bad news?!
Hang on, you say. Not so fast. Yes, nine kgs, but focus on the next two words, no? “On average”.
And what does that mean, asks the old relative testily.
Well, you say, always glad to show off your knowledge of statistics, it means you could lose anywhere between 4.5 kgs to 13.5 kgs. On average you will lose about nine kgs. But could be more, could be less.
Almost never, you say, will it be less than 4.5 kgs. And almost never, you say, will be more than 13.5 kgs. But somewhere within that range, you go on reassuringly, you will lose the kgs.
Ah, goes the old relative. Like that. OK, then. And what about the second pill?
Ah, the second pill, you say. Well, in the case of the second pill, you will lose only 2.25 kgs.
Pshaw, snorts the old relative. Peanuts compared to the first one, no?
Well, yes, true, you concede. But on the other hand, on average it will be somewhere between 2.05 kgs and 2.45 kgs.
Ah, says the old relative. So what you mean to say is that the first pill gives great results, but more uncertainty. And the second pill gives not so great results, but less uncertainty.
Couldn’t have put it better myself, you say. Well done.
Well, that leaves only one question then, doesn’t it?
As an expert in statistics, which one do you recommend?
How I wish I could take credit for this example, since it is such a great question to think about. It is, alas, not my own. It is taken from a lovely book, which happens to be the subject of both this post, and quite a few others in this week. The name of the book is “The Cult of Statistical Significance“, by Stephen T. Ziliak and Deirdre N. McCloskey.
And let’s leave aside for the moment the question of what the book is about – although we’ll get to it, don’t you worry. But for the moment, please do think about the answer to the question: which one do you recommend?
Remember, one helps you lose somewhere between 4.5 to 13.5 kgs, while the other helps you lose somewhere between 2.05 to 2.45 kgs. Do you pick the pill with the greater uncertainty (but more weight loss), or do you pick the pill with the lesser uncertainty (but lesser weight loss)?
This is, of course, a topic that has been discussed before on this blog. We’re talking, all statisticians will tell you, about the signal to noise ratio:
A clear signal with insane amounts of noise ain’t a good thing, and an unclear signal with next to no noise is also not a good thing.
https://atomic-temporary-112243906.wpcomstaging.com/2021/05/18/on-signals-and-noise/
So the first pill – Oomph is its name – has a clear signal (9 kgs!), but also insane amounts of noise (whaddya mean, somewhere between 4.5 kgs to 13.5 kgs?!).
And the second pill – Precision is its name – has an unclear signal (only 2.25 kgs?!), but also next to no noise (wow, plus or minus 200 grams only!).
So we have a situation where we must choose between a not-so-good thing and another not-so-good thing. So what do we choose? What should we choose?
So here’s the thing.
If you’ve been trained in statistics (as I have), you should be saying, choose the pill with the higher signal to noise ratio.
Stop right here, if you’ve been trained in statistics, and tell me if you agree or disagree with me. If you disagree with me, please tell me why. Has to be the signal-to-noise ratio, no?
Right, so let’s go ahead and calculate the signal to noise ratio. Here’s the formula we will use:
The hypothesized null effect is, in both cases, zero. In the first case, the observed effect (on average) is 9, and the variation is 4.5. In the second, the observed effect (on average) is 2.25, and the variation is .2
So: (9-0)/4.5 and (2.25-0)/.2
Giving us, effectively, a signal to noise ratio of 2 and 10, respectively.
Well, you confidently tell your old relative, I’ve run the tests and done the analysis. And my conclusion is that you should take the second pill.
Precision, you mean?
And why is that?
You sigh. Deeply. Explaining statistics to laypeople is such a chore, but someone has got to do it.
Because, you say in your best professorial tone, the signal to noise ratio is the highest in the case of the second pill. Not just higher, you say patiently – five times higher. 10 compared to 2! It’s not even close.
But this old relative of yours is nothing if not curmudgeonly and commonsensical.
So you mean to tell me, goes the o.r., that with Precision, the one that you’re recommending…
… my best case scenario is that I will lose 2.45 kgs.
But, in the case of Oomph, the one that you’re not recommending…
… my worst case scenario is that I will lose 4.5 kgs.
Have I got that right?
For an uncomfortably long period of time, there is a strained silence. And then in a small voice, you say that you will get back to the old relative, and off you go to learn more about where you went wrong while learning statistics for all these years.
So where did you go wrong?
There’s good news, and here’s bad news.
The good news is that you didn’t go wrong. You learnt correctly.
The bad news? Statistics itself took a wrong turn, and hasn’t corrected itself since.
How? We’ll find out soon enough, stay tuned.
EconForEverybody 