Bad outcome does not mean a bad decision has been made.

Bad outcome does not mean a bad decision has been made.

I saw this quote in an accounting textbook. Of all places. But I think it sums up pretty nicely what it means to live in a world that is ruled by probabilities. Too often I see people analyze various failed ventures trying to identify the mistakes. It's quite possible that no mistakes were made; maybe everything has already been done to maximize the chances of success but sometimes the odds just line up against you.

So can we make it official that there is a LOT of bad science out there?

Something that I have been noticed many years ago when I tried looking at some papers on the social and medical studies is that very frequently they perform statistical analysis incorrectly, and draw conclusions unsupported by their own data.

Well I just found a great article which not only fully describes this problem, but gives a lot more additional insight to offer: Odds Are, Its Wrong.

Choice quote from the article:

In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own.

Now when I want to explain people why I don't believe the results of most recent medical studies, I will just give them this link!

Let it be noted, however, that statistical analysis is usually done much more rigorously and soundly in fields like Physics and Chemistry. But it is only achieved by vigilance of the reviewers and constant emphasis on that fact that evaluating the uncertainties in your research is much more important that the actual result. I remember that on almost every paper I worked on, 80% of the total research time was spent analyzing the probabilities of your results being wrong.

From bad science department: environmental impact of walking.

I often lament the lack of common sense in many scientists. Today's exhibit is going to be about a crazy piece of research which was even enthusiastically discussed by Freakonomics blog in Be Green: Drive followed up by More Analysis of the Environmental Impact of Walking vs. Driving. The researchers tried to calculate the carbon footprint of walking 1.5 miles versus driving 1.5 miles. Depending on how bad your diet is, walking could generate more carbon. One author then even went as far as recommend driving instead as more environment friendly.

Let me make a few observation which firmly put all of that research into mad science department.

• Worrying about the carbon footprint of human metabolism is silly for two reasons. First, unless you want to consider killing humans, we cannot do much about metabolism itself. It continues even while we sleep. Second, this carbon footprint comes mostly from food production, so we should work on making food production cleaner, not stop walking around, as the paper seems to suggest.
• Among those who try to achieve a greener lifestyle, walking is not considered an alternative to driving. Environmentally friendly alternatives to driving are living closer to work, taking commuter trains and using bicycles (which are 3-5 times more efficient than walking).
• People need exercise to stay healthy. That's why people should take a walk instead of driving (that, and fresh air).
• Even if all of us walk all day long, we will only increase the pollution levels by a minuscule amount.