So can we make it official that there is a LOT of bad science out there?

Something that I have been noticed many years ago when I tried looking at some papers on the social and medical studies is that very frequently they perform statistical analysis incorrectly, and draw conclusions unsupported by their own data.

Well I just found a great article which not only fully describes this problem, but gives a lot more additional insight to offer: Odds Are, Its Wrong.

Choice quote from the article:

In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own.

Now when I want to explain people why I don't believe the results of most recent medical studies, I will just give them this link!

Let it be noted, however, that statistical analysis is usually done much more rigorously and soundly in fields like Physics and Chemistry. But it is only achieved by vigilance of the reviewers and constant emphasis on that fact that evaluating the uncertainties in your research is much more important that the actual result. I remember that on almost every paper I worked on, 80% of the total research time was spent analyzing the probabilities of your results being wrong.