Lies, Damned Lies, and Medical Science

I just found this excellent article in The Atlantic on the problems that pervade medical research. It focuses on a "meta-researcher" named John Ioannidis. As a meta-researcher, he researches medical research.

The article is amazingly well written, but it is long, so I'll just provide some quotes:

Wasn’t it possible ... that drug companies were carefully selecting the topics of their studies — for example, comparing their new drugs against those already known to be inferior to others on the market — so that they were ahead of the game even before the data juggling began?

...

An obsession with winning funding has gone a long way toward weakening the reliability of medical research.

...

The range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals.

...

“There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

...

To get funding and tenured positions, and often merely to stay afloat, researchers have to get their work published in well-regarded journals, where rejection rates can climb above 90 percent. Not surprisingly, the studies that tend to make the grade are those with eye-catching findings. But while coming up with eye-catching theories is relatively easy, getting reality to bear them out is another matter. The great majority collapse under the weight of contradictory data when studied rigorously.

...

How should we choose among these dueling, high-profile nutritional findings? Ioannidis suggests a simple approach: ignore them all.

...

“The odds that anything useful will survive from any of these studies are poor,” says Ioannidis—dismissing in a breath a good chunk of the research into which we sink about $100 billion a year in the United States alone.

...

Nature, the grande dame of science journals, stated in a 2006 editorial, “Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.”

...

Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correct more often than as flawed—in one case for at least 12 years after the results were discredited.

...

Ioannidis initially thought the community might come out fighting. Instead, it seemed relieved, as if it had been guiltily waiting for someone to blow the whistle, and eager to hear more.

...

In fact, the question of whether the problems with medical research should be broadcast to the public is a sticky one in the meta-research community. Already feeling that they’re fighting to keep patients from turning to alternative medical treatments such as homeopathy, or misdiagnosing themselves on the Internet, or simply neglecting medical treatment altogether, many researchers and physicians aren’t eager to provide even more reason to be skeptical of what doctors do

One thing that struck me that I think is very significant is that the author describes the medical community's reaction to some of these conclusions. They were unanimously "relieved", and not at all surprised. I think that speaks volumes -- if this were about pseudo-scientists or cranks, they would almost certainly circle the wagons and sharpen their pitchforks (sorry for the mixed metaphor). But in fact, it seems clear that the individuals are good people wanting to do real science and uncover the truth, but that they are trapped in a system that's beyond their control.