Are you skeptical of the latest peer-reviewed results?
Perhaps, like American science fiction writer Theodore Sturgeon, you are skeptical of much that is presented to you as fact. I don't mean this little opinion piece I write each fortnight, which everyone understands to be pure malarkey. No, I am referring to scientific medical publications in our top-tier, peer-reviewed journals.
Well, Greek epidemiologist John Ioannidis, MD, from the University of Ioannina School of Medicine, Ioannina, Greece, offers proof that you are right to be skeptical of what you read. He reviewed 49 "important" research articles published in top medical journals between 1990 and 2003. These important articles were subsequently cited more than 1,000 times by other researchers in their papers. Over time, almost one-third of these papers were found to be wrong.
One refuted study looked into whether hormone replacement therapy was safe for women (first it was, then it wasn't), whether vitamin E protected against heart disease (first it did, then it didn't), and whether stents are better than balloon angioplasty for coronary artery disease (they are, but not as much as originally thought).
Inadequate sample sizes, poor study design, researcher bias (and we all have our biases), financial interests, faulty statistical analyses-all of these were found by Dr. Ioannidis to contribute to studies falsely purporting to reveal new medical truths. Even a large, well-designed study with minimal investigator bias will only give the right answer 85% of the time, according to Dr. Ioannidis' calculations.
Researchers aren't trying to deceive us (in general). But even careful studies performed by well-intentioned medical scientists are imperfect. And let's face the fact that full-time academicians need to publish important papers and get grants if they hope to secure promotions and tenure. So one can't necessarily wait for confirmatory evidence before publishing a "landmark" study that offers a new cause (or cure) for cancer, age-related macular degeneration, or other disease.
According to Dr. Ioannidis, Theodore Sturgeon is wrong; 50% (not 95%) of peer-reviewed biomedical science is crap. One of these two gentlemen is evidently full of, shall we say, baloney.
The problem is that medical practice is not a debating society. Changing a patient's medications or performing a different surgical procedure on the basis of a study subsequently proven false is not a good thing.
To me the obvious conclusion is that we doctors should remain skeptical of new claims, even if published in our prestigious peer-reviewed journals. In general, let's wait for the second, confirmatory study from a different group of investigators before changing how we care for our patients on the basis of "exciting" new data.
Ninety-five percent may satisfy Mr. Sturgeon, but not your chief medical editor, who will continue to strive for 100% in this column.
1. Ioannidis JP. Why most published research findings are false. PLoS Med 2005 Aug;2(8):e124. Epub 2005 Aug 30.
2. Scientific accuracy . . . and statistics. Just how reliable are scientific papers? The Economist Sept. 3, 2005.
Peter J. McDonnell, MD is director of The Wilmer Eye Institute, The Johns Hopkins University School of Medicine, Baltimore, and chief medical editor of Ophthalmology Times. He can be reached at 727 Maumenee Building, 600 North Wolfe St., Baltimore, MD 21287-9278 Phone: 443/287-1511Fax: 443/287-1514 pmcdonn1@jhmi.edu
