September 13th, 2011
Is Most Research Done Today a Waste?
Harlan M. Krumholz, MD, SM
Richard Smith (@Richard56), the former editor of BMJ, sent out a tweet earlier today that is worth some reflection. Here it is:
85% of clinical research is waste because the question is unimportant, the design is not right, nothing is published, or the paper is biased
What do you think? Is most research done today a waste? Are researchers really providing knowledge that is valid and useful?
18 Responses to “Is Most Research Done Today a Waste?”
NEJM Journal Watch — Recent Cardiology Articles- Strategies and Criteria for the Diagnosis and Management of Myocarditis
- Sex-Based Disparity in CABG Outcomes: Are We Moving Toward Parity?
- Can Genetic and Social Risk Scores Improve Cardiac Risk Prediction?
- Extensive Ablation vs. Pulmonary Vein Isolation in Persistent Atrial Fibrillation
- Can a Novel Stent Design Improve Long-Term Outcomes of Percutaneous Coronary Intervention?
The question about valid, pertinent, and useful information from current research needs to be asked. While EBM is a wonderful idea whose time has obviously come, and I am a fervent supporter, common sense in design, interpretation, and application of results in practice has a place too. Noncritical acceptance and incorporation of data sometimes results in harmful, unwanted, confusing, unacceptable, or absurd consequences. There are also hidden agendas of sponsors, authors, and those who spin the information.
The other side of the coin is the present difficulty (cost, administrative effort) of designing and administering a large RCT using hard end points–outcomes rather than surrogates–so that is it sufficiently powered to answer the question at hand. Many worthy questions remain unanswered because of these obstacles.
Richard Kones MD
As Marcello Truzzi said it best, “An extraordinary claim requires extraordinary proof” (this quote is often wrongly attributed to Carl Sagan).
Let us see if Richard Smith can ‘tweet’ the proof!
Dr. Kaul, the original quote is from the giant of thinking Scotish philosopher and skeptical David Hume, he said “extraordinary assertions require extraordinary proofs”. Very good Truzzi and the great Sagan follow the track.
What if we considered the claim from the perspective of those who would use the knowledge generated from research? How often is the generated knowledge used by and making an impact for patients, policymakers, payers, and providers??
What if we considered the claim from the perspective of value? Does the research seek to produce personal value (academic promotion, fame) and commununal value (improve the life of those participating in a specific community or group)? Or does the research seek to produce public value (open source access to data and models) and civic value (improve the life of everyone in society and the world)??
I would pose that we can and should do better by engaging the consumers (patients, providers, payers, policymakers) of the research upstream and striving to produce public and civic value…
Also even research without actual positive findings also is an invaluable experience for a trainee….
Competing interests pertaining specifically to this post, comment, or both:
Avid interest in clinical research.
Most cardiology research is done for marketing purposes. yes. It is a waste. I have had those same thoughts!
Richard Smith is correct. Most research sponsored by drug companies is for marketing, and is generally spun so hard as to be useless.
Competing interests pertaining specifically to this post, comment, or both:
None
What a refreshing moment to hear somebody have the b–ls to ask this question. I have my journal catch up moments when I fly,lugging 20+ journals in my backpack to read. It is a great and intense way to read and think. I am always struck by the number of articles written, funded, and printed that give no new information, no new insights, etc. It’s what I call “the duh experience”. Just think what we could do with all those research dollars if they were directed towards real, ground breaking investigations. Very depressing and frustrating.
Competing interests pertaining specifically to this post, comment, or both:
None except the cost of my journal subscriptions.
There is a lot of useful research being done everyday, but a lot more useless research. From observation, research is mandatory to survive in academia (publish or perish) or even to enter the field/advance for further in training. There are many examples of useless research (open a journal with a low impact factor and take your pick–finding a useless study is like shooting fish in a barrel).
During the application process for fellowship, it became abundantly clear that research was of major importance when even applying; poster presentations, original research as a first author, etc. As such, there is a huge amount of pressure on internal medicine residents (who all will agree do not know the most about cardiology) to put something forward. It should come to no one’s surprise that most of this is garbage. Why is there such a high demand for research? Probably to separate otherwise similar candidates…. There have to be less expensive and meaningless ways to do it, I think. This is not sour grapes as I have entered fellowship, but I believe that the focus on research, especially meaningless research, is a little ridiculous.
Further reading:
1) Contradicted and Initially Stronger Effects in Highly Cited Clinical Research, by John P. A. Ioannidis, MD
Conclusion–
“Contradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.”
http://jama.ama-assn.org/content/294/2/218.abstract
2) Origin and funding of the most frequently cited papers in medicine: database analysis by Nikolaos A Patsopoulos, John P A Ioannidis, Apostolos A Analatos
Conclusion–
“Academic affiliations remain prominent among the authors of the most frequently cited medical research. Such research is increasingly funded by industry, often exclusively so. Academics may be losing control of the clinical research agenda.”
3) ‘They would say that, wouldn’t they?’ A reader’s guide to author and sponsor biases in clinical research by Hilda Bastian
http://jrsm.rsmjournals.com/content/99/12/611.full
I agree, particularly having reviewed the output of my PhD. However the process taught me a great deal and I hope I am a better doctor for it.
As I signed on to the net today, Google informed me today is the 108th anniversary of the birth of Albert Szent-Györgyi, Nobel laureate biochemist and physiologist. Later in his career, when he found it increasingly difficult to obtain funding for his work, I believe it was he who lamented the failure of the underwriters of medical research to recognize the role of intuition in scientific investigation. Pragmatic and utilitarian interests have clouded our ability to recognize and reward genius. Every candidate for medical residency has a bibliography these days. Most of the articles listed are about as original as the black suits all of the prospective house officers are told to wear to the interview.
Competing interests pertaining specifically to this post, comment, or both:
None, but I do enjoy a glass of fresh orange juice.
The “importance” of research is – largely – a subjective matter and is, as such, open to debate by society, funding agencies and (of course) former editors of prestigious journals. However, issues of methodology (including bias and study design) would appear to be more objective in nature, and therefore less open to subjective interpretation.
Just reflections from reading, with recommendations:
Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124
A must-read, by a statistician, with persuasive arguments that we’re mostly ingesting garbage (my interpretation.)
And, to make more cynics:
What’s the evidence for Evidence-Based Medicine?
Body, R., Emerg Med J 25(12):841, December 2008
It appears the evidence for evidence-based medicine is opinion, which is exactly what EBM is trying to get away from.
Competing interests pertaining specifically to this post, comment, or both:
No conlicts of interest.
Xanthelasmata Identified as Independent CV Risk Factor.
A typical example of waste.
We know this already more than 30 years.
I agree with most of the above comments. However, the research is of paramount importance and the 21st century wouldn’t be the same without it. I think that basic research, which is probably even more important, is less impacted by outside influences and is under-appreciated. As far as cardiology research goes it is very commercialized and in muddy waters. The stakes are so high that it affects the outcomes. It gets reported all over the place:TV, newspapers, magazines, etc. As far as EBM is concerned, an even bigger issue is implementation.
Competing interests pertaining specifically to this post, comment, or both:
none
I would add that researchers often seem to feel strongly compelled, in the context of securing stable grant support and academic promotion, and having led themselves down the garden path, to keep going through the garden and ever deeper into the woods until the truth is completely lost and all that matters is continued forward motion.
Competing interests pertaining specifically to this post, comment, or both:
none
I read last year that at any given moment there are approx. 40,000 (yes, 40 x 10e3) trials going on world-wide.
Multiply that by the cost for investigators, personnel, equipment, offices, indirects, etc and you can see that institutional/academic medicine thrives/lives on these subsidies regardless of their intrinsic value.
The income co-opts faculty, deans, and presidents of even our greatest institutions at a time when student tuition is rising precipitously.
The whole issue of the financing of medical teaching institutions and research needs re-examination and critical evaluation free from the usual encomiums trumpeting how research makes our lives better through drugs and technology.
Competing interests pertaining specifically to this post, comment, or both:
None
yes yes yes. It seems like every day I read a new article in a peer reviewed journal describing outcome of funded research that is so ridiculous, so “old”, so “obvious” that I cannot continue with the journal. Who decides to fund these absurd “studies”. Millions spent on useless or repetitive or unncecessay investigation while the actual enterprise of healthcare and patient care is in chaos…
Competing interests pertaining specifically to this post, comment, or both:
NONE. Private practice, dinosaur solo internist