September 20th, 2012
Why I Applaud the Skeptical Physicians
Joseph S. Ross, MD, MHS
In this week’s New England Journal of Medicine, Kesselheim and colleagues publish findings from their randomized study of 269 internists who were asked to interpret the results, described in scientific abstracts, of three hypothetical clinical trials of three made-up drugs: lampytinib, bondaglutaraz, and provasinab. The participants knew that the trials were not real and that the study’s aim was to investigate the role that disclosure of funding plays in physicians’ interpretation of clinical trial research.
The trials were randomly configured as having high, medium, or low methodological rigor, depending on the use of randomization and blinding, sample sizes and dropout rates, length of follow-up, and endpoints. In addition, the trials were randomly identified as being funded by industry, the NIH, or having no external funding.
To the great relief of teachers of evidence-based medicine, who have spent countless hours instructing medical students and house staff in how to evaluate articles in the medical literature, physicians appeared to appropriately assess the trials’ methodological rigor: They were least willing to prescribe drugs tested in low-rigor trials and most willing to prescribe drugs tested in high-rigor trials.
But to the consternation of industry and the Journal’s Editor-in-Chief, who wrote the accompanying editorial, disclosure of industry funding was associated with physicians’ propensity to downgrade the rigor of a trial, their confidence in the results, and their willingness to prescribe the drugs. And physicians had the greatest confidence in NIH-funded trials.
Dr. Drazen writes in his editorial: “We think that decisions about how trials influence practice should be based on the quality of the information conveyed in the full study report.”
I can only partially agree with this statement. Although the full study report is the major consideration in evaluating any study, I believe that physicians were right to be skeptical of industry-funded trials.
Kesselheim and colleagues varied several different aspects of the hypothetical trials’ designs in order to create differences in their rigor. However, a large number of details and decisions are made in the process of designing, conducting, and reporting on a trial, so it would be impossible to capture them all in a scientific abstract. And in truth, it’s even impossible to capture them all in an 8-page article.
Physicians’ experiences with rofecoxib, rosiglitazone, Epogen, gabapentin, olanzapine, and many other pharmaceuticals have taught them the hard lesson that when the stakes are high with billions of dollars at risk, the stakeholders try to cast the drugs in the most positive light possible.
Maybe negative studies are never published. Maybe adverse events are not reported. Maybe a primary outcome is swapped for a secondary outcome that had a larger magnitude of effect. Maybe the comparator is less potent.
But decisions are made that do not obviously affect the apparent rigor of the trial — the trial is still randomized, blinded, with a large sample and long follow-up. So at first glance, it appears to be of high quality. But these decisions distort the trials’ findings and have distorted the medical literature.
While we wait for better evidence in today’s world — which requires trial registration before enrollment, prespecification of primary and secondary outcome measures and safety outcomes, and reporting of results within a year of trial completion — I think physicians are right to remain skeptical.
Are you with me in siding with these skeptics?
4 Responses to “Why I Applaud the Skeptical Physicians”
NEJM Journal Watch — Recent Cardiology Articles- Transcatheter Valve Replacement for Severe Tricuspid Regurgitation
- Time in Blood Pressure Target Range and Outcomes in Hypertension: Real-World Data
- When to Start Anticoagulation After Ischemic Stroke in Patients with Atrial Fibrillation
- Aortic-Valve Replacement vs. Clinical Surveillance for Asymptomatic Severe Aortic Stenosis
- Secondary Hypertension in Young Adults
Great piece Joe! We need to work hard to instill trust. The pharmaceutical industry has many outstanding professionals and have produced many good studies – we need collaborative efforts to ensure that their work is strong and worthy of trust. I have also written a piece on this topic with a suggestion to the pharmaceutical industry to obtain greater trust by more open sharing of data. I placed it on Pharmalot, a location that attracts a lot of industry eyes. If you are interested, here is the link: http://www.pharmalot.com/2012/09/the-op-ed-a-suggestion-to-restore-faith-in-pharma-studies/
You must always consider funding source bias whether the research is industry or government funded. Each has their own agenda and each has reasons to favor certain outcomes. Industry funded research is more likely to be biased towards the profit motive, while government funded studies may be biased towards saving money, or other things that will affect the government or research budget. Some government scientists use their position as a stepping stone to enter industry later. This also can affect the way results are presented. I am not saying anything is wrong with this, on the contrary I am trying to convey the idea that the human nature of the author will always affect the way they present the results, whether consciously or not. We, as physicians, must always keep in mind that the art of medicine is a work in progress; research has inherent flaws which we must consider, and to place good patient care and the best interests of our patients first. Statistics, although they are the best scientific tool we have in interpreting study results have limitations. Good intuition and clinical judgement are as important in understanding research as they are in patient care.
I do believe Dr. Drazen’s publication accepts industry funding and makes a pile of money from reprints. I imagine there are even more industry ties.
By the nature of these relationships, they cannot be managed. They have to cease.
As well said by James Surowiecki, New Yorker, “Transparency is well and good, but accuracy and objectivity are even better. Wall Street {medicine} doesn’t have to keep confessing its sins. It just has to stop committing them.”
http://www.newyorker.com/archive/2002/12/09/021209ta_talk_surowiecki
I very much agree with Dr. DeMedio comment.
I would like to take it just a bit deeper. Just beyond the garden variety skepticism sanctioned by teaching staff of the academic institutions (which essentially excludes them – as they are already “skeptics”). Just beyond NIH sanctioned skepticism of Pharma funded research (which excludes NIH sanctioned research and pushes all the suspicion on Pharma) thus pretending that personal career advancement by using public money, a de facto MO of most government run organization, does not exist.
Study of logic was banned from both grade school and medical school, and for good reason. No clinical research will pass the test of elementary rule of “fallacy of affirming the consequent”.
http://en.wikipedia.org/wiki/Affirming_the_consequent
Instead, we are using statistics (worst of the three type of lies (lies, damn lies and statistics) to convince ourselves and others that what we are being sold by pharma, NIH and academia is trustworthy and actionable – while – let us be honest – it is not. No clinical trial and no Grade A1 evidence can overcome this elementary logical threshold. This is why logic is and should be banned from study in any of the socially responsible medical educational institutions in this country – as it thankfully is.
Thus, is it a surprise then that half of “firmly established truth” in medicine does not survive 5 years? That 2/3 to ¾ of biomedical articles published in most respected journals like Science and Nature end up being a fraud or just simply wrong in 10 years?
Why are we so sure that what comes out of the same institutions now is better that what they produced 10 years ago?
We have very limited and primitive, and mostly wrong ideas about function of human bodies in health and disease, and especially about effect of metabolic poisons (we call them pharmaceuticals).
Scientific theories, according to Carl Popper, are divided into two categories: ones that are already proven to be wrong and ones that are still waiting to be proven to be wrong.
http://en.wikipedia.org/wiki/Karl_Popper
Let us admit it – we are just monkeys with the keyboards….
And – for anyone still hoping for “accuracy and objectivity” in this world – I would like to respond with a quote: “There are no facts, only interpretations” (Nietzsche)
Therefore – if we want to “applaud skeptical physician” – let us take skepticism to the next level – beyond the officially endorsed one…where we can be “skeptical about the skeptics” as well.
Dmitri Vasin,
Founding Member of SCANA (Skeptical Cynics Association of North America)