For more than a decade, cardiac stem cell therapy has attracted an enormous amount of attention, promise, and research dollars. Now an original and important new study published in BMJ finds that many of the most promising results in the field are illusory and that the potential benefits of stem cells to treat heart disease are probably far more modest than we’ve been led to believe. The study also raises disturbing questions about ethics and research conduct (and misconduct) in a high-flying field.
Researchers in the U.K., led by Darrel Francis, closely scrutinized 133 reports from 49 different clinical trials testing autologous bone marrow stem cells in patients with heart disease. They found an astonishingly large number of discrepancies in the reports — altogether more than 600 discrepancies, ranging from minor oversights to serious unexplained errors and apparent deceptions. Many errors were mathematical or statistical errors, while others were more general, such as conflicting descriptions of studies as either a prospective randomized trial or a retrospective observational study.
The key finding of the BMJ study is that there was a strong correlation between the number of discrepancies in a study and the reported improvement in heart function as measured by left ventricular ejection fraction (LVEF). The five trials with no discrepancies reported no improvement in LVEF (-0.4%). In stark contrast, the five trials with the highest number of discrepancies — each with more than 30 discrepancies — reported a very large and, if true, clinically significant improvement in LVEF (+7.7%). This effect was consistent: the more errors there were in the study, the more likely the study reported a large treatment effect.
The authors summarized their finding:
Our study shows that scientists who achieve progressively better consistency of reporting find progressively smaller effects on ejection fraction of treatment with stem cells derived from bone marrow. In trials with a discrepancy count of zero, the ejection fraction effect seems to be zero.
The authors said they were unable to obtain explanations for the discrepancies: “We do not know the cause of the discrepancies. We have asked for resolution of over 150 discrepancies through journals. None were resolved, although we found it triggered correspondence from lawyers.”
By sheer coincidence, the BMJ publication occurs simultaneously with the publication of a Cochrane review of stem cells for heart disease. The two papers don’t completely overlap, but they are in many ways congruent. Analyzing the literature, the Cochrane reviewers found “some evidence that stem cell treatment may be of benefit.” But, they noted, “the quality of the evidence is relatively low because there were few deaths and hospital readmissions in the studies, and individual study results varied. Further research involving a large number of participants is required to confirm these results.”
Similarly, the BMJ authors note that “viewing all the studies together as a single entity, there is on average a positive effect.” But, they warn, “averaging effect size across all studies might therefore not be wise because it does not reflect their varying factual accuracy.” Although Cochrane is known for its rigorous methodology, it is generally not within the purview of Cochrane reviewers to seek out errors and inconsistencies in individual studies.
You can follow any responses to this entry through the RSS 2.0 feed.
Both comments and pings are currently closed.
NEJM Journal Watch — Recent Cardiology Articles
Amidst recent scandals of retractions related to research misconduct and controversies regarding conflicts of interest, the stem cell house of cards is being shaken to its core.
The BMJ report raises a few questions:
Q1. Are such discrepancies unique to the field?
I don’t think so. Many ‘fledgling’ disciplines will likely expose their warts if subject to similar scrutiny. Science isn’t immune to fraud despite checks and balances. Nonetheless, it is important to point out these flaws to ensure reliable and valid research.
Q2. Are the discrepancies serious enough to invalidate science behind the stem cell research?
This is difficult to judge, but sufficient questions and doubts have been raised by this report that require independent (and expeditious) adjudication by the key stakeholders – journal editors, professional societies, the involved institutions, and the oversight agencies. As they say, “Sunlight is the best disinfectant”.
Q3. Are these discrepancies easy to pick up in the original published reports?
No which is all the more reason for journal editors to be transparent in addressing and allaying the concerns raised by such reports post haste. The responsibility of the journal editors to disseminate reliable and valid research does not end with peer review and editorial oversight.
Q4. Are the expectations of stem cell research aligned with facts?
No. Fueled by hype and hope, the expectations are unrealistically high, rendering the field vulnerable to errors of omission and commission. Progress in science is often incremental along a continuum and occurs at the so-called ‘speed of science’. Thoughtful reflection rather than research rhetoric is required to ‘unblur’ the line between hype, hope and reality and recalibrate our expectations.