June 5th, 2012
Data, Drugs, and Deception – A True Story
D H Newman, MD
The following guest post by Dr. David Newman is reprinted with permission from his website and blog, Smartem.Org. Dr. Newman is an Emergency Physician and Director of Clinical Research at Mt. Sinai School of Medicine in the Department of Emergency Medicine.
Recently The Lancet published a meta-analysis of 27 statin trials, an attempt to determine whether patients with no history of heart problems benefit from the drugs — true story. The topic is controversial, and no less than six conflicting meta-analyses have been performed — also a true story. But last week’s study claims to show, once and for all, that for these very low risk patients, statins save lives — true story.
Actual true story: the conclusions of this study are neither novel nor valid.
The Lancet meta-analysis, authored by the Cholesterol Treatment Trialists group, examines individual patient data from 27 statin studies. Their findings disagree with an analysis published in 2010 in theArchives of Internal Medicine, and with analyses from two equally respected publications, the Therapeutics Letter and the Cochrane Collaboration.* Despite this history of dueling data the authors of last week’s meta-analysis, in a remarkable break from scientific decorum, conclude their report with a directive for the writers of statin guidelines: the drugs should be broadly recommended based on the new analysis.
As an editorialist points out, if implemented, the CTT group recommendations in the United States would lead to 64 million people, more than half of the population over the age of 35, being started on statin therapy — true story.
Where is the magic, you ask, in this latest effort? What is different? In some ways, nothing. Indeed just a year and a half earlier The Lancet published a meta-analysis of 26 of the same 27 studies, with the same results, by the same authors (true story, and an odd choice on the part of the journal). So the findings aren’t new. They are, however, at odds with other meta-analyses. Why? It is the way they calculated their numbers. This meta-analysis, like the earlier one from the same group, reports outcomes per-cholesterol-reduction. The unit they use is a “1 mmol/L reduction in low density lipoprotein (LDL)”, in common U.S. terms, a roughly 40-point drop in LDL.
That’s the magic: each of the benefits reported in the paper refers to patients with a 40-point cholesterol drop. Voilá. One can immediately see why these numbers would look different than numbers from reviews that asked a more basic question: did people who took statins die less often than people taking a placebo? (The only important question.) Instead, they shifted the data so that their numbers corresponded precisely to patients whose cholesterol responded perfectly.
Patients whose cholesterol drops 40 points are different than others, and not just because their body had an ideal response to the drug. They may also be taking the drug more regularly, and more motivated. Or they may be exercising more, or eating right, and more health conscious than other patients. So it should be no surprise that this analysis comes up with different numbers than a simple comparison of statins versus placebo pills. Ultimately, then, this new information tells us little or nothing about the benefits someone might expect if they take a statin. Instead it tells us the average benefits among those who had a 40-point drop in LDL.
But LDL drop cannot be predicted. Some won’t drop at all, some will drop just a bit, and some may drop more. Therefore the numbers here tell an interesting story about certain patients who took statins, but they have no relevance to patients and doctors considering statins. And yet, the latter group is the target of the study’s concusions.
True story: in prior meta-analyses that found no mortality benefit the investigators simply looked at studies of patients without heart disease and compared mortality between the statin groups and the placebo groups. No machinations, no acrobatics, no per-unit-cholesterol. They took a Joe Friday approach (just the facts, ma’am), and found no mortality benefit.
Perhaps never has a statistical deception been so cleverly buried, in plain sight. The study answers this question: how much did the people who responded well to the drug benefit? This is, by definition, a circular and retrospective question: revisiting old data and re-tailoring the question to arrive at a conclusion. And to be fair they may have answered an interesting, and in some ways contributory, question. However the authors’ conclusions imply that they answered a different, much bigger question. And that is not a true story.
Guideline writers, doctors, patients, journalists, and policy makers will all have to pay close attention to avoid the trappings of deceptive data, dressed up as a true story.
——————–
*The Cochrane Collaboration analysis reports an overall mortality benefit with statins (RR=0.86), however their summary suggests that statins should be used for primary prevention “with caution.” In particular on p.12, after a discussion of the biases in many of the trials that led to their numerical finding, they clearly state that using statins for patients with anything less than a 2% per year risk of coronary events “is not supported by existing evidence.” This cutoff encompasses virtually all people that would be considered candidates for primary prevention.
7 Responses to “Data, Drugs, and Deception – A True Story”
NEJM Journal Watch — Recent Cardiology Articles- Anticoagulation in Patients with Cancer-Associated Pulmonary Embolism: How Long is Long Enough?
- Spironolactone Use After Percutaneous Intervention for Acute Myocardial Infarction
- CRISPR/Cas9 Gene Editing for Transthyretin Cardiomyopathy
- Does Colchicine Reduce Cardiac Events After Myocardial Infarction?
- First-Line Therapy for Ventricular Tachycardia After MI: What's the Best Approach?
Surely the attempts to portray this issue as a yes/no argument are misguided. Interventions such as the use of statins do tend to produce a fairly consistent relative risk reduction no matter what population they refer to. For those groups that have a low absolute risk to start with, it is predictably going to be difficult to demonstrate a clear reduction in absolute risk of CVD events and it is not surprising that different studies and analyses will yield different results. The other paradox is that a small benefit to a large group may theoretically save many events but the benefit to an individual may be tiny and outweighed by cost, inconvenience and potential side effects of therapy. While explaining risk and risk reduction to individual patients in quantifiable terms is challenging, this does need to be addressed honestly rather than adopting blanket policies of “should” or “shouldn’t”.
According to the Lancet paper the benefit for people at low risk concerned non-fatal events only. I am confident that he crucial question for most people is not whether they can avoid a non-fatal event by lowering their cholesterol, but whether they can prolong their life, and whether they can do it without risk of getting serious adverse effects from the treatment.
From table 3 it is possible to calculate that the chance to be alive after five years for people without vascular disease and whose 5-year risk of a major vascular event was lower than 10% was the same, whether they were on statin treatment or not and whether they had a vascular disease or not.
There was a benefit for people whose risk lay between 10% and 20%, but it was trivial at most. Using the figures from the table it is possible to calculate that their chance of being alive after five years was 90.7% against 89.9% for those in the control groups.
Do this small benefit really exceed any known hazards of statin therapy, as the authors wrote? According to an analysis of 225,000 new statin-users treated by British practitioners, Hippisley-Cox and Coupland concluded that more than four per cent run a risk of moderate or serious adverse effects from the muscles, liver, kidneys or eyes (BMJ May 20, 2010). These numbers were even underestimated, because liver damage was recorded only if the alanine transaminase concentration was more than three times higher than the upper limit of normal, and to record muscle problems they required the creatine kinase concentration to be four or more times higher than the upper limit of normal. Furthermore, the authors had not examined the frequency of diabetes, impotency or cerebral symptoms, which have been reported by other authors.
Thank you David ! The lesson I take away is that “statistical deception” is nothing new and is a regular feature in the many trials that exaggerate benefit in relative rather than absolute terms !
Once again we are witnessing an example of surrogate measurement to make predictions of longevity and mortality, without any reference to to the distinctive differences between subcomponents of LDL (heterogeneity in particle size) and their role in risk prediction or risk reduction.
Suggestions that the findings in this study should be incorporated into future guidelines is mindboggling !
Very nice and informative discussions.
Meanwhile, in the opposite direction, I have partly struggled with the conclusions in the Archives Meta-Analysis you referred to. The authors call a risk ratio of 0.91 (95% confidence interval, 0.83-1.01)for reduction of all cause mortality in moderate to high-risk patients as being non-significant.
Statistically, I agree. Yet, realistically, the numbers provided are far more suggestive of statins’ efficacy for moderate to high-risk patients. How much should we read into “1.01” until we have further evidence?
And I wonder how much we should worry from statin side effects. There are a few, very infrequent, serious ones. Elevated liver enzymes or elevated blood sugar levels are merely surrogates -we don’t know clearly yet how they translate into patient-important outcomes among statin users.
I come to this issue as a health reporter, not a physician and my medical expertise is limited. However, I have been looking into the study in question and am troubled by Dr. Newman’s choice of “deception” to describe the meta-analysis. In his blog post, he suggests that something other than the LDL reduction could account for the documented risk reduction, and that somehow we have been duped. He says,
“Patients whose cholesterol drops 40 points are different than others, and not just because their body had an ideal response to the drug. They may also be taking the drug more regularly, and more motivated. Or they may be exercising more, or eating right, and more health conscious than other patients. So it should be no surprise that this analysis comes up with different numbers than a simple comparison of statins versus placebo pills.”
But in a meta-analysis of this type and size, would not the within-group differences be “washed out” by the combining of data? It is perhaps a question for an expert in biostatistics.
What I have noticed in my reporting is that medical opinion on the use of statins in primary prevention seems to break along an ideological divide, with some physicians leaning extremely one direction or the other based on issues other than data and good patient care. If you read the blog, it’s hard not to see the insinuation that we have been duped DELIBERATELY, and the generally high emotional tone. These are “risk factors” for ideological bias, are they not?
Is there something about the statistics in the Lancet study that are incorrect? And when a study produces a result that does not fit our politics, is it fair to cry “deception”? Isn’t this what is going on in the global warming non-debate? I would not wish that time-wasting, pointless political meshegas on medicine.
Yes..intention to treat analysis is the gold standard. But trials are biasrd because they are trials. Adhence is often high, and “run-in phases” are common. Statins are quite safe. The gulf between intention to treat and per protocol analysis seems narrower for these statin trials, compared with the potential gulf between a novel agent and warfarin for example.
An intention to treat analysis may also be biased depending on the potency of the chosen statins in the trials. If a preponderance of patients were in trials with lower potency statins, this could bias results against statins. Analysis by LDL reduction may creat a more even playing field.
At least the study supports a contention that robust LDL reduction is favorable. Neither of the meta-analyses is “right” or “wrong”.
the realproblem with statins and other similar preventive drugs such as antihypertensives is adherence.
Most studies show poor adherence , varing with age, socioeconomic factors, costs, and number of pills per RX.
In fact, adherence to diet and exercise is far from idea.
Walk into any mall and see the obese elderly eating fast foods.
Get back to me if you have a solution.