November 18th, 2013

Controversy Erupts Over Accuracy of Cardiovascular Risk Calculator for Guidelines

In the face of a highly critical story in the New York Times by Gina Kolata about the new cardiovascular guidelines, authors of the guidelines and leaders of the American Heart Association (AHA) and the American College of Cardiology (ACC) defended the value and integrity of the guidelines.

The Times story claims that the cardiovascular risk calculator used to assess individual risk in the new guidelines is deeply flawed: “In a major embarrassment to the health groups, the calculator appears to greatly overestimate risk, so much so that it could mistakenly suggest that millions more people are candidates for statin drugs.” The story quotes former ACC president Steve Nissen: “It’s stunning. We need a pause to further evaluate this approach before it is implemented on a widespread basis.”

But the guideline authors and AHA/ACC officials strenuously defended the guideline at a news conference Monday morning at the AHA meeting in Dallas. They said the new risk calculator is far superior to previous efforts, incorporating far more data that now includes stroke assessment and, for the first time, provides specific predictions for African Americans.

The assembled officials rejected any call to delay implantation of the guidelines, but one author, Donald Lloyd-Jones, said that “over time we will modify the risk scores so that they get better and better.”

“We think we’ve done our due diligence,” said AHA president Mariell Jessup. “We have faith and trust in the people who developed the guidelines.”

The Times story is based on a commentary scheduled for publication in the Lancet on Tuesday that is highly critical of the new calculator. The paper, written by Harvard Medical School’s Paul Ridker and Marcia Cook, states that the calculator overpredicts risk by 75% to 150%. At a hastily assembled meeting on Saturday night at the AHA meeting in Dallas, Ridker presented his findings to leaders of the AHA and ACC.

One source of confusion is that Ridker and Cook had sent their criticism to the National Institutes of Health’s National Heart, Lung, and Blood Institute (NHLBI) a year ago, but, according to Kolata, their concerns were never passed on to the AHA and the ACC. At the Saturday night meeting Ridker presented data showing that the risk calculator calculated much higher risk for the population of people who had been followed in three large studies: the Women’s Health Study, the Women’s Health Initiative, and the Physician’s Health Study. Kolata writes:

“…the calculator overpredicted risk by 75 to 150 percent, depending on the population. A man whose risk was 4 percent, for example, might show up as having an 8 percent risk. With a 4 percent risk, he would not warrant treatment — the guidelines that say treatment is advised for those with at least a 7.5 percent risk and that treatment can be considered for those whose risk is 5 percent.”

In a draft of the Lancet article I obtained, Ridker and Cook write:

“…it is possible that as many as 40 to 50 percent of the 33 million middle-aged Americans targeted by the new ACC/AHA guidelines for statin therapy do not actually have risk thresholds exceeding the 7.5 percent level suggested for treatment. Miscalibration to this extent should be reconciled and addressed in additional external validation cohorts before these new prediction models are widely implemented.”

But, according to the guideline authors, Ridker’s position is not new and is discussed in the guidelines. The people included in the three trials presented by Ridker are unusually healthy and are “not really representative of the broad risk,” said Lloyd-Jones. “So it doesn’t surprise me at all that our equations overestimate risk in these groups.” He also said that he thought the overestimation of risk was largely because patients with the highest risk were already taking risk-reducing statins.

AHA president Mariell Jessup said that she wished “there had been this much attention in the past when we used prior risk calculators. If you were an African American and had a stroke” the guidelines didn’t help.

Lipid panel co-chair Neil Stone said that  7.5% is not “an absolute cutoff. This is not the end of the discussion, it’s the start of the discussion.”

Press Conference

7 Responses to “Controversy Erupts Over Accuracy of Cardiovascular Risk Calculator for Guidelines”

  1. So here’s the bottom line according to the risk calculator…..

    Every white male over the age of 62 with optimal risk factor levels has a 10 yr risk of 7.5% or more

    Every white female over the age of 70 with optimal risk factor levels has a 10 yr risk 7.5% or more.

    Every AA male over the age of 65 with optimal risk factor levels has a 10 yr risk 7.5% or more.

    Every AA female over the age of 69 with optimal risk factor levels has a 10 yr risk of 7.5% or more.

  2. carol vassar, MD says:

    Does Ridker have any conflict of interest on this point that complicates interpretation of his objections?

  3. The ability of any risk assessment strategy to correctly identify individuals at risk depends upon the prevalence of disease in the population that is being tested.

    It is important to note that none of these risk scores (including Framingham and Reynolds) has ever been shown in a randomized control trial to reduce CV events.

    So if you select a low risk trial population you will overestimate the risk.

  4. I think the focus has been excessively directed towards the calculator when it is the lack of targets that is really the big deal in these guidelines (and is a good thing).

  5. Uffe Ravnskov, MD, PhD says:

    The risk calculator is not the only problem with the new guidelines; there are two other major problems as well. One of them is relevant for all previous guidelines.

    During all the years cholesterol-lowering treatment has been recommended for both sexes. At a conference in1992 arranged by the National Heart, Lung, and Blood Institute the results from follow-up studies including 124,814 women showed that high cholesterol was not a risk factor for women, neither for cardiovascular or total mortality (1), and no statin trial has ever succeeded in prolonging the life for women. Why should women take statins?

    It is well-known that diabetes is a side effect of statin treatment. In the Women´s Health Initiative for instance more than 3 % suffered from diabetes after three years treatment (2). According to the new guidelines this is no reason for stopping statin treatment. The consequences of diabetes are serious. It is possible that the benefit as regards cardiovascular events is larger than 3% but what happens after ten or twenty years of statin treatment?. How many have become blind? How many have ended on dialysis treatment? How many have lost a leg or two? We do not know.

    1. Jacobs et al. Circulation. 1992;86(3):1046-60.
    2. Culver et al. Arch Intern Med. 2012;172(2):144-152.

  6. Siqin Ye, MD says:

    My understanding is that although statin may increase incidence of diabetes slightly, in terms of hard outcomes this effect was either non-significant or favors statin use at least in the medium term.

    An analysis of this in Jupiter:
    Another analysis of this in a Taiwanese cohort:

  7. Sergio Kaiser, M.D., MSc, PhD, F.A.C.C. says:

    Let me share with my colleagues what I suppose to be a relevant concern: CAC score is ideally suited for patients with an intermediate probability of CAD. If one assumes that statins should be offered to every patient with an absolute 10-year risk of CVD above 7.5%, for which situation should a CAC score be recommended? For educative purposes? And we do not 3ven have the evidence that supports event reduction achieved by CAC – dri en therapy…