May 21st, 2014
The Risk-Prediction Conundrum: Individual Risk vs. Population Risk
John W McEvoy, MB BCh BAO
CardioExchange’s John Ryan interviews John W. McEvoy regarding his recent review article, published in the American Journal of Cardiology, about how to interpret cardiac risk for an individual patient when risk estimates are more accurate for populations of patients.
Ryan: How do you explain the difficulties of risk prediction with patients?
McEvoy: This is a difficult but necessary exercise. So much of what we do in medicine is based on risk. Indeed, fully informing patients about the risks and benefits of their care depends on having at least a simple understanding of risk. In addition, putting risk scores aside, knowledge derived from randomized controlled trials regarding best practices in medical therapeutics is heavily influenced by the notion of risk. For example, randomized studies have shown that ACE inhibitors reduce the incidence of cardiac events among post-MI patients, but many individuals in the control arms of these studies did not experience events — and some in the treatment arm did. It is sobering to recognize how much of what we do is based on probability and uncertainty.
One of the reasons my coauthors and I were motivated to write the review article is that the concept of individual risk applied — truly applied — to any given person is an oxymoron. Risk for an individual is like a square peg for a round hole. We can never know, or estimate, one person’s risk. In fact, if you do the math, the confidence interval for a given risk estimate in one person would range from a 0% to a 100% chance of a cardiac event. Thus, risk is not “personalized” and I think of risk as a “group-phenomenon.” I am sure that many other physicians do, too.
This may not seem new or too important to some CardioExchange readers, but to me it does say one important thing: I cannot present risk to my individual patients as “your” risk. Thus, I never say, “Mr. Jones, your risk of a heart attack is 15% over the next 10 years,” because I cannot vouch for the statement’s accuracy. Instead, I present the concept this way: “Mr. Jones, if I were to take 100 patients exactly like you, 15 of them, on average, would have a heart attack in 10 years.” Perhaps this approach is too pedantic, but I happen to think it is a more honest message. Along those lines, I highly recommend the Mayo clinic’s statin treatment decision app (statindecisionaid.mayoclinic.org ). It is a user-friendly, simple tool that helps to inform patients about this important matter, especially regarding risk as a group phenomenon.
It’s debatable whether the notion that individual risk is an oxymoron goes any further than this simple change in how I choose to present risk to my patients. Ultimately, I cannot say that the individual-risk problem is an issue when it comes to allocating therapies. As doctors, we all have more than one patient. Over the daily, weekly, and yearly course of our work, we participate in decisions for large groups of patients. Thus, the use of risk to guide therapeutic decisions will, on average, be accurate for our entire group of patients.
Ryan: With the Schrödinger cat comparison in your article, I think you have to look at the experiment from the point of view of the cat and, in this case, the patient. The issue is not whether disease is present but, rather, if the patient is at risk for a cardiac event and what that means to the patient. Is that a fair critique?
McEvoy: Maybe I should answer this question with two other questions: What puts the patient at risk for the cardiac event that he or she fears? Can the patient be at risk for a cardiac event if he or she does not have the disease (atherosclerosis)?
I would venture that the answer to the first question is “disease” and that the answer to the second question is “no.” Therefore, I think that understanding disease burden, assuming it is actually knowable for a given patient, can be useful. If the patient really wants to know (without a doubt) about his or her cardiac status, I can noninvasively measure the disease with good accuracy (e.g., with coronary artery calcium [CAC]). However, I can never estimate that person’s individual risk to as high a level of certainty.
Again, this is mostly pedantic thinking, and I don’t usually let this influence my management — unless I think a given patient could be an outlier and the risk-prediction algorithm puts that patient in the wrong risk group (e.g., because he or she has a strong family history, a type A personality, even an earlobe crease!).
Some critics would say that everyone has some atherosclerosis, so knowing it is present is not useful. However, the extent and burden of disease varies widely, and we know from CAC studies that patients with a low burden of disease (atherosclerosis), as reflected by zero CAC, have exceedingly low event rates. Thus, if I want to be certain about something (or if something about the patient makes me think he or she is a risk outlier), then I know I can at least be certain about his or her presence and extent of disease (by getting a CAC score). This contrasts, if you ask me, with the inherent uncertainty of a risk estimate for a given person.
Again, this underlying reality does not affect my usual day-to-day care, but I do keep it in mind if something about a patient is worrying me. Unfortunately, as a cardiology community, we do not know whether being certain about the presence or absence of disease (e.g., through CAC testing) can influence outcomes as part of a therapeutic strategy. Thus, I think we really need a CAC trial to be supported and conducted, if for no other reason than to build a better evidence base for a test that physicians are increasingly ordering and patients are increasingly seeking.
Our review paper was meant as an interesting platform for this discussion. The physics analogies will not suit everyone’s taste, but I also know that many readers really enjoyed them.
7 Responses to “The Risk-Prediction Conundrum: Individual Risk vs. Population Risk”
NEJM Journal Watch — Recent Cardiology Articles- Sex-Based Disparity in CABG Outcomes: Are We Moving Toward Parity?
- Can Genetic and Social Risk Scores Improve Cardiac Risk Prediction?
- Extensive Ablation vs. Pulmonary Vein Isolation in Persistent Atrial Fibrillation
- Can a Novel Stent Design Improve Long-Term Outcomes of Percutaneous Coronary Intervention?
- Are Angiographically-Derived Coronary Physiology Measurements Ready for Use in the Cath Lab?
The radiation exposure and the cost make the CAC testing less viable than some other test, such as the hcCRP, though that has its own problems. But the unknown risk of taking a statin for 20, 30 or 40 years, as well as the cost, may balance out the risk and cost of the CAC. Some calculations could support the need for the study.
Regarding applying calculations of risk based on the presence of risk factors, wouldn’t you want to allow that the 100 people put into the group were NOT identical to him. That group would be of a wide variety of people who shared the characteristics considered in the calculation of risk. There would be no one else quite like him in the group. That would reinforce the idea of population risk estimates. The public has been hearing about such risks for a long time. Even if we used the CAC to help refine our risk estimates, wouldn’t we still be doing population risk estimates.
In my own opinion, when people take a ‘preventative’ medicine e.g. a statin, or an ACE-inhibitor, they are not that bothered about what they may die of, but when they may die. Or, to put this another way. How much longer are they likely to live (We we know, no drug can ‘prevent’ death, only extend life). Would it not be more important to be able to say to the patient. If you take this drug for x years, you are likely to live – on average – y more days/weeks/months/years. Is this not the most important outcome measure
There is a simple way to inform the patient. Present a chance prediction instead of a risk prediction. Using the figures in analysis 1.4. in the Cochrane meta-analysis of the primary-preventive statin trials from 2013 it is possible to calculate that the chance of not having a non-fatal heart event during the following 3-4 years is about 97%. Tell the patients that they can increase the chance to about 98% by taking a statin drug every day, but don´t forget to mention the possible side effects and also that many non-fatal heart events may heal with little future discomfort or none at all. The chance is of course a little lower for old patients but the benefit does not differ very much. And do not forget to tell the patients that no primary-preventive statin trial has succeeded in prolonging the life for the participants, and that no trial, neither primary or secondary preventive, has succeeded with that for women.
McEvoy is to be congratulated for raising this essential issue. Measures of risk based on one or more variables apply to the group. Ravnskov’s way of looking at this is the only real way of doing so, but this too is for the group, as he elaborates. Even our measures for the group are given meaning for us based on a statistical probability; an arbitrary number that would not survive testing of that probability. It is bad enough that we delude ourselves that we know when we don’t. We should also not pretend to our patients that we know for a fact what is in their best interests. Our “science” on risk is helpful to epidemiologists, who designed it for their purposes. It is not at all clear that it is helpful to the medical practitioner or to the patient. It is high time to take stock of what we have done and consider what else we must do to be better than we can be with what we have.
My thanks to all who commented on this. This type of discussion, inclusive of valuable opinions, diverse perspectives, and a wealth of clinical experience is a real strength of the cardioexchange forum.
However, I would like to refocus the conversation a little, if possible. This piece is not about the pros and cons of statins in primary prevention. Personally, I feel there is a very strong evidence base for statins in targeted higher-risk primary prevention individuals. Recent data, including those from the Cochrane Collaboration, support this. However, I am not trying to force this belief on any doubters or make this the focus.
Rather, this conversation relates to a more fundamental concept, that of risk. The primary focus on the patient-physician conversation in the new guidelines requires that we now explain risk accurately to patients. This was the motivation for the piece. I have been humbled to recognize how difficult it is to truly know an individual person’s risk. Our AJC review explores many reasons for this, incorporating analogies from other fields of science that also depend on risk and probability.
This uncertainty about individual risk required a simple change in how I personally think about risk and, more importantly, how I explain it to patients. Risk estimates are accurate for groups but are not personalized and the confidence interval is extremely wide for a given individual. I am concerned that some providers may not recognize this limitation and could over-rely on risk-scores, obtaining a false sense of security, in some persons who may be ‘outliers’ in the risk model (where the model does not ‘fit’ and underestimates their personal risk situation). Admittedly, this is concept is not new, indeed it gets a brief disclaimer in the guidelines, but it appears to be under-recognized in general practice.
I would be interested to hear how other physicians describe risk to their patients
Finally, I generally endorse both the risk assessment and cholesterol treatment guidelines, both landmark documents, simply because I do not have just one patient, and for groups of patients I know that by following the guidelines I will get things right on average. However, by not over-relying on risk estimates, and by using my clinical acumen when necessary, I hope to get things right more often than on average.
I really enjoyed your recent article.
You mentioned in your last comment that the confidence interval of the risk estimate is extremely wide for a given individual. I’m not sure that it even makes sense to talk about a confidence interval around the calculated risk estimate of an individual.
The calculator itself has no variabity. It will give you the same result with the same input variables, time and time again. It has a confidence interval of zero.
The model upon which the calculator is based probably has a confidence interval, but how would you calculate it? I suppose a statistician could somehow combine the confidence intervals associated with all the beta coefficients that go into the model to give a cumulative confidence interval, but that is way over my head.
But even the beta coefficients don’t have one confidence interval. The beta coefficients give a relationship between a unit change in a covariant and it’s affect on the overall model. But within each cohort, the confidence interval for the beta coefficient will vary over the range of the covariant. There are data sparce and data dense areas within the cohort. In data sparce areas, the confidence interval for the beta coefficient would be much broader.
So I’m not sure that it makes sense to think about a confidence interval around the calculated ASCVD risk. It is only an estimate. It is a number that provides a starting point for a discussion with the patient about risk factors and statin treatment. The number only makes sense when it is put into context. How does an individual’s estimate compare and is the number high enough to warrant a statin?
I totally agree with your point in your article. Estimating individual risk is a difficult thing to contemplate.
Thanks John. Your comments are very well considered.
Like you said, the calculator gives the same point estimate from the model each time you enter the same input variables. There is no variation in that.
However, as you also astutely infer, the model itself could be used to yield confidence intervals for these point estimates. The calculation is complex and I believe requires the raw data and variance-covariance matrices. I have discussed this with senior bio-statisticians and am told it can be done though.
I think there is some value in thinking, at least for theoretical purposes, about confidence intervals in this situation. Sure, the risk score is only an estimate, but it is an estimate based only on your study sample and may not be true for the underlying population. If you base the decision not to treat with statins on an estimate of 7.0%, wouldn’t one be interested to know the actual range that covers the true risk 95% of the time? (especially if the upper end of this range goes well past the treatment threshold of 7.5%)
As an aside, another thing we are interested in is looking at natural variation in the risk prediction estimates from the model. For example, we are interested in repeating lipids and BP 24 hours after the initial entry of the input variables into the risk model. This is another way to think about variability that is less statistical.
That said, I am not trying to direct the conversation to the technicalities of confidence limits, rather I was using this idea of extremely wide limits simply to underscore the fact that individual risk is, as you say, indeed a difficult thing to contemplate.
Many thanks.