CardioExchange Archive

David J. Cohen, the principal investigator of an observational PCI registry study of drug-eluting versus bare-metal stents, sheds light on a risk-adjustment technique called “instrumental variable analysis.” CardioExchange welcomes your thoughts on the value of this method and on the study it was used to elucidate.

The Study

Using data from a prospective observational PCI registry, researchers compared clinical outcomes of drug-eluting stent (DES) recipients with those of bare-metal stent (BMS) recipients. Three techniques were used for risk adjustment: multivariable regression, propensity matching, and a relatively new method called instrumental variable analysis.

All three risk-adjustment techniques showed a significant advantage of DES over BMS with respect to target-lesion revascularization. However, with respect to mortality, multivariable regression and propensity matching showed a significant advantage of DES over BMS — but instrumental variable analysis did not.

What Is Instrumental Variable Analysis?

The researchers describe instrumental variable (IV) analysis as follows:

“compar[ing] patient groups that differ in the likelihood of receiving a treatment, determined by a randomly distributed ‘IV,’ rather than comparing patients with respect to the actual treatment received (which may be biased). An IV is an observable factor that is associated with a specific treatment pattern but is otherwise unrelated to underlying patient characteristics and does not directly affect the outcome of interest.”

In this study, the instrumental variable was defined as the period of enrollment in the PCI registry (2004–2006 vs. 2007).

The Expert Responds

Q: Should instrumental variable analysis become a standard element in observational studies, alongside multivariable regression and propensity matching? Can consumers of observational research, particularly clinicians in practice, trust the value of observational data if all three types of analysis are not offered by the researchers?

David J. Cohen: It is probably premature to “require” instrumental variable analysis alongside all observational studies, for several reasons. First, it may not be possible, for all analyses, to identify an appropriate instrument – i.e., one that is associated with the treatment of interest but is otherwise unrelated to patient characteristics or to the outcome of interest. Second, IV analysis and more-standard risk-adjustment methods address different questions. Standard risk adjustment attempts to replicate the question addressed by a randomized clinical trial but outside of the experimental setting — in other words, to determine the relative difference in outcomes between 2 alternative treatments (or treatment strategies) for an “average” individual. That is more or less what we as clinicians want to know. In contrast, IV analysis (defined strictly) examines the treatment difference among “marginal patients” – i.e., the subset of patients whose treatment differs according to the instrument. (In our study, the marginal subset was the 20% of patients who would have received a DES from 2004 to 2006, but a BMS in 2007.)

In that sense, IV analysis is a way of understanding the impact of different rates of a given therapy and is, therefore, more directly related to health policy than to clinical decision making. Nonetheless, we often extrapolate these estimates of the marginal treatment effect to the overall population. As noted in our study, IV analysis can be useful — despite these issues with interpretation — primarily because it should be less subject to confounding than standard risk adjustment, assuming the researchers have identified an appropriate instrument.

Given these considerations, requiring all 3 types of analyses for all observational studies would not be appropriate. Nonetheless, if all 3 can be provided and the results are concordant, my faith in the observational data would be strengthened. If the results are discordant, however, multiple issues may be at play and interpretation is far more challenging.

Q: How can readers of research assess whether the authors chose the right “instrument” for an instrumental analysis?

Cohen: In our study, the instrument was the time period of the stent procedure. Other instruments used in the medical literature include day of the week (e.g., weekend vs. weekday) or the distance from an individual’s home to a specific type of hospital. I often think of these instruments as “natural experiments” occurring within an observational dataset that are beyond a clinician’s direct control.

For the reader who is trying to evaluate an IV analysis, the first key question is whether the instrument is correlated with the exposure of interest – and to what degree. It is a fairly straightforward determination: A stronger instrument provides greater separation between the treatment patterns and thus greater statistical power, whereas a weak instrument that provides relatively little treatment separation may offer very little power, even in an extremely large dataset.  Similarly, one can readily test whether the IV is related to observed patient characteristics (it should not be). But the requirement that the IV be unrelated to unobserved patient or treatment characteristics cannot be directly assessed (given that the covariates are, by definition, unobserved) – it can be inferred only if you know the specific circumstances of the study population. The reader must use his or her own judgment to make this critical determination.

Q: According to this study’s instrumental analysis, there was no mortality difference with drug-eluting stents versus bare-metal stents. Does that finding change the bottom line for clinicians on what type of stent to choose for their patients?

Cohen: I do not think that our study will change the bottom line on stent selection for most clinicians, mainly because I don’t believe that many clinicians ever believed the observational studies that suggested a mortality reduction with DES.  There is simply too much RCT data showing no mortality difference between DES and BMS recipients. The goal of our study was to point out the major challenges in understanding and interpreting the large amount of published observational data — in particular, to raise the bar for such studies if we are to use them to inform treatment or policy decisions under the rubric of “comparative effectiveness.”

Share your thoughts about instrumental variable analysis and this stenting study here on CardioExchange.