February 17th, 2011

Unreasonable Expectations for Quality Improvement

At a recent committee meeting, my hospital’s administration announced new quality measures and targets. Striving for top performance, the board of the hospital system set the bar extraordinarily high. The bonuses of senior management are tied to achieving the targets, so the announcement had everyone’s attention.

One target that caught my interest was for achieving a door-to-balloon time of less than 90 minutes in STEMI patients. As an interventional cardiologist who helped to organize the national D2B Alliance for Quality, I have spent some time thinking about door-to-balloon times. The hospital set the target of less than 90 minutes at 96%. I pointed out that this was unreasonable, but I was told that the number came from a payer’s pay-for-performance program and from the board.

If we are going to use statistics to determine pay and recognition, it is important to use the statistics correctly. A quality target is an estimate, and because of random effects that have nothing to do with quality, there is a confidence interval around that estimate that should be factored into the cutoff value. The random effects have different implications for the payer than they do for the payee. If a hospital gets close to the target but falls short of the cutoff, the payer receives effort for quality but pays nothing — not bad. For the payee, however, falling short of the cutoff means putting forth effort and receiving absolutely nothing for it — a demoralizing outcome to say the least. The randomness around the cutoff value is a bad, all-or-nothing bet for the payee but not for the payer. So, setting a reasonable target that accounts for random effects is an important consideration as you attempt to engage the people who actually do the work of achieving quality — and to keep them engaged.

Small numbers at individual hospitals can amplify these random effects. Of the STEMI alerts that are called, some will be for patients who have insignificant lesions or small vessels and who, therefore, do not undergo intervention. Some will have surgical disease. Others will be excluded from the statistics for a variety of reasons such as the need for intubation or a balloon pump. After all of those exclusions, the number of primary STEMI patients at a typical hospital may be around 40 per year. (Regionalization of STEMI care could increase that number, but that is topic for a different commentary.) So with 40 patients per year, if you have a door-to-balloon time of 91 minutes for 2 patients, you fail to reach a target of 96%. Enormous effort could go into creating a first-rate program, all for naught.

But these are really sick patients, you say. Why not set a high bar and intensify efforts? Why not 100%? Because achieving consistent 100% performance is impossible for any hospital. There is too much ambiguity in making the diagnosis. A patient can wander up to the triage desk of your emergency room complaining of back or abdominal pain, then sit for an hour before being brought back to the treatment area, and 30 minutes later, an ECG could show a STEMI. Or a patient with atypical symptoms could have an ECG that is ambiguous, due to left-ventricular hypertrophy or other artifacts. Despite the very best efforts of good caregivers, the diagnosis of some STEMI patients is occasionally delayed. But when we try to explain these difficulties to non-medical administrators, we are often accused of making excuses. Clinical ambiguity is not an excuse — it is an inescapable part of practicing medicine.

Setting an unreasonably high bar can have unintended side effects. One is to create a “hair trigger” for calling STEMI alerts. Hastily rushing patients with ambiguous clinical findings to the lab could be unsafe if the diagnosis is an aortic dissection, pulmonary embolus, or any number of other diagnoses that can sometimes mimic a STEMI. Excessive false-positive STEMI alerts also can erode the staff’s dedication and engagement, which are necessary to maintain a well-functioning system of care. Finally, there are opportunity costs when we apply excessive resources to chase after impossible statistical goals and neglect other areas of need for quality improvement.

Reliable care for STEMI patients remains an important goal. We should indeed set high standards. But we should not set unachievable goals that ignore the play of chance and clinical ambiguity. If we want to create new quality-improvement challenges, we can define new measures, or use composite measures, or all-or-nothing measures, or outcome measures. But ratcheting up the cutoffs to unachievable levels for individual targets is statistically unjustified and should be avoided.

3 Responses to “Unreasonable Expectations for Quality Improvement”

  1. The aberrant element in this construct is to link the target(d2b) to bonus pay for administrators but not include bonus pay for members of the team actually doing the work to achieve the goal. Also, the ‘hair trigger’ phenomenon is real in our system in that the ED MD’s, contracted by the hospital, are held to a bar of ECG to decision time interval to activate the STEMI team(so called door to decision time) that results in excessive false positives. This latter effect creates STEMI team fatigue and skepticism as to the earnestness of the triage process.

  2. John,
    Like always, your points are insightful and incisive. I believe that the improvements in D2B times throughout the country are extraordinary – and the reduction in AMI 30d mortality testifies to the improvements we are making in all aspects of AMI care. It is quite unfortunate that hospital administrators – and society – put so much pressure on numbers that are easily measured, rather than the quality of decision-making, doctor-patient communication and other such equally important aspects of patient care. Measuring performance is clearly a great advance for our profession. However, the excessive emphasis on single aspects of care – especially when they are statistically meaningless – is unfortunate. I hope that you can ‘talk sense’ to your administration. Good luck!!

    Competing interests pertaining specifically to this post, comment, or both:

  3. David Srour, Md says:

    Something else to consider is that the decision to send a patient to the cath lab maybe adversely effected out fear of missing target time. eg, A patient who might benefit from PCI ,who for whatever reason has a delay in dx ,might not be sent for PCI because it would “mess up the numbers”.

    “Not everything that can be counted counts, and not everything that counts can be counted.”

    Albert Einstein