December 8th, 2011
How Randomness Affects Quality of Care
John E Brush, MD
Each month I meet with administrators at my hospital to review the quality of our cardiology program. At several meetings I’ve complained that our performance thresholds are too high and fail to account for the random variation that is a part of everyday medicine. My administrators don’t want excuses, though — they aim for perfection. But a discussion at our last meeting about door-to-balloon times for STEMI patients changed their minds.
Last month a STEMI patient presented to the ER and underwent a timely PCI. During the procedure, a second STEMI patient arrived. Plans to quickly finish and transfer the first patient to the ICU were thwarted because the ICU was full. The team developed a work-around solution to take the first patient back to the ER to accommodate the second patient in the catheterization laboratory. Despite fast and creative action, the second patient had a door-to-balloon time of 118 minutes, falling short of the 90-minute threshold.
So how were we to respond to this apparent lapse in quality? As we discussed the situation in our meeting, the answer became clear: No corrective action was required. The team had acted admirably. The lapse was due not to a systems defect but to the uncommon, unpredictable presentation of two STEMI patients simultaneously. The administrators realized how randomness affects our measured outcomes.
Cognitive psychologists tell us that our minds are hardwired to try to find causes for occurrences. We often jump to erroneous conclusions by ascribing unwarranted explanations to events that happen randomly. Intuition can sometimes be helpful, as I’ve discussed in previous blog posts, but it can also lead us astray. Apparently, we evolved the tendency to jump to conclusions long before statistics and probability theory explained random variation.
The quality gurus Walter Shewhart and W. Edwards Deming recognized this when they described “chance causes” versus “assignable causes” of variation. Shewhart also used another term, “special cause variation,” to describe variation that may stem from system deficiencies. Both men recognized that overreacting to “noise” in measurement could lead to wasted quality-improvement efforts or, worse, poor staff morale and even an atmosphere of fear.
Pay-for-performance plans and co-management agreements often have quality thresholds that fail to build in an allowance for the play of chance. Quality thresholds for door-to-balloon time are sometimes aggressively set at 96%. But most hospitals have only about 50 door-to-balloon opportunities per year. If by chance they miss a few, like in the above example, they fall below the 96% threshold. It’s great to set lofty goals, but when you turn goals into reward thresholds, missing the threshold because of chance only discourages diligent practitioners from trying to provide quality care.
Failure to recognize randomness can also affect individual performance. Experts in any field gain experience by using intuition to recognize regularities in their environment. Use of objective measurement and feedback is ideal, but much of what constitutes experience is gained through subjective self-monitoring and observation over time. I suspect that the experts who seem to “get it” are those who can filter out the noise of random variation and focus on the signal observations, which enables them to gain meaningful experience.
We should remain vigilant for trends and patterns in medicine that provide opportunities to improve the quality of care. But clinical medicine also has a Brownian-type motion that is unpredictable and uncontrollable because of randomness. Knowing that our natural tendency is to jump to conclusions about causation, we should remember to account for chance and the inherent randomness of events as we monitor our individual and institutional performance.
5 Responses to “How Randomness Affects Quality of Care”
NEJM Journal Watch — Recent Cardiology Articles- Sex-Based Disparity in CABG Outcomes: Are We Moving Toward Parity?
- Can Genetic and Social Risk Scores Improve Cardiac Risk Prediction?
- Extensive Ablation vs. Pulmonary Vein Isolation in Persistent Atrial Fibrillation
- Can a Novel Stent Design Improve Long-Term Outcomes of Percutaneous Coronary Intervention?
- Are Angiographically-Derived Coronary Physiology Measurements Ready for Use in the Cath Lab?
How can you argue that the long door to balloon time in this case is not a systems problem? An ideal system would ensure that there is a place for patients to go to avoid clogging the cath lab. The innkeeper always keeps a room open for such occurrences but hospitals choose not to. There is nothing random about this. Chaos theory predicts that like events tend to occur in clumps- not evenly spread out intervals.
Thanks for your comment. I would argue that the occurrence of two STEMI patients presenting almost simultaneously was an unpredictable and uncommon event with no underlying pattern or explanation – i.e. random. The delay in treating the second patient was compounded by another chance and unrelated occurrence of having a full ICU. Expanding the ICU, or building and staffing a second cath lab are options for creating what might be considered a more ideal system, but these options aren’t realistic, and I think would represent an over-reaction to this single occurrence. Our resources would be better spent elsewhere.
I’m not familiar with the Chaos theory that you mentioned, but I am familiar with in interesting study that was performed by Amos Tversky and others. They decided to test the validity of the “hot hand theory” in professional basketball. They were looking for the “clumps” that you mentioned, the apparent streaks in performance by particular players in particular games. They analyzed thousands of sequences and found that there was no causal explanation – the streaks occurred randomly.
I wonder if physicians might be particularly prone to over-generalize about events. After all, we generalize all day long when we use inductive reasoning to place a particular patient in a general diagnostic category. Because generalizing is a necessary part of our daily routine in practice, I think we physicians should be particularly careful not to over-generalize.
We don’t want to commit an alpha error by over-reacting, nor do we want to commit a beta error where we under-react because our scope of observations is too narrow. In between is the “sweet spot” of good judgment.
Competing interests pertaining specifically to this post, comment, or both:
None
agree entirely, john. very simply you can not plan, spend, anticipate, thwart every vicissitude of fate. even if you could imagine all the challenges to complying to a fixed (but arbitrary) door to ballon times it would bankrupt the system and destroy the ability to provide the 95 plus or higher quality compliance. medicine is practiced by humans in an imperfect world. we should be judged not by our innate frailties, but the vigor and success of our attempts to deal with them. medicine and hospitals are businesses, of course resource allocation is the issue involved. that is not a damning comment but a fact. every complication or death after cardiac surgery is a tragic event, but some are just not preventable. we can not be callous and dismissive, but we can not have the hubris to suppose we are infallible.
Competing interests pertaining specifically to this post, comment, or both:
i have made preventable mistakes in 30 years practice of cardiac surgery and have tried to learn from them and others’ mistakes in a continuing attempt to render safe and effective care to patients. and my conscience is mostly clear and i believe my colleagues and patients respected my professional ethics and results. and whom among us as care providers, administrators, regulators,i nsurerers or teachers has not lived in that glass house?
How much “surge capacity” a health facility has is, at the end of the day, a political i.e funding decision. There is no point in financing an elaborate system for something that will hardly ever happen. If the inevitable does occur, a resilient system will usually improvise and extend itself. Sometimes even that is not enough, for instance in mass casualties. Did the 28-minute delay in the case you describe have any detrimental long-term effects on your patient? After all, the 90-minute limit itself is an arbitrary compromise between the ideal and the achievable. Congratulations to whoever thought of taking the patient back to the ER for monitoring, and to the ER for accepting him. If the problem keeps reoccuring, you will probably have to make plans – and divert room, staff and equipment – to unclog the cath lab when ICU is full. Not an easy task in these times.
Of course the evaluation of DTB times must disregard all data that involve two simultaneously arriving infarct patients. The failure to do this indicates a complete lack of understanding of statistical analysis. The DTB time for a second simultaneously arriving infarct patient requires that those patients be evaluated separately. The fault here lies in the ignorance of those people in your hospital or of those who have established the evaluation criteria. The incident you relate is actually one that should be of the evaluation of the quality of those persons who are doing the evaluation and their lack of understanding of basic statistics. What needs to be done is to establish criteria to evaluate the quality of those doing the evaluations as to their fundamental understanding of what they are charged to do. If you don’t understand basic statistical analysis you shouldn’t be doing statistical analysis anymore than than a person not trained to do acute coronary interventions should be doing acute PCI.
Competing interests pertaining specifically to this post, comment, or both:
None, I am retired.