|
More on Variation—Part 1
In October 2015 the Centers for Medicare & Medicaid Services instituted mandatory reporting of a bundle of steps thought to be associated with best practice for patients with severe sepsis or septic shock, commonly designated SEP-1. The rationale was the high mortality rate and the marked center to center variation in outcomes observed. Applying “best practices” in a uniform way (reducing variation) was thought likely improve outcomes. Now Barbash and associates from the University of Pittsburgh have reported their analysis of treatment patterns and clinical outcomes in their 11 hospitals for septic patients following introduction of SEP-1.[1] They had data on 156,262 patients and compared 29,051 patients before the bundle to 22,759 patients after the bundle (Jan 2016-Dec 2017.) This is a “big data” study, so individual clinical variables should fade in importance. The defined as the four dependent variables measures from SEP-1: appropriate IV antibiotic therapy, initial serum lactate measurement and IV crystalloid administration within three hours of presentation and repeat lactate measurement within six hours. They did patient level adjustments for severity of sepsis, age, and co-morbidities. They found a significant, but small, increase in early antibiotic administration to 49.8% (net gain of 4.7%), a much larger increase in lactate measurements to 70.2%, (net gain of 23.7%), and an increase in large volume fluid administration to 13.2%, a gain of 3.4%. All of these increases were statistically significant, but did not result in a detectable change in sepsis mortality or discharge home alive. Their conclusion: application of the sepsis bundle in their hospitals did not affect mortality. In the accompanying editorial, Klampas and Rhee noted the primary impact of the bundle appeared to have been an increase in serum lactate measurements, but this value is elevated in conditions other than sepsis and measuring it does not lead to clinical actions.[2] After noting other limitations, they suggest the way forward is via improved rapid diagnostic tests that are more specific for sepsis, so inappropriate therapy can be limited and patients can begin treatment as early as possible in hopes this will “rescue” many of them. Someone who thought the guidelines were best practice could note the modest increase in the application of the recommended measures, suggesting general non-adherence to guidelines by professionals. I will defer a discussion of the problems with guidelines for another day, but this is a recent example where serious effort has been devoted to an important clinical problem with minimal results. It may still be true reducing variation is a key component of improving outcomes, but it is certainly true the linkage between process measures and patient outcomes is not as clear and direct as one would wish. All of these issues are examples of the problem of variation in healthcare. Variation is a problem in the sense that individual patients may receive less than optimal care, but the problem is compounded in that we often don’t know what the “correct” answer is. This is illustrated by an experience I had as chief of staff. I presented a blinded data set to one medical department showing five doctors did a particular, but unnamed, procedure in less than 2% of patients, five did the procedure in about 5% of patients, and one did the procedure in about 20% of patients. (Another 10 did not do the procedure at all.) The doctors agreed the best interpretation of the data was there were two schools of thought about using the procedure, but one member was clearly an outlier. Now the data don’t show why that physician was an outlier. He may have had a different population of patients, an argument he certainly made. I suggested that in the absence of published data, the department needed to develop an internal consensus statement about when the procedure was indicated. This “solution” reduces variation, but does not decide which approach is “correct.” They may all may be wrong. Variation is not peculiar to healthcare, but is a general issue with the way the people think, and occurs whenever judgment is needed and the data are fuzzy. Several interesting examples from other fields are presented by Daniel Kahneman and associates in a new book called Noise.[3] They looked at sentencing of convicted defendants by Federal judges, rating decisions by insurance actuaries and claims adjusters, and hiring decisions. In each instance there is considerable variation, which they termed noise. I agree with their premise that reducing noise is desirable to reduce the sense that what happens to you is a lottery, but medicine has an additional problem in that the patient always has a choice. What happens is not just a function of medical decision making, but the interaction of the physician, the patient, the patient’s family, community and so forth. I would point to the issues with mass vaccination against COVID-19 as a case in point. Payers, in particular, have been pushing for reduced variation in order to set better prices, but, as pointed out in the book, they often don’t recognize the noise in their own decisions. It turns out when experienced actuaries and claims adjusters are given a series of standardized cases, the observed intra- and inter-observer variability is strikingly similar to what is seen in medical studies. It turns out we are all human. In situations where the best, or correct, answer is not known and may not be knowable, judgment will be required, hence variability will occur. I suggest the issue we need to study is how much is acceptable? I will consider this issue at length in a subsequent article. The study reported here does not prove guidelines are useless, but it does illustrate how difficult it is to create a guideline which improves patient outcomes. What is needed, in my view, is a more rapid cycle approach. A proposed guideline can be developed and tested in practice over a shorter time, refined, replaced, or scrapped, depending on what happens. Of course, none of these attributes are characteristic of government programs where money is involved. But there are many smaller institutions capable of doing this work. Maybe more research money should be directed toward testing these projects before a Federal mandate is issued. I am encouraged there seems to be some movement in this direction.[4] 22 July 2021 [1] Barbash IJ, Davis BS, Yabes JG, Seymour CW, Angus DC, and Kahn JM. Treatment Patterns and Clinical Outcomes After the Introduction of the Medicare Sepsis Performance Measure (SEP-1). Ann Intern Med 2021;174(Jul):927-935. doi: 10.7326/M20-5043. [2] Klompas M, Rhee C. Has the Medicare Sepsis Performance Measure (SEP-1) Catalyzed Better Outcomes for Patients with Sepsis. Ann Intern Med 2021;174(Jul):101-1011. doi: 10.7326/M21-1571. [3] Kahneman D, Sibony O, Sunstein CR. Noise: A Flaw in Human Judgment. (New York: Little Brown Spark, 2021.) [4] Centor RM, Klompas M, Rhee C. Annals on Call—Does the Sepsis Bundle Improve Outcomes? June 2021. Accessed 22 July 2021at https://www.acpjournals.org/doi/10.7326/A20-0015. |
Further Reading
Measurement Error Measurement error is recognized in the laboratory, but not in US healthcare, which is causing problems. More on Biases A recent series of articles in the New England Journal of Medicine provide more insight into the issue of bias in medical decision making. Performance Measurement An expert panel has concluded less than half of current measures used by CMS to assess value for primary care services are valid. What does this tell us about current pay-for-performance efforts? Reliability or Dependability Is reliability too narrow a goal? Shouldn't we strive to be dependable, with its connotation of both reliability and trustworthiness? Standardization Versus Innovation Two recent articles present starkly different approaches for healthcare organizations dealing with the stresses of the pandemic and healthcare reform. The Anchoring Heuristic Businessmen and health policy experts fail to recognize the limits imposed by the experiential nature of medical practice, both of which impact achieving the "triple aim." Uncertainty Dealing with uncertainty is at the core of practicing medicine. Have we tried to escape this reality? Variation in Health Care Is variation in health care good, bad, or inevitable? The answer may determine future medical practice. |