More on Variation, Part 2
 
            Those of us who have studied statistical process control are familiar with common-cause and special-cause variation. In their book Noise: A Flaw in Human Judgment,[1] Kahneman and associates make a distinction between bias and noise whenever expert judgment is called for. They define bias as a situation when most errors in a set of judgments are in the same direction, meaning bias is the average error. Other variation the authors call “noise.” In their view noise is quantitatively more important, and more insidious as we are often not aware of noise. “The surprises that motivated this book are the sheer magnitude of system noise and the amount of damage it does.” (p. 365).
            System noise is broken down into level noise and pattern noise. Level noise is defined as the variability of the average judgment made by different individuals. I suspect this is what we commonly examine when doing continuous quality improvement efforts. The other sort of noise they label pattern noise. In a medical context, this might be the distinction between “cowboys and comforters.” On average the cowboy is prone to recommend aggressive therapy whereas the comforter is more likely to recommend a less aggressive or palliative approach. Of course, the “correct” answer is not knowable, so the term here should not be thought of as pejorative—it reflects the fact that human judges see the world differently. However, pattern noise also has a transient component they call pattern noise. There are good data that people decide the same issue differently depending upon all sorts of things that are extraneous to the situation. In a medical context, this has been studied by looking at things like the effect of time of day, sleep deprivation, and the impact of the preceding case/s.
           
       “Noise is not a prominent problem. It is rarely discussed, and it is certainly less salient than bias. You probably had not given it much thought. Given its importance, the obscurity of noise is an interesting phenomenon in and of itself. Cognitive biases and other emotional or motivated distortions of thinking are often used as explanations for poor judgments…Bias has a kind of explanatory charisma, which noise lacks. If we try to explain, in hindsight, why a particular decision was wrong, we will easily find bias and never noise. Only a statistical view of the world enables us to see noise…
       Another reason is that professionals seldom see a need to confront noise in their own judgments and in those of their colleagues. They expect that colleagues would agree with them, and they never find out whether they actually do.”
 
            The authors have a number of recommendations on how to reduce noise in decision-making (judgment) in both routine and one-off situations. I recommend their book to your review to understand their argument. Here I want to look at the extreme instances of variation—malpractice and disciplinary actions.
            Most physicians have a dread of being named in a malpractice suit and many try to reduce the odds by practicing “defensive medicine.” One problem is that we all know errors are common and that at least half of what we do is wrong or not supported by evidence. In the first part of this series, I discussed recent observations of the SEP-1 guidelines for identifying and managing suspected sepsis and the disappointing results in the study reported. This failure does not mean the effort was wrong, but it does mean it cost a great deal of time and effort and did not work, so the guideline writers will have to go back to the drawing board.
It also reflects the uncertain nature of medical practice. This means that almost any decision or series of decisions can be criticized in retrospect. The question “why did you not do x” is not answerable. However, malpractice suits usually require an irritated patient or family member in addition to errors, hence the rediscovery of the importance of bedside manner. As hospital medicine, in particular, has become more impersonal and corporate, bedside manner has come to mean more than just the doctor—it is also a system attribute, and many large organizations have come to realize how difficult “guest excellence” is to create and maintain.
            Actions against a physician’s hospital privileges also tend to focus on a “sentinel event,” an exceptionally bad outcome. Having been through a number of these investigations, I have noted most physicians, when judging colleagues, look to see if the clinical logic is sound and documented. If the logic is sound, the tendency is to note the bad outcome, but not to be punitive. If the logic is not apparent, or there is a clear lapse of judgment, the doctors tend to be less lenient.
            On the other hand, we have not made as much progress in taking a statistical approach to decision-making. In the first part of this series, I mentioned a procedure that showed one physician was an outlier. What if that physician, when shown the data and the consensus guideline written by his local colleagues, fails to moderate his practice pattern? In theory, it should be easier to make a decision to restrict privileges in this case, but in practice it is not. Attorneys will assure you hospital privileges are a property right, and are protected as such, and the law tends to look for specific cases of malfeasance, not statistical patterns of behavior. The tension between due process for the individual physician and the desire to reduce unexplained variation is not easily resolved.
            Hence, we have come full circle. The authors of Noise believe it is an important, but largely unrecognized source of errors, even when the “truth” is not knowable. Yet the forces of resistance are powerful. Many years ago, I recognized that physicians understand Gaussian distribution, even when they can’t do the math, because they all got into and out of medical school on the curve. My rule of medical management has been that if you show a physician good data that he/she is an outlier, he/she will fix the problem and move to the middle before you can figure out what the problem really is. Failure to do so is, in my cynical view, is evidence of a personality disorder, and they are not fixable. The only way to deal with such issues is to separate them from the organization. But that is easier said than actually accomplished.
            As for the problem of noise, perhaps my advice to the medical young remains pertinent. We do what we do and sometimes it works and sometimes it does not. If we don’t take too much credit when it works, we won’t take quite as much blame when it does not. Also, being open minded helps—if what you decided isn’t working, try something else, and you can always ask for help from colleagues—they might have a better idea.
 
9 August 2021


[1] Kahneman D, Sibony R, Sunstein CR. Noise: A Flaw in Human Judgment. (New York: Little Brown Spack, 2021.)

Further Reading

Actionable Data
Medical organizations have a lot of data, much of which is not "actionable." However, if taken as a vital sign, such data can lead to important actions that indirectly improve "the numbers."

Changing Physician Behavior

​More on Biases
A recent series of articles in the New England Journal of Medicine provide more insight into the issue of bias in medical decision making.

More on the Quality Paradox
The quality paradox is the number may improve while the experience of care worsens. What's new?

More on Variation - Part 1
Variation is not peculiar to healthcare, but is a general issue with the way the people think, and occurs whenever judgment is needed and the data are fuzzy.

​Performance Measurement
An expert panel has concluded less than half of current measures used by CMS to assess value for primary care services are valid. What does this tell us about current pay-for-performance efforts?