|
Measurement Error
One idea you learn in science labs during the pre-med years is measurement error—all measurements have degrees of precision and none is perfect. Sometimes, but not always, the measurement error can impact the observed results and lead to erroneous conclusions. But measurement error is not confined to the laboratory. Some 40 years ago, my boss sent me to a course on “Management by Objectives.” The idea was that being quantitative about one’s goals made it easier to determine if you had reached your objective. The example used was conserving budgeted funds and living within that budget. Of course, being the government, if you showed you could get by with less, you got less next year to spend, so there was a perverse incentive. I did not win any points for making this observation out loud. Ari Robicsek, Chief Medical Analytics Officer for Providence St. Joseph Health System recently made some observations on measurement errors.[1] He started by pointing out the difference between measuring what is easy and what is important. He also notes the importance of “balanced” risk adjustment and the attribution problem when care is delivered by teams. His most important item, though, is the notion that we should measure to learn what works, not to rank order some aspect of care, a notion I fully support. When I first began talking about continuous quality improvement (CQI) in dialysis units, I did so with the idea that we could do several things with this tool. First, we could define if our process was stable and delivering the care we intended. Second, we could learn if deliberate changes in our process were good, bad, or indifferent. Third, we could provide an operational definition of success to give the dialysis team feedback on their craftsmanship, since death rates were, and remain, high. Unfortunately, the CQI process morphed into “QAPI” where the goal is to avoid monetary penalties on a series of measures. The bundle of measures has been changed when they did not provide adequate rank-ordering. It makes me think of the children’s book “The Story About Ping,”[2] when the duck who is last hides to avoid the spanking and is almost eaten instead. Somebody is going to be last and penalized in any rank order, even though it is largely random. This is another type of measurement error. Another example is the EHR debacle.[3] Begun with the intention of make care better, safer, and cheaper, ten years and $36 billion later we have a system that serves none of its goals. “By one measure, the effort has achieved what it set out to do. Today, 96% of hospitals have adopted EHRs, up from 9% in 2008. But on most counts the newly installed technology has fallen short… ‘Every single idea was well-meaning and potentially of societal benefit, but the combined burden of all of them hitting clinicians simultaneously made office practice basically impossible. In America we have 11 minutes to see a patient, and, you know, you’re going to be empathetic, make eye contact, enter about 100 pieces of data, and never commit malpractice. It’s not possible.’” The article goes into the issues in much greater depth, but the bottom line is adoption was driven by financial incentives and the systems were created and installed to capture those dollars with insufficient attention paid to the issues raised when older workflows were disturbed. Yes, the older workflows could stand improvement, but the attitude was very much one of making the clinician adapt to the software, not the other way around. The third article examines the RAND report on hospital price data.[4] “Controversy over methodology aside, the RAND report makes one point absolutely clear. The most basic information needed to create a functioning health care market—data on health care prices—is lacking in the United States. The obvious question: why is it so hard for purchasers to find out what they are paying for health care? The main reason is that insurers and providers don’t want to release the data.” The authors argue what is needed is a databank of the actual negotiated prices obtained by insurance companies, not the list price on the charge master. I would argue that the real issue is not price, but the cost. For most hospitals a majority of their payments are from the government, which pays a fixed fee. But the commercial payers account for the profit margin, so providers are reluctant to provide leverage to reduce revenue on that side of the ledger. More importantly, efforts spent trying to deal with this issue detract from the need to figure out what care costs the hospital or clinic to provide. This is measurement error by misdirection and misallocation. Lastly is an article that tries to measure the impact of patient behaviors and preferences on costs and outcomes.[5] He quotes a 2017 JAMA article estimating 74% of the outcome differences across countries are explained by life-style differences, and only 10-15% by medical care efforts. He also notes a recent survey of 10,000 people showing only 31% considered costs an issue in health-care decisions, but the doctor’s compassion is important to 85%. “American patients don’t like to be told that unexplained symptoms aren’t ominous enough to merit tests…American patients’ disregard for routine care is another problem…Finally, the U. S. stands out as a place where death, even for the very aged, tends to be fought tooth and nail, and not cheaply.” By focusing on easily obtained clinical metrics and rank-ordering physicians and organizations, and focusing on price rather than cost, we have made a measurement error. Maybe we should instead be focusing on what patients want and need and figuring out what can be done to moderate inappropriate demand. But it won’t be simple. 8 July 2019 [1] Robicsek A. Six Modest Proposals for Health Care Measurement. 29 May 2019. https://catalyst.nejm.org/videos/six-modest-proposals-health-care-measurement.html. [2] Flack M. The Story About Ping. (New York: Viking Press, 1933.) [3] Schulte F, Fry E. Death By A Thousand Clicks. 18 March 2019. https://khn.org/news/death-by-a-thousand-clicks/. [4] Gustafson L, Seervai S, Blumenthal D. The U. S. Can’t Fix Health Care Without Better Price Data. Harvard Business Review, 30 May 2019. Accessed 31 May 2019 at https://hbr.org/2019/05/the-u-s-cant-fix-health-care-without-better-data/. [5] Freedman DH. The Worst Patients in the World. The Atlantic, July 2019. Accessed 16 June 2019 at https://theatlantic.com/magazine/archive/2019/07/american-health-care-spending/590623. |
Further Reading
Beyond Evidence-Based Medicine The problem with EBM is that we are trying to use the method where it does not really apply. Confronting The Quality Paradox - Part 1 Quality Metrics Risk, Reward, and Other Reasons Patients Don't Follow Medical Advice Patients often don't do what their doctors recommend. The problem is important and contributes to "bad" outcomes, yet we have little insight into the problem. Stupid Stuff Wouldn't it be wonderful if we got rid of stupid stuff? |