When is action needed?
At any given point in time, a large hospital system might be tracking its performance on a couple of hundred different clinical quality measures. If that sounds surprising, consider that hospitals report metrics to a bunch of regulating bodies each year in the name of quality improvement. New measures are being proposed all the time.
But it’s not just about reporting. The mission of any hospital is to deliver high quality care. So imagine that job. As an organization you are watching hundreds of trend lines each day, working to improve care quality. Should you worry about every blip on the radar? What’s considered “a problem”? Does the answer to that change with time, as data improves, or as the population coming in the door shifts?
To a great extent an organization cannot really be told what constitutes “concerning” variation in its data. There are exceptions, where a big change unequivocally signifies a call to action. But over a large body of metrics, the more common situations require consensus building and the development of a system of rules. Variation that is regarded as problematic is highly dependent on what matters to the organization, as well as the context of the variable data point.
Two good points of departure toward a system for monitoring quality metric anomalies are to start to understand a. what quantitively happens “seldom”; and b. why the shift occurred.
Background
New York-Presbyterian hospital (NYP) is a leader in the delivery of high-quality clinical care. NYP supports numerous quality improvement initiatives and was interested in exploring work around an anomaly detection pipeline to integrate with weekly reporting. The goal was to systematically alert end users to trend line variation that would be considered alarming or worthy of action on a metric-by-metric basis.
Challenge
Since anomaly detection models had not previously been developed for most metrics, there was no baseline for what constitutes a true anomaly.
Solution
Part 1: Quantitative Analysis & Modeling
A series of quantitative approaches were developed to detect rare variation on several metrics, including a seasonal decomposition model; DBSCAN clustering models; and simpler month over month calculations. The idea was to analyze trend data through both crude and more sophisticated approaches to move toward consensus about anomalous variation per metric.
The results of different methods were overlaid with one another to triangulate on where the data seemed to consistently indicate “this rarely happens”.
Part 2: Root Causes
The second part of this project was to consider reasons for why anomalies might occur and how that could be systematically detected. For that, the classic framework of proximal to distal drivers of health provided a useful way of organizing thinking about clinical data that could be peeled back over time, since work was expected to be ongoing.
This framework offered a tidy phased approach to investigating root causes of metric variation, enabling a minimum viable product that could be put in place immediately while lending more and more value over time.
Part 3: Pipeline
A key additional element of this work involved generating end user messages that provide context as to why the metric may be out of range. This could easily be integrated into weekly reporting as consensus was gained over time about what values should raise red flags for what clinical groups.
Results
This was exploratory work conducted for New York-Presbyterian hospital. This project laid critical groundwork for the hospital’s capacity to monitor a large volume of clinical quality measures, while also lending important contextual information about what’s driving the observed changes.
Delivered:
Want more solution inspiration? Return to the gallery to read more.