CMS, There's a Fly in Your Soup

October 21, 2016

Authors: Herb B. Kuhn, MHA President and CEO, and Mat Reidhead, Vice President of Research and Analytics

The old adage, “waiter, there’s a fly in my soup!” dates to the 1880s when it was used metaphorically to describe the prevailing social conditions of the day. Being “in the soup” was used colloquially to describe a problem posing great difficulty. The fly apparently added insult to injury.

We recently published a commentary and preliminary research on the New England Journal of Medicine’s Catalyst website comparing the new Centers for Medicare & Medicaid Services’ star-ratings system to the culinary equivalent of a failed reduction sauce. Accurately measuring reflections of hospital quality and melding a proper reduction sauce present in-the-soup problems to health services researchers and chefs alike. The difficulty of these tasks lies in the precision of the ingredients and time required to get it right. Even slight deviations — a fly in the stock pot or flawed assumption — will render the batch unpalatable and meaningless.

The overall hospital quality star-ratings system is a statistical reduction of 64 existing quality measures into a five-point ordinal scale. Here’s a reductionist recipe of the rating system:

  • The 64 measures are broken down into seven domains.
  • Each hospital gets a score that is derived by a latent variable model for each domain in which they have enough data.
  • The domain scores are weighted subjectively and re-weighted for missing domains.
  • A clustering algorithm assigns each hospital 1-5 stars.

We began analyzing the data when they were released publicly in July and found evidence supporting all of the concerns raised by the industry and even Congress prior to the release. Most notable was the clear relationship between the star ratings and the sociodemographic mix of hospitals’ patients and communities. As we stated in the Catalyst piece, “The ingredients of the star ratings depend heavily on inputs that fail to account for the upstream social determinants of health that largely determine downstream health outcomes for patients from indigent communities, who rely on the nation’s safety net hospitals for access to the entire continuum of care — primary to quaternary.” That played out prominently in the data — Table 1 shows the SDS differences we detected in the communities of one-to-five star hospitals.

Table 1 CMS Star Ratings

We found multiple other issues around the sensitivity of the models to the number of domains available, and the inclusion of suspect measures. We submitted all of our findings to CMS Acting Administrator Andy Slavitt on Sept. 23 with no response to date.

Again, a successful reduction sauce requires good ingredients and time. CMS rushed the release of the star ratings through an opaque review and endorsement process at NQF, even with a brief delay induced by Congress, during which time they addressed very few industry concerns. But the larger problem is with their ingredients. The continued decision to exclude social factors from the risk adjustment of measures underlying the star ratings is an elephant-sized fly in the soup.

Check out the post on NEJM Catalyst and leave a comment with your perspective. It’s a quick, but telling read.