HSS Study Highlights Flaws in CMS Star Ratings

News provided by

Mar 22, 2019, 09:00 ET

NEW YORK, March 22, 2019 /PRNewswire/ -- A leading quality rating for the nation's hospitals appears not to adequately account for the risks of undergoing certain procedures at certain hospitals, particularly joint replacement surgery, according to a new study by researchers at Hospital for Special Surgery (HSS) in New York City.

The study, reported in the JBJS Open Access, found that the Overall Hospital Quality Star Ratings program from the Centers for Medicare & Medicaid Services (CMS) is unreliable in several respects. In particular, the star system significantly understates the risk of complications among patients who undergo total joint arthroplasty (TJA) at hospitals that perform relatively few of these surgeries. The researchers also showed that the star algorithm fails to fully capture the typical observation that higher surgical volume is associated with better quality outcomes.

"The apples to oranges comparisons across hospitals in the Hospital Stars program produce ratings that are of uncertain usefulness," said Catherine H. MacLean, MD, PhD, a rheumatologist and chief value medical officer at HSS, a co-author of the new study. "The Hospital Stars Program misses the mark in providing actionable information to consumers, who generally are looking to understand the quality for specific procedures."

The study is particularly timely, as CMS announced recently that it was considering an overhaul of the way it calculates the controversial star rating system. Any changes would not take effect until 2020, according to the agency, which will be accepting public comments on the proposed revision.

CMS launched the star ratings in July 2016 as part of a broader effort to promote value-based care -- higher quality outcomes at the lowest possible cost. The system, which is due for an update this month, currently includes 57 performance measures covering seven categories. Together they capture mortality, patient safety, readmission to the hospital, effectiveness and timeliness of care and other relevant factors in hospitalization. Hospitals can receive an overall rating of between one and 5 stars.

"However, in creating the system, CMS did not fully account for the impact of the volume of procedures in its algorithm," says Mark Alan Fontana, PhD, a data scientist and health economist at HSS and lead author of the new study. Hospitals that perform many of a particular surgery or intervention typically have better outcomes for those procedures than facilities that see fewer such patients. But the star system does not include measures for hospitals that perform fewer than 25 (but more than zero) procedures over a three-year period in some cases. As a result, he and his colleagues speculated that the ratings would change, perhaps significantly, if those data were incorporated into the model.

"We hypothesized that if some quality measures for certain hospitals are excluded from CMS's calculations because of low volume, and if low volume is associated with worse outcomes, then including those omitted quality measures would negatively impact associated hospitals' ratings," they wrote. Moreover, because the star system links all hospitals through relative ratings, the changes could affect other facilities in the database, too.

The HSS researchers assessed four measures, two for TJA—complications and readmissions—and two for cardiac surgery—mortality and readmissions, for which high-volume hospitals tend to perform better than low-volume hospitals. They used three methods to estimate values for the missing low-volume facilities from the public CMS database.

For three of the four measures, including the estimates had no effect on the overall ratings—suggesting that the star ratings do not reflect the volume-outcome relationship for these measures.

For the fourth, complications after TJA, nearly 40% of hospitals saw their score change once the estimates of the low-volume data were added to the model. Of those, roughly a third gained a star or more while the remaining two-thirds lost a star or more. Although the exact percentages differed depending on the method the researchers used to estimate the missing values, the overall trend was the same for each of the three approaches, they said. "Overall, although low-volume hospitals were more often hurt than helped after imputation [of the missing, low-volume outcome measures] … higher-volume hospitals were also more often hurt than helped," they report.

The researchers also showed that the underlying safety domain model is not stable. This is because it heavily weighs one quality measure. Slight changes to the underlying data, like complications after TJA, force the model to "flip" to heavily weigh another quality measure. This can dramatically change a hospital's star rating.

The findings underscore the importance of making quality metrics as accurate as possible. "As health-care incentives move toward value-based programs in which higher quality is rewarded financially, defining quality in relative terms may very well prevent some providers with high absolute quality scores, but lower relative quality scores, from achieving those rewards," the HSS researchers write. "Relative ratings will tend to overemphasize the differences between hospitals, even if true differences are minimal. This overemphasizing of differences, in turn, could engender counterproductive competition by discouraging hospitals from sharing best practices and collaborating."

About HSS | Hospital for Special Surgery

HSS is the world's leading academic medical center focused on musculoskeletal health. At its core is Hospital for Special Surgery, nationally ranked No. 1 in orthopedics (for the ninth consecutive year) and No. 3 in rheumatology by U.S.News & World Report (2018-2019). Founded in 1863, the Hospital has one of the lowest infection rates in the country and was the first in New York State to receive Magnet Recognition for Excellence in Nursing Service from the American Nurses Credentialing Center four consecutive times. The global standard total knee replacement was developed at HSS in 1969. An affiliate of Weill Cornell Medical College, HSS has a main campus in New York City and facilities in New Jersey, Connecticut and in the Long Island and Westchester County regions of New York State. In addition, HSS will be opening a new facility in Florida in late 2019. In 2018, HSS provided care to 139,000 patients and performed more than 32,000 surgical procedures, and people from all 50 U.S. states and 80 countries travelled to receive care at HSS. In addition to patient care, HSS leads the field in research, innovation and education. The HSS Research Institute comprises 20 laboratories and 300 staff members focused on leading the advancement of musculoskeletal health through prevention of degeneration, tissue repair and tissue regeneration. The HSS Global Innovation Institute was formed in 2016 to realize the potential of new drugs, therapeutics and devices. The HSS Education Institute is the world's leading provider of education on musculoskeletal health, with its online learning platform offering more than 600 courses to more than 21,000 medical professional members worldwide. Through HSS Global Ventures, the institution is collaborating with medical centers and other organizations to advance the quality and value of musculoskeletal care and to make world-class HSS care more widely accessible nationally and internationally.

SOURCE Hospital for Special Surgery