Can You Rely on Hospital Report Cards?

By  |  October 15, 2019 | 

This from NEJM:

“However, it is unclear whether current rating systems are meeting stakeholders’ needs. Such rating systems frequently publish conflicting ratings: Hospitals rated highly on one publicly reported hospital quality system are often rated poorly on another. This provides conflicting information for patients seeking care and for hospitals attempting to use the data to identify real targets for improvement.

However, to our knowledge, there has been no prior systematic review or evaluation of current rating systems that could help inform patients, clinicians, and policymakers of the various systems’ methodologies, strengths, and weaknesses.”

CMS Hospital Compare Overall Star Ratings, Healthgrades Top Hospitals, Leapfrog Safety Grade and Top Hospitals, and U.S. News & World Report Best Hospitals.

Note the disparity in the table below and the designs used by each organization. Also, note, each submitting hospital utilizes different approaches to coding and reporting–the inputs, and thus, risk adjustment may increase (or decrease) their odds of coming out on top; reporting adverse events to regulators (or not) does the same, and comparing each hospital’s typical population mix (high versus low SES) can be an apples to oranges over an apples to apple game.

Look at the circles in red. Go with USNWR, and you are sailing smooth. But oh wait, Healthgrades says avoid that same facility at all costs. A grown-up size problem and whose methods and grades do you trust?

Evaluation science has a long way to go—and this is just one of many examples. There are too many inaccuracies and unaccounted for reporting domains (“what does the patient say and how are they doing,” for one). One wonders how useful these kinds of assessments are in our present metric milieu—aside from which, do patients use them and prioritize their findings over what their family or PCP says. Most of the studies we have today are a mixed bag.

I would also add, some everyday rating sites like Yelp can approximate what the big boys mentioned above do. Wisdom of the crowd type stuff and who knows what the public will see a decade from now.

Also, note the comments at the bottom of the piece. LOTS of pushback.

Here is a sample of why we things are amiss in rating land:

Unfortunately, these administrative data, collected for billing rather than clinical purposes, have notable, well-described shortcomings. The data used are generally limited to those 65 and older who participate in the Medicare Fee-for-Service program. The data often lack adequate granularity to produce valid risk adjustment. Moreover, outcomes reported in administrative data have been shown to have high false-negative and false-positive rates. There are also notable ascertainment or surveillance bias issues that invalidate some of these measures (e.g., the PSI-12 VTE outcome measure). 

One Comment

  1. Avatar
    Rupesh Prasad October 24, 2019 at 9:20 am - Reply

    Hi Brad,

    Thanks for sharing this interesting article highlighting the differences in major hospital rating systems. While their methodologies may differ, I think the way forward would be to standardize and validate metrics across these rating organizations. This would require transparency on their part with regard to the data collection and statistical analyses, but would ultimately help generate meaningful reports that could be used by all stakeholders including health systems, patients and families, and clinicians.


Leave A Comment

About the Author: Bradley Flansbaum

Bradley Flansbaum
Bradley Flansbaum, DO, MPH, MHM works for Geisinger Health System in Danville, PA in both the divisions of hospital medicine and population health. He began working as a hospitalist in 1996, at the inception of the hospital medicine movement. He is a founding member of the Society of Hospital Medicine and served as a board member and officer. He speaks nationally in promoting hospital medicine and has presented at many statewide meetings and conferences. He is also actively involved in house staff education. Currently, he serves on the SHM Public Policy Committee and has an interest in payment policy, healthcare market competition, health disparities, cost-effectiveness analysis, and pain and palliative care. He is SHM’s delegate for the AMA House of Delegates. Dr. Flansbaum received his undergraduate degree from Union College in Schenectady, NY and attended medical school at the New York College of Osteopathic Medicine. He completed his residency and chief residency in Internal Medicine at Long Island Jewish Medical Center in New York. He received his M.P.H. in Health Policy and Management at Columbia University. He is a political junky, and loves to cook, stay fit, read non-fiction, listen to many genres of music, and is a resident of Danville, PA.


Related Posts

June 22, 2020 |  0
Traditionally, researchers run clinical trials (RCTs) in three steps: They formulate the trial. They conduct the trial via a preconceived design. They analyze the data according to a pre-specified plan. When you have only a few bucks, little time to spare, and billions of dollars on the line, what might you do to economize the […]
June 18, 2020 |  1
The kids are navigating the crisis with maturity and aplomb and way too much “Clash Royale.” Online school somehow went through the wash and has shrunk to 3 hours, including lunch. They continue to seek ways to fill up the day. We’ve tried to set some clear goals, pulled out the QI toolkit and made […]
By Khaalisha Ajala, MD, MBA
June 12, 2020 |  4
One might immediately think of the murders of Eric Garner, George Floyd or even the fictional character Radio Raheem from Spike Lee’s critically acclaimed film, Do the Right Thing, when they hear the words “I can’t breathe.” These words are a cry for help. The murder of these unarmed black men is devastating and has […]