Can You Rely on Hospital Report Cards?

By  |  October 15, 2019 | 

This from NEJM:

“However, it is unclear whether current rating systems are meeting stakeholders’ needs. Such rating systems frequently publish conflicting ratings: Hospitals rated highly on one publicly reported hospital quality system are often rated poorly on another. This provides conflicting information for patients seeking care and for hospitals attempting to use the data to identify real targets for improvement.

However, to our knowledge, there has been no prior systematic review or evaluation of current rating systems that could help inform patients, clinicians, and policymakers of the various systems’ methodologies, strengths, and weaknesses.”

CMS Hospital Compare Overall Star Ratings, Healthgrades Top Hospitals, Leapfrog Safety Grade and Top Hospitals, and U.S. News & World Report Best Hospitals.

Note the disparity in the table below and the designs used by each organization. Also, note, each submitting hospital utilizes different approaches to coding and reporting–the inputs, and thus, risk adjustment may increase (or decrease) their odds of coming out on top; reporting adverse events to regulators (or not) does the same, and comparing each hospital’s typical population mix (high versus low SES) can be an apples to oranges over an apples to apple game.

Look at the circles in red. Go with USNWR, and you are sailing smooth. But oh wait, Healthgrades says avoid that same facility at all costs. A grown-up size problem and whose methods and grades do you trust?

Evaluation science has a long way to go—and this is just one of many examples. There are too many inaccuracies and unaccounted for reporting domains (“what does the patient say and how are they doing,” for one). One wonders how useful these kinds of assessments are in our present metric milieu—aside from which, do patients use them and prioritize their findings over what their family or PCP says. Most of the studies we have today are a mixed bag.

I would also add, some everyday rating sites like Yelp can approximate what the big boys mentioned above do. Wisdom of the crowd type stuff and who knows what the public will see a decade from now.

Also, note the comments at the bottom of the piece. LOTS of pushback.

Here is a sample of why we things are amiss in rating land:

Unfortunately, these administrative data, collected for billing rather than clinical purposes, have notable, well-described shortcomings. The data used are generally limited to those 65 and older who participate in the Medicare Fee-for-Service program. The data often lack adequate granularity to produce valid risk adjustment. Moreover, outcomes reported in administrative data have been shown to have high false-negative and false-positive rates. There are also notable ascertainment or surveillance bias issues that invalidate some of these measures (e.g., the PSI-12 VTE outcome measure). 

One Comment

  1. Avatar
    Rupesh Prasad October 24, 2019 at 9:20 am - Reply

    Hi Brad,

    Thanks for sharing this interesting article highlighting the differences in major hospital rating systems. While their methodologies may differ, I think the way forward would be to standardize and validate metrics across these rating organizations. This would require transparency on their part with regard to the data collection and statistical analyses, but would ultimately help generate meaningful reports that could be used by all stakeholders including health systems, patients and families, and clinicians.

    Rupesh.

Leave A Comment

About the Author: Bradley Flansbaum

Bradley Flansbaum
Bradley Flansbaum, DO, MPH, MHM works for Geisinger Health System in Danville, PA in both the divisions of hospital medicine and population health. He began working as a hospitalist in 1996, at the inception of the hospital medicine movement. He is a founding member of the Society of Hospital Medicine and served as a board member and officer. He speaks nationally in promoting hospital medicine and has presented at many statewide meetings and conferences. He is also actively involved in house staff education. Currently, he serves on the SHM Public Policy Committee and has an interest in payment policy, healthcare market competition, health disparities, cost-effectiveness analysis, and pain and palliative care. He is SHM’s delegate for the AMA House of Delegates. Dr. Flansbaum received his undergraduate degree from Union College in Schenectady, NY and attended medical school at the New York College of Osteopathic Medicine. He completed his residency and chief residency in Internal Medicine at Long Island Jewish Medical Center in New York. He received his M.P.H. in Health Policy and Management at Columbia University. He is a political junky, and loves to cook, stay fit, read non-fiction, listen to many genres of music, and is a resident of Danville, PA.

Categories

Related Posts

November 8, 2019 |  0
Do you have a stack of journals piling up on your desk, beside your bed or in your email inbox? In 1950, medical knowledge was estimated to double every 50 years, but now the doubling time is every few months. At this rate, it is impossible to keep up with the literature, but a group […]
November 5, 2019 |  0
I have written frequently over the last few years on topics related to the sustainability of the hospital medicine practice model, beginning with this post in August, 2015. I continue to be concerned by what I see as a confluence of significant trends that are conspiring to challenge hospital medicine’s status quo. On one hand, […]
October 31, 2019 |  4
I have hit this topic before-copying and pasting notes—about a year ago. Let’s revisit. First, note the figure below. To the far right of the x-axis, you see the number of notes with > 60% + of manual entries, i.e., not pasted, in a large sample of studied notes—this from a JAMA study out of […]