Having Chest Pain? First, a Nosh.

I do hospitals, not meat loaf or fried chicken.  Stick with me.  Lots of great tables and charts to follow.

One thing you need to know about New York City is the overabundance of eating establishments on our streets.  On my block alone, there is Thai, not one, but two Turkish restaurants, Burmese, Italian, Chinese, and Persian.  Yes, that is just my block.

You will also notice inspection scores in the windows nowadays—A, B, C, etc.

One year ago, the NYC Department of Health required restaurants to display these letter grades in their front windows:

The placards are ubiquitous, almost to the point of invisibility; the letters have reached a saturation point like the “TEAMWORK” signs omnipresent in corporate lunchrooms.  You see them, but you do not.

This recent WSJ piece obliged some redeliberation:

One year after the city started branding restaurants with letter grades based on health inspections, one thing is clear: Most restaurants are getting the highest mark, an A. Department of Health officials say 69% of restaurants inspected had received A’s as of July 1.

Meanwhile, 15% scored B’s and 4% have C’s, the lowest mark. The remaining 12% are contesting their scores.

“The goal was transparency and food safety and I think the grades really speak for themselves,” said Susan Craig, a health department spokeswoman.”

On my street, I counted eight A’s, one B, and one pending.

As I strolled however, I pondered, a) would I eat in the ‘B’ restaurant and, b) what distinguishes an ‘A’ and ‘B’ score.

This is not an accident, as I visit Hospital Compare occasionally and consider how laypeople navigate this site, interpret outcomes scoring, and more importantly, how they utilize its findings.

Unlike hospitals however, I am unfamiliar with the sanitary state of restaurants, what constitutes excellence versus mediocrity, and whether an ‘A’ versus ‘B’ mark captures a significant difference in food preparation—or at least one I care about.  Perhaps my training and linear take on the world skews my perspective (a critical one), but this exercise levels the playing field.  I am a novice in this domain.

Therefore, what concerns me when I view a score?  Simple.  Are my chances of succumbing to food borne illness greater than average?

A priori, applying my health system understanding and skepticism in HHS hospital ranking methodology, I anticipated eateries would not stray too far asunder, i.e, I am not overflowing with optimism.

What then merits a top grade?  Tally up to 13 demerits and you bag an ‘A.’ However, collect 14 and earn a ‘B.’ From the same WSJ piece:

The Wall Street Journal analysis showed many more restaurants on the cusp received scores of 12 and 13 than 14, which marks the change between an A and a B.

There were 2,304 restaurants scoring 12 points and 1,656 restaurants scoring 13 points, but only 287 restaurants scoring 14 points.

That seems random, huh?  Your refrigerator is off a degree or the bleu cheese is too close to the peanut butter, and you now reside in the second tier.  You ought to see my kitchen.

For the relevance, view this DOH release; virtually all folks surveyed consider these grades when making dining choices.  But again, the selection is not difficult when the distribution curve is heftily skewed and the scoring system does not correlate with a specified outcome, mainly, will I get sick?

Once more, am I at  greater risk of illness if a restaurant scores a 14 versus 12? Damned if I know.  I doubt the restaurant eating public knows either, despite their interest in the rankings.

Expected Distribution?

Alright, that was the nosh.  Now the “chest pain.”

Perhaps you sensed the direction I was headed.  Before I proceed however, look at this table from Hospital Compare.  You will notice parallels between what I presented above, and the data here:

Decipherable Meaning?

The similarities?

• Vague characterizations: What is better?  What is no different?
• It is dense and difficult to discern HHS methods (this is asking too much for a layperson to disentangle).  I could not quickly find what “better,” the numeric tail end of the curve, signifies, e.g., top 5, 10, or 20%.
• Representativeness: Does choosing one of these facilities really give me better odds of experiencing a superior outcome.

Comparing restaurants and hospitals, it is notable how on first appraisal, the scores assessing them express such “import.”  Examined more critically however, one is left wanting.  Do they really assist the consumer and patient in achieving their objective of safer meals or care.

On the hospital side, the literature base is vast (and mixed) on the”what we are measuring, and what the people care about” effort.  We have much to learn regarding the association between the metrics we collect and the outcomes we pursue.  This study from JAMA is a helpful example.  Skim the abstract.

Also, patient selections are likely correlated to institutional reputation and recommendations from our family and friends, as this citation from JHE affirms:

[…] First, consumers’ perceptions of reputation and medical services contribute substantially to utility for a hospital choice. Second, consumers tend to select hospitals with high clinical quality scores even before the scores are publicized. However, the effect of clinical quality on hospital choice is relatively small. Third, satisfaction with a prior hospital admission has a large impact on future hospital choice. Our findings suggest that including measures of consumers’ experience in report cards may increase their responsiveness to publicized information, but other strategies are needed to overcome the large effects of consumers’ beliefs about other quality attributes.

Finally, from this 2011 RWJF poll, individuals express an analogous view, that is, your inner circle matters more than expert opinion.

Nearly 6 out of 10 Choose Based on Reputation

It is demoralizing that with this effort— collecting, analyzing, and disseminating reams of data—the magnitude of the response does not match up in kind.

I have no solution, but I am certain we are missing the mark.  The “end user,” the intended audience for this information, is lost in a haze of material.  That the expectation “because x, then y” is deposed is ostensibly clear.

However, the consequences of these efforts are not futile.  The effects from publicity via the press (take a look) exert a substantial deterrent, and this is the potent force.  That is propelling change–more than we appreciate, and the linkages between the rankings and any corrective actions hospitals (or restaurants) engage in are rooted in this association.  Future work must account for this response.

I believe patients (and diners) do not fully grasp the breadth of the data.  My take: they need a connector, maven, and salesman to complete the circle.

Even with transparency however, I fear, like 70% of restaurants earning A’s, hospitals are traversing the same path.  When virtually all facilities earn a “you pass,” how does the public respond?  The tools currently at our disposal today are too blunt to sort the wheat from the chaff, and folks are not mindful.  These are not a compilation of toaster ovens in Consumer Reports.

Perhaps, ultimately, our grading system will resemble that of our institutions of higher learning (dynamite stuff).  We all regress to an expanding mean, aka, playing to the measures:

I conclude, other than to state, we are not alone: teachers are experiencing similar growing pains, and if you were researching surgery for your child, would you find this heartening?

Unraveling this challenge will take time, more than a decade.  It is not a happy ending, I am aware, but the truth speaks.  We have an abacus. We need Watson.

—————————————————————————————————————————————

*BONUS*

Finally, I read this and smiled.  I love Mel Brooks, and despite his age, his humor is as sharp and contemporary as ever.  Blazing Saddles ranks among the funniest films ever made (if you have to guess what I linked to, you are not a true fan :)).

From a recent interview with Dick Cavett, when speaking on Mr. Brook’s deceased wife Anne Bancroft, this gem of a quote appeared:

After the reunion show a woman from the audience said, “I wonder what it would be like to be married to a man like that.”

The late Anne Bancroft, who was, when asked a similar question had replied, “When he comes home at night and I hear his key in the lock I say to myself, ‘Oh good! The party’s about to begin.’”

It does not get better than that.