More on Quality Reports: Lessons From SAT Scores

My older son is gearing up to apply to college (:-\

and so I bought him one of the Bibles, the Fiske Guide. The book is cleverly written – enough academic factoids to get parents to spring for it, leavened with enough social scene skinny to get kids to read it. The Guide dutifully lists ranges of SAT scores for accepted applicants at 300 schools, but then adds this shocking caveat:

their zeal to make themselves look good in a competitive market, some
colleges and universities have been known to be less than honest in the
numbers they release. They inflate their scores by not counting certain
categories of students at the low end of the scale, such as athletes,
certain types of transfer students, or students admitted under
affirmative action programs. Some colleges have gone to such extremes
as reporting the relatively high math scores of foreign students, but
not their relatively low verbal scores…(p. xix)

Since writing a piece in JAMA a couple of months ago on the need for standards in quality reporting (described in this blog posting), we’ve gotten a bit of blowback from folks who worry that healthcare is too complicated for such standards, or that this kind of overregulation is making the practice of medicine numbingly robotic. The meta-message is that we can police ourselves perfectly well, thank you.

But I think Fiske’s SAT story is instructive: once hospitals are really competing on quality and safety, gamesmanship is inevitable (“Oh, yes – we exclude patients over age 80 in calculating our mortality figures, and widows and widowers in our readmission rates. It’s all there in the fine print – didn’t you see it?”), and we’ll need standards to create (and enforce) a level and credible playing field. I don’t want to minimize the challenges, but the SAT shenanigans are a window into what happens when the standard-setting can is kicked too far down the road.

1 Comment

  1. tholt on December 14, 2007 at 4:51 am

    I come from the land of Lake Wobegon where “all the women are strong, the men are good looking and the children are above average”. We simply don’t accept public announcements of our weaknesses. This is particularly true of physicians who like to compete and win. If we get public reporting there will be manipulation of the data so every clinic’s quality is strong, every physician’s patient satisfaction report is good looking and every hospital’s safety record is above average.
    I believe public reporting of good, useful data is very important and worth pursuing. I wonder if it will really work.

Leave a Comment