Checklists (The Sequel), 5 Million Lives, and the Magic of Measurable Results

Last month’s New England Journal included another astounding checklist study, an international extravaganza that found nearly 50% reductions in mortality and complications after implementation of pre- and post-op surgical safety checklists.

Wow.

Coincidentally, I read the study, conducted by a research team led by surgeon/author extraordinaire Atul Gawande, on my way home from a meeting at the headquarters of the Agency for Healthcare Research and Quality (AHRQ). The AHRQ gathering brought together the advisors to a new rollout of the Hopkins/Michigan checklist program to prevent central line-associated bloodstream infections (CABSI) to 10 additional states. You remember that study, published in the NEJM in 2007: implementation of a simple 5-item checklist in more than 100 Michigan ICUs led to over 1000 lives saved.

(Parenthetically, this was the study, led by Genius-Award Winner Peter Pronovost, that the Feds tried to shut down over the failure to obtain informed consent, an action that caused me to blow my gasket last year. The subsequent broo-ha-ha, fueled in large part by readers of this blog, led to a change in the federal policy regarding informed consent for quality improvement programs. Accordingly, the new 10-state roll-out of the CABSI program is considered “quality improvement” and thus exempted from the need for individual consent from providers and patients, Heaven be Praised.)

Last month’s surgical checklist study, one of the first initiatives of the WHO’s World Alliance for Patient Safety (spearheaded by the UK’s indefatigable and charismatic Chief Medical Officer, Sir Liam Donaldson) is, if anything, even more amazing than the Hopkins/Michigan effort. Why? Instead of changing the behavior of intensivists and ICU nurses, this one involves surgeons. And instead of changing practices in ICUs in a single U.S. state, this one did it in hospitals in 8 different cities around the world, ranging from Seattle to Manila, and Auckland to Amman. It was an audacious effort, which makes its results all-the-more-remarkable.

(In fact, it is hard to fully explain the tremendous decrease in surgical deaths and complications based on the nature of the intervention, leading some to question whether the results owe to a Hawthorne effect and are replicable. We’ll see, but for now I think the study is impressive and the intervention is likely to work elsewhere, albeit perhaps not quite as well as reported in the NEJM. OK, that’s enough parenthetical paragraphs for one posting.)

Gawande articulated (a particularly apt word in the case of this gifted writer) his zeal for checklists in his New Yorker article, “The Checklist.” And Pronovost is passionate about everything – he could make you joyful about doing your laundry. But what is so impressive about both these superb leaders and the projects they spawned was something more wonky than stirring: their insistence on a rigorous measurement strategy.

Which brings me to the recent wrap-up of the 5 Million Lives Campaign, the Institute for Healthcare Improvement’s (IHI) much-touted sequel to its 100,000 Lives Campaign. As you may recall, I critiqued the earlier campaign for using fuzzy math in estimating its “lives saved,” among other things. I worried aloud that the short-term high that came from celebrating the 100,000 Lives Campaign’s achievements might prove to be as ephemeral as that achieved by crack cocaine: without measurable results, it is awfully hard to generate sustainability, and the day-after hangover can be nasty.

And so it was with the 5 Million Lives Campaign, which ended with a relative whimper, sans the press hoopla that accompanied the 100,000 Lives effort. To IHI’s credit, there was no preening this time about lives saved, or defending of largely indefensible statistics. In fact, as the Boston Globe reported in December after the campaign’s end,

[the IHI] does not have numbers to measure the effect of its efforts over the last two years, IHI vice president Joseph McCannon said… But stories of reduced infection rates, improved medication management, and better cardiac care have been flowing in from the more than 4,000 hospitals participating in the campaign, he said…

The next stage, added McCannon, was the addition of three new “planks” to the 5 Million Lives practices: linking hospitals’ patient safety efforts to cost savings, preventing nosocomial urinary tract infections, and promoting the WHO surgical checklist program described above. In addition, IHI launched a tool, known as the “Improvement Map,” to help hospitals keep track of their various quality and safety initiatives.

This is all fine, but I get the sense that the absence of results took some wind out of the IHI’s sails. (Of course, that’ll be nothing compared to the wind removed if – as has been reported in the blogosphere – IHI founder and CEO Don Berwick becomes head of the Centers for Medicare & Medicaid Services).

What is the lesson in all of this? While baking rigorous measurement into large-scale tests of quality improvement or patient safety practices costs some time and money (doubtless the reason why IHI chose a Nike-like, “Just Do It” implementation strategy), the choice to eschew such measurement comes at a high cost. When the Michigan ICU study ended in a seminal New England Journal paper, there was tremendous momentum to do more, based largely on the unassailable evidence of lives saved. Not only did AHRQ jump at the opportunity to fund a rollout to 10 more states, but two private philanthropists gave Pronovost another $2.5 million to disseminate it even more widely! We haven’t heard the upshot of the WHO surgical checklist study yet, but you can bet that it will spark more efforts to promote this and similar interventions. Not just because the premise (marrying the use of checklists with culture change) made sense or because Gawande is a wonderful writer and a great guy, but because they demonstrably worked.

At the end of the AHRQ meeting last month, all the advisory board members were asked to identify the highlight of the day. Mine was this: I was floored by the blend of passion and scientific rigor shown by Pronovost and the other members of the project team (he’s being aided by the American Hospital Association’s Educational/Research Trust, along with several leaders from the Michigan effort).

Why did I focus on this combination of passion and scientific rigor? Because too little of the former and you can’t get projects this complex and messy off the ground. And too little of the latter, you risk finishing the project and not knowing whether you’ve accomplished anything of importance.

In other words, this is a Goldilocks problem and both Gawande and Pronovost, and their superb teams and funders, have gotten it just right.