Tipping a sacred cow: QI under the microscope

So much media and journal space has been devoted to financial conflicts of interest, particularly within and related to pharma and device manufacturers, that to write any more about it may be redundant. On this site we have also intermittently addressed COI from other perspectives, such as financial interest of the members of the American College of Radiology in maintaining mammography screening status quo, thinly veiled in its own version of the pernicious "death panel" language. We have also spoken a bit about the non-financial COI. And even though we are so very much aware of COI's potential to lurk around every corner, there are still some surprises.

Take the sacred cow of "quality improvement" in healthcare. Even the name, much like the "pro life" moniker, suggests that it is untouchable in its purity and nobility of purpose. So necessary is it because of the epic magnitude of morbidity and mortality attributed to healthcare itself, that the billions of dollars spent on it seem unquestionably justified. Indeed, much like our public education system, the QI movement garners higher and higher allocations simply due to the sheer face validity of the assumption that more of it is better. And the most fascinating aspect is that, in our current zeal for sensible economic allocation through evidence, QI, much like education, appears immune to scrutiny. This is the very definition of politics driving policy.

I return to the case of ventilator-associated pneumonia, or VAP, as the poster child for this movement. I have already alluded to the fact that definitionally VAP is a slippery slope: its diagnosis varies based not only on the tools used to diagnose it, but also depending on who is doing the diagnosing. Yes, indeed, what one clinician calls VAP another may call absence of VAP. I have also dissected the weak evidence behind some of the strongest purportedly evidence-based recommendations aimed at VAP prevention. But what if VAP itself is the wrong endpoint? What if we are spending untold dollars and other resources on a futile pursuit?

Do you feel yourself bristling yet? If you said "yes", it is a normal response I get from my colleagues and people who read my scholarly papers. Because how can anyone be against QI? Well, I am not against QI. I am simply against sanctifying QI as a sacred cow and thus shielding it from a sensible and rational evaluation.

So, if you are over the initial shock, allow me to explain myself. I am sure you have heard of surrogate endpoints. Here is a definition from Wikipedia:

In clinical trials, a surrogate endpoint (or marker) is a measure of effect of a certain treatment that may correlate with a real clinical endpoint but doesn't necessarily have a guaranteed relationship. The National Institutes of Health (USA) defines surrogate endpoint as "a biomarker intended to substitute for a clinical endpoint".^[1][2]

Surrogate markers are used when the primary endpoint is undesired (e.g., death), or when the number of events is very small, thus making it impractical to conduct a clinical trial to gather a statistically significant number of endpoints. The FDA and other regulatory agencies will often accept evidence from clinical trials that show a direct clinical benefit to surrogate markers. ^[3]

This begs the question of what constitutes a "real" clinical endpoint. Well, in my simplemindedness I think of them as endpoints that matter to the patient or in the long run. So, death, disability, quality of life, functionality, these are the real endpoints. Something that alters one's life or threatens it is a real endpoint. Thus, blood pressure and cholesterol are surrogate endpoints, since they usually, but not always, correlate with the risk of a myocardial infarction or death. But what if such a correlation did not exist? Furthermore, what if a cholesterol level was measured with, say, tea leaves, and therefore was subject to a tremendous variation in detection? Would we then spend hundreds of billions of dollars on trying to alter this factor or would we calmly and rationally walk away and look for something that truly impacts the real outcome of a heart attack or death? I think I am making my point fairly clearly.


Let me explain why I think that VAP is but a surrogate outcome, and, given its diagnostic challenges, not a sensible one in the least. VAP by definition occurs in patients on mechanical ventilation (breathing machine), whose quality of life is fairly badly damaged in the short term. The literature would suggest that not all VAP impacts mortality adversely, but some forms of VAP indeed do, particularly VAP that develops late in the course of illness. So in this VAP does correlate with a real endpoint. Also, there is very little doubt that getting VAP prolongs one's dependence on mechanical ventilation, and increases the duration of the stay in the ICU and hospital overall. So, this can be considered not a very good, albeit real, outcome. An additional point to remember is that VAP engenders the use of additional, usually broad spectrum, antibiotics, putting both the individual and the society at risk for such unwanted consequences as the emergence of highly resistant microorganisms.


So, even though VAP is a surrogate endpoint, it certainly seems to fit the bill for something we would want to prevent. But here is the monkey wrench in this argument: what seem to be great surrogate endpoints do not always end up correlating with clinical reality. The association of VAP with morbidity and mortality has been detected in mostly retrospective observational studies. Trials of VAP prevention rarely, if ever, report any endpoint other than VAP. And, given how elusive VAP diagnosis is, there is plenty of room for gamesmanship so pervasive in the real world to make any data fit our preconceived hypotheses and political needs. 


So, what is my point? My point is that if QI wants to be a science, it needs to be subject to the same rules that all other science is guided by. Since we do not even know how much money we are spending on the ubiquitous QI efforts (likely hundreds of billions), and since we are not sure what they are accomplishing (see my many prior posts on the lack of validity of current claims in VAP prevention), we need to pause and ask ourselves whether the cheering alone justifies such an investment. I hate to say it, but can we really trust those with most to lose, financially and politically, if in reality QI does little more than lather the masses, to be the oracles of truth about the results of these efforts? The cognitive biases alone should disqualify them from being the arbiters of their own success. So, if we do not want to continue to indulge the principle of diminishing returns in QI, we need to take a sober look at what we have invested and what this investment has accomplished. Then and only then can we claim to practice evidence- rather than politics-or dogma-based policy.