Quality measures: Process, outcome, or both?

In the last week I wrote about our quality improvement, or QI, efforts in healthcare. And although there is a burgeoning field representing itself as the "science" of QI, I question much of its scientific validity. As always, VAP is my poster child for these discussions, where neither the definition of the condition itself nor its prevention efforts are subject to much scientific scrutiny. This makes VAP have a surreal, ghost-like quality: now you see it, now you don't. And this alone makes it difficult to assess prevention efforts. Much as in the heated mammography debate, where passionate anecdote prevails, the sanctity of the QI rubric blunts the usual critical approach to the data.

So, the central point that I made in this post was essentially to devalue the VAP eradication efforts as not grounded in solid scientific evidence. What has occurred to me, however, is that this position may be in fact at odds with a realization I blogged about here and here, wherein I agreed with Dan Arieli's suggestion that outcomes in the real world, where they are influenced by so much randomness, are not the thing to reward. It would be much more rational to reward best efforts at best results, thus the process rather than the outcome. So, here is the apparent contradiction: On the one hand I agree that outcomes may be too unpredictable, being that they are influenced by too many factors that are not in our control, yet I am also advocating that we start measuring such outcomes as antibiotic use associated with VAP and its reduction. What gives?

Well, on the one hand, I am OK with contradiction; life is full of instances where we have to hold conflicting information and feelings together. But as a scientist it is my predisposition to analyze (which literally means splitting into smaller, more manageable chunks), so I have given this ostensible paradox more thought. What I came up with is that measuring process is the right thing to do, but only under very specific conditions. Avedis Donabedian, who is considered the father of quality science, introduced the triad of structure-process-outcome as the backbone of quality science. This relationship certainly lends validity to the "process" metrics as surrogates for "outcome." But the condition that has to be met is that there be an actual correlation between the process and the said outcome. If there is no such solid correlation, then we are simply going through the motions, doing a rain dance to cause rain.

So, what I have said about VAP prevention in particular is that we are nowhere near being able to say that the recommended processes correlate with any changes in meaningful clinical outcomes. And because the data on these interventions are so weak, throwing massive resources behind implementing them is irrational and resembles religious fervor more than scientific pragmatism.

It is entirely understandable that we would jump on this bandwagon so rapidly, given the magnitude of harm in our healthcare system combined with the need to reign in the healthcare spending. But there is a more subtle point to be made here too. It relates to the fertile soil of our American psyche, where doing something is always perceived as better than thinking about our course of action, which is frequently referred to with contempt as "doing nothing." In the end, this crisis response mentality is good in a crisis, but potentially detrimental in the long term: we are unlikely to be altering meaningful outcomes, and we are spending billions of dollars on interventions lacking evidence.

So, I stand behind both of my assertions and maintain that they are not mutually exclusive. Yes, outcomes are subject to much randomness; yes, processes known to alter these outcomes are the sensible measures of our efforts to improve quality; and yes, these processes need first to be rigorously validated for their impact on the outcomes in question. Anything short of this pathway is not just a waste of our collective resources, but a manipulation of the public trust. And that is as far from the intent of science as it can get.

Read More

Health surveys: Run the other way!!!

Sometimes when I get an unsolicited call about answering survey questions, I feel a karmic obligation to participate; after all, if everyone said no to everything, I would not have any data to analyze. So, for this very reason, I just got off the phone with a poor young woman conducting a survey who called me randomly. The survey had to do with healthcare delivery, and she had no idea what she was getting herself into. First, I queried her whom the survey was for. She proceeded to tell me that she did not have that information specifically, but gave me a general idea of who the customers tend to be. Then she launched into the survey questions.

Now, I realize that they all have to ask the same questions the same way in order not to bias the data. But man, who writes these questions? "What would you say is the reputation of the cardiac surgery program at thus-and-such a hospital in your area: a). good locally, 2). good locally and state-wide, 3). good locally, statewide and regionally, 4). good locally, statewide, regionally and nationally, 5). good locally, statewide, regionally, nationally and internationally, or 6). not good at all?" Well, what the heck do you mean by "reputation"? You mean what is the gossip about Dr. Smith in my community? Or do you mean what kind of care they provide in terms of timeliness, evidence, shared decision making, post-operative complications, what? Then came "if you or your family member needed a cardiac procedure, how comfortable would you be going to this facility? 1). very comfortable, 2). somewhat comfortable, 3). somewhat uncomfortable, and 4). not at all comfortable?" How the heck should I know? I have not researched all the local facilities, I have not checked on their outcomes, I have not interviewed all of their cardiac surgical teams (yes, including anesthesia), I do not know what their infection control track records are, and, most importantly, how willing they are to treat me as an individual rather than a source of income. And then, for every hospital she mentioned (and there were quite a few), she went through the same litany of meaningless questions.

And then she asked me if I am familiar with some of the well-known quality-rating organizations. And she included US News and World Report Hospital Ratings! And I don't even believe the CMS got it anywhere near right!!! Oy! What do the answers to these questions from someone who is not steeped in the data mean anyway? If researchers and providers have not arrived at the appropriate metrics for quality, how meaningful are the lay public's opinions on these matters?

And finally, a group of questions that let the cat out of the bag as to the purpose of the survey. She told me a story first, of a large regional medical center in the area building a new multi-million dollar state-of-the-art cardiac care facility. Sexy new equipment, individual patient rooms, targeted and individually-tailored treatment plans, all the buzzwords of the brave new world of medicine. And then she asks me would I be comfortable going to this facility. What am I supposed to say? I have no idea! How do I tell this poor child that the questions are written in an absurd way and smack of marketing? How do I explain to her that this facility will probably need to recoup their capital investment, and, therefore, has a conflict of interest when it comes to caring for me? How do I teach her that this is the problem with American medicine, this very over-reliance on reputations and expertise to tell us to over-indulge in interventionism at the expense of our health and budgets?

Anyway, I will not belabor this further. My advice to survey fielders: If you want to market to the gullible, go ahead and call people randomly and ask your market-building question. And if a person tells you she is a physician and a health services researcher to boot, run, don't walk, the other way.

Read More

Tipping a sacred cow: QI under the microscope

So much media and journal space has been devoted to financial conflicts of interest, particularly within and related to pharma and device manufacturers, that to write any more about it may be redundant. On this site we have also intermittently addressed COI from other perspectives, such as financial interest of the members of the American College of Radiology in maintaining mammography screening status quo, thinly veiled in its own version of the pernicious "death panel" language. We have also spoken a bit about the non-financial COI. And even though we are so very much aware of COI's potential to lurk around every corner, there are still some surprises.

Take the sacred cow of "quality improvement" in healthcare. Even the name, much like the "pro life" moniker, suggests that it is untouchable in its purity and nobility of purpose. So necessary is it because of the epic magnitude of morbidity and mortality attributed to healthcare itself, that the billions of dollars spent on it seem unquestionably justified. Indeed, much like our public education system, the QI movement garners higher and higher allocations simply due to the sheer face validity of the assumption that more of it is better. And the most fascinating aspect is that, in our current zeal for sensible economic allocation through evidence, QI, much like education, appears immune to scrutiny. This is the very definition of politics driving policy.

I return to the case of ventilator-associated pneumonia, or VAP, as the poster child for this movement. I have already alluded to the fact that definitionally VAP is a slippery slope: its diagnosis varies based not only on the tools used to diagnose it, but also depending on who is doing the diagnosing. Yes, indeed, what one clinician calls VAP another may call absence of VAP. I have also dissected the weak evidence behind some of the strongest purportedly evidence-based recommendations aimed at VAP prevention. But what if VAP itself is the wrong endpoint? What if we are spending untold dollars and other resources on a futile pursuit?

Do you feel yourself bristling yet? If you said "yes", it is a normal response I get from my colleagues and people who read my scholarly papers. Because how can anyone be against QI? Well, I am not against QI. I am simply against sanctifying QI as a sacred cow and thus shielding it from a sensible and rational evaluation.

So, if you are over the initial shock, allow me to explain myself. I am sure you have heard of surrogate endpoints. Here is a definition from Wikipedia:

In clinical trials, a surrogate endpoint (or marker) is a measure of effect of a certain treatment that may correlate with a real clinical endpoint but doesn't necessarily have a guaranteed relationship. The National Institutes of Health (USA) defines surrogate endpoint as "a biomarker intended to substitute for a clinical endpoint".^[1][2]

Surrogate markers are used when the primary endpoint is undesired (e.g., death), or when the number of events is very small, thus making it impractical to conduct a clinical trial to gather a statistically significant number of endpoints. The FDA and other regulatory agencies will often accept evidence from clinical trials that show a direct clinical benefit to surrogate markers. ^[3]

This begs the question of what constitutes a "real" clinical endpoint. Well, in my simplemindedness I think of them as endpoints that matter to the patient or in the long run. So, death, disability, quality of life, functionality, these are the real endpoints. Something that alters one's life or threatens it is a real endpoint. Thus, blood pressure and cholesterol are surrogate endpoints, since they usually, but not always, correlate with the risk of a myocardial infarction or death. But what if such a correlation did not exist? Furthermore, what if a cholesterol level was measured with, say, tea leaves, and therefore was subject to a tremendous variation in detection? Would we then spend hundreds of billions of dollars on trying to alter this factor or would we calmly and rationally walk away and look for something that truly impacts the real outcome of a heart attack or death? I think I am making my point fairly clearly.


Let me explain why I think that VAP is but a surrogate outcome, and, given its diagnostic challenges, not a sensible one in the least. VAP by definition occurs in patients on mechanical ventilation (breathing machine), whose quality of life is fairly badly damaged in the short term. The literature would suggest that not all VAP impacts mortality adversely, but some forms of VAP indeed do, particularly VAP that develops late in the course of illness. So in this VAP does correlate with a real endpoint. Also, there is very little doubt that getting VAP prolongs one's dependence on mechanical ventilation, and increases the duration of the stay in the ICU and hospital overall. So, this can be considered not a very good, albeit real, outcome. An additional point to remember is that VAP engenders the use of additional, usually broad spectrum, antibiotics, putting both the individual and the society at risk for such unwanted consequences as the emergence of highly resistant microorganisms.


So, even though VAP is a surrogate endpoint, it certainly seems to fit the bill for something we would want to prevent. But here is the monkey wrench in this argument: what seem to be great surrogate endpoints do not always end up correlating with clinical reality. The association of VAP with morbidity and mortality has been detected in mostly retrospective observational studies. Trials of VAP prevention rarely, if ever, report any endpoint other than VAP. And, given how elusive VAP diagnosis is, there is plenty of room for gamesmanship so pervasive in the real world to make any data fit our preconceived hypotheses and political needs. 


So, what is my point? My point is that if QI wants to be a science, it needs to be subject to the same rules that all other science is guided by. Since we do not even know how much money we are spending on the ubiquitous QI efforts (likely hundreds of billions), and since we are not sure what they are accomplishing (see my many prior posts on the lack of validity of current claims in VAP prevention), we need to pause and ask ourselves whether the cheering alone justifies such an investment. I hate to say it, but can we really trust those with most to lose, financially and politically, if in reality QI does little more than lather the masses, to be the oracles of truth about the results of these efforts? The cognitive biases alone should disqualify them from being the arbiters of their own success. So, if we do not want to continue to indulge the principle of diminishing returns in QI, we need to take a sober look at what we have invested and what this investment has accomplished. Then and only then can we claim to practice evidence- rather than politics-or dogma-based policy.

Read More

Whose perspective?

After a long hiatus filled with travel, work and lack of inspiration to write anything, I have chosen this arguably hottest day of the year to venture forth again. But I will make this brief, as it seems that everything that needs to be thought and said has already been thought and said. Yet who is listening?

Anyhow, to suspend my natural cynicism, I want to talk about perspective. No, not the perspective that makes parallel lines converge in the distance, but the one that gets lost in many of our political, civic, business, and, yes, even scientific discussions. I am talking about my perspective, your perspective, societal perspective, etc. I was inspired to write this because of these tweets by Gary Schwitzer to Kaiser Health News about a story on their web site:

Linking out to the story, I learned that a consulting arm of Disney is teaching hospitals about hospitality. Since there is going to be a financial incentive for these hospitals to deliver good customer service, many are feeling that an investment in this type of training will help them maximize these new reimbursements. Hurrah and ta-da! 

Well, Gary likes to burst these one-sided bubbles, and so he rightfully asked about the costs. What was baffling to me was the response by the KHNews who did not seem to appreciate the importance of reporting the costs or the various perspectives that these costs represent. So, this seemed like a teachable moment, and here is the teaching.

In outcomes research, we are always interested in understanding the perspective for both the costs and the benefits of interventions. In health outcomes these perspectives are broadly represented by the patient, the provider, the hospital, the payer, the employer, the manufacturer, the society, to name a few. These are just some of the examples of the usual stakeholders involved in healthcare decisions. Because our healthcare is such a fragmented disaster, many of these perspectives find themselves at odds with one another. Just think of the patient who wants to get what she perceives as a life-saving treatment that in reality has a 1% chance of helping at a cost of $600,000 per treatment course. From her perspective, since she is insured, this investment is well worth the cost. For a payer, however, this means $600,000 (multiply this by 100 in order to determine the cost to save 1 life) that cannot be spent on something else that can help more people more predictably. And if this payer is the taxpayer, the societal perspective enters the picture, where we have to decide what amount of money is worth spending on possibly saving one life -- is $60 million reasonable? Perhaps. But these are not simple questions, and, as such, do not have simple answers. In addition, all conversations that we hear or engage in have multiple perspectives. This is why a black-and-white approach is so divisive: it generally emphasizes two diametrically opposed perspectives. 

So, next time you hear about death panels or Mickey Mouse teaching hospitals how to maximize their revenue, consider the broader implications from may different perspectives. Chances are you will find yourself agreeing with more than one point of view. And when this happens, you will know that you have learned an important lesson and can now start engaging in more nuanced and thus productive debates, many of which will shape our society's future.

h/t to @garyschwitzer for this KHN story      

Read More