Andrew Foy says there is efficacy-effectiveness gap: is he right?

Guideline directed therapy in heart failure

One of my favorite pieces in recent weeks in Sensible Medicine is by Andrew Foy. He describes a randomized controlled trial of a popular cardiology drug that had a massive benefit, and a real world observational study from Ontario province that suggests the actual impact of the therapy is negligible. If you haven't read it, read it here.

A Cautionary Tale of Clinical (Mis)translation

The phenomenon Foy believes is happening— an efficacy-effectiveness gap— runs rampant in cancer medicine. Some new, $200,000 a year therapy eeks out a 1.5 month PFS or OS benefit in a randomized study. Oncology KOLs and the fellows that aspire to be them jump out of their chairs applauding. What few discuss is that the patients in these randomized studies are different than the average patients in our clinic. Patients on trial are younger, often as much as a decade. They have better performance status, which means they're more active around their house. They tend to have fewer comorbidities. Their liver function and kidney function are generally preserved. And they've waited to be enrolled on study without the disease exploding— they may have more indolent biology. Even in this setting, there are toxicities, but we're always reassured that they are manageable. That's a nice word to say when you're not the one suffering.

Some look at these studies and argue the median is not the message. Although the median person has a one-month improvement in survival, look at the tail, there are some people living 18 months or more. Problem I have with this argument is: look at the other tail. The group of people not assigned the magic new drug, assigned the older medicine, sometimes assigned placebo also has a tail. Is your tail that much higher? Are you sure? How many people are still at risk? Is this a statistically robust conclusion?

And there's the other argument. It's entirely possible that somebody does have seven or 8 months of life added by taking the drug, but if the median is 1, wouldn't that mean there's somebody whose life is shortened?

If you survey gamblers coming out of a Vegas casino, you may find that on average, they lose a little bit of money. If you find someone who scored big, then probably is someone who lost big. Of course this is the case with cancer drugs, we have all taken care of people who've experienced a grade 5 adverse event—which is death. The challenge with medicine is that it is difficult to identify the individual counterfactual.

Inevitably, with time, cancer therapies become used in populations not originally studied. Older people, people with comorbidities, people with second cancers, people with diminished kidney function. Almost always the benefits are eroded and not enhanced. The toxicities become more formidable. People experiencing them do not always use the word manageable, instead they replace that word with an expletive.

The heart failure community was quick to criticize Andrew Foy. They continue to pound on the drum of guideline directed therapy— the bizarre idea that it is in the best interest of someone with newly diagnosed heart failure to start on all of the medicines simultaneously, rather than sequentially.

Often in real clinical practice, this poses pragmatic challenges. Some people are overwhelmed by the number of tablets. Others struggle with diminished blood pressure, and there is a limit on how much these drugs can be pushed. A patient ends up on a homeopathic dose of four medicines rather than a disease-modifying dose of one or two.

Remember, the randomized studies supporting approval often do not support diminished doses, and don’t support starting all at the same time.

One argument made in response to Foy is that randomized controlled trials generate on average treatment effects, but we can use covariates to create individualized treatment effects. You can predict someone's relative and absolute risk reductions, which do not reflect the average relative and absolute risk reductions.

We should remember this is a hypothesis in need of empirical verification. The way you would verify it is to run a large randomized study, generate these subgroup specific effect sizes, and then pick a few subgroups and run the randomized study again to see how closely you predict the actual effect. If you don't do this, you are a tremendous risk of overfitting.

You could also split your data set.

Or you could generate estimates for the people on the edges of the study and then test those estimates formally in a pragmatic randomized study of people ineligible for your trial. That would be my preferred option.

But anyway you slice it. I have not seen any of these external verifications and as such it's just a convenient story without scientific backing.

Is the guideline directed therapy paradigm good? Well that depends on for whom. For the manufacturers of the therapies, it is a windfall. How do you sell a drug that is an add-on drug with additive side effects in a marketplace that has many effective, generic options. Creating a mantra that starting them all together is preferable to starting with the tried and true drugs and expanding, if and only if they are tolerated, is a great marketing strategy. But the burden of proof is on he who lays charges. Run a study where starting all of them is compared against starting on an ace or an ARB and cranking that up and then moving from there, logically and sequentially. That has never been done. Instead some cling to observational studies that compare people who can tolerate multiple drugs against those who can tolerate only one or two. But you don't need to be a rocket scientist to know these are different people.

It's true that most medical care is not supported by randomized evidence. It's also true that you don't need randomized evidence to change practice, or approve new therapies and there are many examples of such products. But there is also a juggernaut of people encouraging the use of medical practices that are incredibly costly, of dubious efficacy, and that generate massive profits. If patients had skin in the game and had to use their own money, we would have a slew of third parties evaluating this evidence, skeptically and making prudent recommendations. But because there is little skin in the game, and the money is siphoned off public funds, the only answer is to give more medicine, earlier, in combination, and we have a legion of KOLs happy to endorse that view— evidence not required.