Sorry, Dr. Matson, respectfully disagree. Your assertion that A-lines exist for "vasopressor titration, timely detection of hypotension, and safe arterial sampling" better not be true. They should be means to the larger end of improving patient outcomes. All, if doing their job, should be reflected in 1m survival, not in provider convenience. If invasive, painful, potentially dangerous A-lines offer benefits only to providers, that's worth knowing and dealing with. Noninferiority designs have indeed been abused—smart, important point—but IMHO not in EVERDAC, a contributory and revealing study.
Interesting question, good one. I'll narrow it down to what I assume you're asking: In the ICU (where mortality is high enough to readily test it) invasive, resource intensive, potentially risky, process-based technologies should clear the mortality bar, yes. That's not been the tradition but it should be, IMHO. And when they're shown not to, they should be withdrawn.
My hesitation with the design is simply that a ~3% difference in mortality either direction would be practice-changing in most ICU contexts, yet within EVERDAC’s noninferiority margin it would still be deemed “non-inferior.” I don’t place them but if a trainee approaches me to do one I suppose I could imagine patient centered indications but EVERDAC didn’t even measure patient discomfort in a balanced way or vasopressor dose… which I would have loved to have as ammunition to disarm that trainee and eschew more intervention
Point taken. OTOH the trial you chose enrolled >1000 and found an absolute 2.6% difference favoring no A-line. The noninferiority claim therefore doesn't seem either misleading or ambiguous. I assume you sought out the margins in the Methods section, which is smart, and your point seems like a good one when applied to findings from many other non-inferiority trials, which can mislead. But this one strikes me as on point.
True, point also taken here. Probably would land better with other NI trials. I guess I was miffed by the design… I’m no a line evangelist but felt the bias in design jumping off the page… but I share their perspective and just felt the case could have been strong manned
Well the trial could report all that data and leave you, the reader, to your own conclusions. In fact, that data might be gleaned from the data presented. I don't think there are binary answers available to us in most medical issues. What if the study showed a 2-3% marginal difference in death rates but also a greater risk of sepsis and longer hospital stays? Is 2-3% really significant? Would a repeat trial perhaps NOT show a 2-3% difference IN THE OTHER DIRECTION? Most studies are never replicated! Medicine is a field chock full of epistemic uncertainty. It always will be.
Scott, this is a great look at the 'messy' side of research incentives. I’ve found that the non-inferiority margin is usually where the real politics of a trial happen—it’s that tricky gap where a 'terrific guide' like the JAMA one meets the reality of funding and deadlines.
For the people I talk to 'on the street,' the difference between 'better' and 'not worse' is huge, but it often gets lost in the noise of the headlines.
Do you think the current regulatory environment actually punishes researchers who take the risk on a superiority design?
Would love to hear your take on whether we're prioritizing marketing speed over actual clinical leaps."
Dr. Vinay Prasad's recent refusal to review Moderna's mRNA influenza vaccine was very appropriate given the methods of the study comparing the mRNA influenza vaccine to a less effective standard flu vaccine in patients older than 65. Plus, there is no third arm of the study of folks that don't receive vaccines. Why not? Maybe you would find no difference in cases if you study it, so don't study it, to make your new drug/vaccine look better than it really is. Vinay Prasad's departure once again is a huge failure. It is so obvious the system is corrupt. mRNA may have a role in the future, but injecting it into muscles (that are blood-enriched and distribute to whole body well) can turn the body into a pathogen generating factory with adverse consequences like a 280 fold increase in interferon causing tissue damage for mRNA SARS-COV2. But the $70 billion Moderna generated and the $100 billion Pfizer/BNT generated pulls a lot of weight in Washington DC. I look forward to hearing Vinay Prasad's perspecitive when he is back on his own platforms. We need more gatekeepers like Prasad as opposed to enablers!!
Cheating in study design or just straight up fraud in science is a clear and present danger… likely rampant and would be even more so if the rewards were greater
The antiquated academic mantra (incentive) “publish or perish” significantly dilutes the quality of research in all fields of study—medicine being no exception. Perhaps we have too many research institutions, many of which should focus more on teaching. Perhaps the increasing number of non- inferiority studies is an indicator that too many federal dollars are made available. Can the country (taxpayers) continue to afford research that provides nebulous outcomes? My radical idea is that ALL federal funding of research be put on hold for 5-10 years; instead, promote private and corporate funding of research with tax credit incentives.
In my view, limiting the pool of dollars won’t change these incentives. A smaller pool of dollars will concentrate the research funding in those researchers best at optimizing for these incentives and by any current academic metric these researchers were successful in this endeavor. The problem isn’t the spending of dollars, it’s the incentives that drives the recipients. More or less money with better incentives will mean better research. Much that’s being done is good, but much could be better.
Love the take on EVERDAC. It helps explain why all the ICU docs I talked to it about were unimpressed and didn’t feel like it changed anything. Feels like more studies are like that now.
Agree. Industry incentives are overt and obvious. Nice to have a post that shines a light on academic incentives.
I would say that my generation had “hard outcomes” drilled in as the de facto gold standard, but this is a good reminder that hard outcomes, upon which an intervention has no hope of impacting, will simply make “non inferiority” inevitable. This lesson should come in a box-set, paired with Dr. JMM’s oft-issued refrain where composites of outcomes that move in opposite directions also make non-inferiority the default result.
I wonder if the heuristic might be, if you know the outcome before that trial even starts, then that’s a bogus study.
I don't think that "arterial line manufacturers" did the study cited. There was no apparent bias in the design or outcome of the study in favor of arterial lines being used.
Right. But I commended this post for speaking to academic incentives (not industry). It wasn’t a manufacturer designing a study that could only be positive (ie non inferior), but in this case, the investigators.
Sorry, Dr. Matson, respectfully disagree. Your assertion that A-lines exist for "vasopressor titration, timely detection of hypotension, and safe arterial sampling" better not be true. They should be means to the larger end of improving patient outcomes. All, if doing their job, should be reflected in 1m survival, not in provider convenience. If invasive, painful, potentially dangerous A-lines offer benefits only to providers, that's worth knowing and dealing with. Noninferiority designs have indeed been abused—smart, important point—but IMHO not in EVERDAC, a contributory and revealing study.
Does every intervention in medicine need to clear the 1m mortality bar?
Interesting question, good one. I'll narrow it down to what I assume you're asking: In the ICU (where mortality is high enough to readily test it) invasive, resource intensive, potentially risky, process-based technologies should clear the mortality bar, yes. That's not been the tradition but it should be, IMHO. And when they're shown not to, they should be withdrawn.
My hesitation with the design is simply that a ~3% difference in mortality either direction would be practice-changing in most ICU contexts, yet within EVERDAC’s noninferiority margin it would still be deemed “non-inferior.” I don’t place them but if a trainee approaches me to do one I suppose I could imagine patient centered indications but EVERDAC didn’t even measure patient discomfort in a balanced way or vasopressor dose… which I would have loved to have as ammunition to disarm that trainee and eschew more intervention
Point taken. OTOH the trial you chose enrolled >1000 and found an absolute 2.6% difference favoring no A-line. The noninferiority claim therefore doesn't seem either misleading or ambiguous. I assume you sought out the margins in the Methods section, which is smart, and your point seems like a good one when applied to findings from many other non-inferiority trials, which can mislead. But this one strikes me as on point.
True, point also taken here. Probably would land better with other NI trials. I guess I was miffed by the design… I’m no a line evangelist but felt the bias in design jumping off the page… but I share their perspective and just felt the case could have been strong manned
Well the trial could report all that data and leave you, the reader, to your own conclusions. In fact, that data might be gleaned from the data presented. I don't think there are binary answers available to us in most medical issues. What if the study showed a 2-3% marginal difference in death rates but also a greater risk of sepsis and longer hospital stays? Is 2-3% really significant? Would a repeat trial perhaps NOT show a 2-3% difference IN THE OTHER DIRECTION? Most studies are never replicated! Medicine is a field chock full of epistemic uncertainty. It always will be.
Scott, this is a great look at the 'messy' side of research incentives. I’ve found that the non-inferiority margin is usually where the real politics of a trial happen—it’s that tricky gap where a 'terrific guide' like the JAMA one meets the reality of funding and deadlines.
For the people I talk to 'on the street,' the difference between 'better' and 'not worse' is huge, but it often gets lost in the noise of the headlines.
Do you think the current regulatory environment actually punishes researchers who take the risk on a superiority design?
Would love to hear your take on whether we're prioritizing marketing speed over actual clinical leaps."
Dr. Vinay Prasad's recent refusal to review Moderna's mRNA influenza vaccine was very appropriate given the methods of the study comparing the mRNA influenza vaccine to a less effective standard flu vaccine in patients older than 65. Plus, there is no third arm of the study of folks that don't receive vaccines. Why not? Maybe you would find no difference in cases if you study it, so don't study it, to make your new drug/vaccine look better than it really is. Vinay Prasad's departure once again is a huge failure. It is so obvious the system is corrupt. mRNA may have a role in the future, but injecting it into muscles (that are blood-enriched and distribute to whole body well) can turn the body into a pathogen generating factory with adverse consequences like a 280 fold increase in interferon causing tissue damage for mRNA SARS-COV2. But the $70 billion Moderna generated and the $100 billion Pfizer/BNT generated pulls a lot of weight in Washington DC. I look forward to hearing Vinay Prasad's perspecitive when he is back on his own platforms. We need more gatekeepers like Prasad as opposed to enablers!!
https://bariweiss.substack.com/p/science-has-a-major-fraud-problem?r=pbogx&utm_medium=ios
This was in the Free Press today. Related to your point about scientific incentives, and as cautionary tales of maladaptive expressions thereof.
Cheating in study design or just straight up fraud in science is a clear and present danger… likely rampant and would be even more so if the rewards were greater
The antiquated academic mantra (incentive) “publish or perish” significantly dilutes the quality of research in all fields of study—medicine being no exception. Perhaps we have too many research institutions, many of which should focus more on teaching. Perhaps the increasing number of non- inferiority studies is an indicator that too many federal dollars are made available. Can the country (taxpayers) continue to afford research that provides nebulous outcomes? My radical idea is that ALL federal funding of research be put on hold for 5-10 years; instead, promote private and corporate funding of research with tax credit incentives.
In my view, limiting the pool of dollars won’t change these incentives. A smaller pool of dollars will concentrate the research funding in those researchers best at optimizing for these incentives and by any current academic metric these researchers were successful in this endeavor. The problem isn’t the spending of dollars, it’s the incentives that drives the recipients. More or less money with better incentives will mean better research. Much that’s being done is good, but much could be better.
Love the take on EVERDAC. It helps explain why all the ICU docs I talked to it about were unimpressed and didn’t feel like it changed anything. Feels like more studies are like that now.
Agree. Industry incentives are overt and obvious. Nice to have a post that shines a light on academic incentives.
I would say that my generation had “hard outcomes” drilled in as the de facto gold standard, but this is a good reminder that hard outcomes, upon which an intervention has no hope of impacting, will simply make “non inferiority” inevitable. This lesson should come in a box-set, paired with Dr. JMM’s oft-issued refrain where composites of outcomes that move in opposite directions also make non-inferiority the default result.
I wonder if the heuristic might be, if you know the outcome before that trial even starts, then that’s a bogus study.
I don't think that "arterial line manufacturers" did the study cited. There was no apparent bias in the design or outcome of the study in favor of arterial lines being used.
Right. But I commended this post for speaking to academic incentives (not industry). It wasn’t a manufacturer designing a study that could only be positive (ie non inferior), but in this case, the investigators.
Thank you. I will put this and the JAMA resource into my fall evidence-based practice for nursing course.
I learned something new to me - thanks…!
Well written and I agree with this article.