But I will not agree with you about crude ARR. In this design it doesn't mean much, and indeed it was negative: crude mortality was higher in screening group than in control (3.2 vs. 3.1%). Adjusted RR is primary outcome in designs like this where calendar time is important covariate. I was really stunned by graphical abstract where both crude risks and adjusted RR are reported. At first glance they seem incompatible, unless you dive deep into the design.
And adjusted absolute risk reduction doesn't seem clinically non-significant to me: 0,85 aRR with ~3% baseline risk translates into 0.5% absoulte risk reduction, which means NNT ~200 (for all inpatients, not only sepsis patients!).
Still there are limitations (inherent to cluster RCT), and I'm personally sceptical about eHR alerts etc. Especially their external validity: wil eHR alert work in the context of other multiple and boring alerts? How alert fatigue can be assessed and influence the effectiveness? And I think that the most important argument is the absence of obvious reasons why normal (patient-level) ranomization was not performed -- it was completely feasible and will preclude most of the methodological critique.
“well trained doctor”….such an irony given the deep DEI wokness entrenched in most if not all medical schools and most staff….we sure would need some sort of objective analysis of the potential to discover sepsis…at times a very challenging condition. If detected early you could limit to delayed effects and future mortality. Reminds me of those who overlooked rupture appendix which had wall itself off. In this time of immune compromise from the covid vac it becomes even more critical to be tested rather than wait until a seasoned MD/DO puts their hands on the patient. The question about in-hospital vs overall mortality is s good one…by using in-hospital mortality you are only taking responsibility for actions at your hospital….and the transfer MO to wash your hands of others who may or may not be your well trained doctor. I believe all healthcare studies should have a simple question added to the intake…covid vac status…maybe a better indicator whether sepsis is on the differential. JAMA studies…well they were often Faucci mouthpieces…like the New England Journal….dropped both!
OMG, this sepsis stuff is the bane of the Hospitalist’s existence! First of all, you know you’re in trouble when you can’t even agree on a definition. Then a predictable trade-off exists between under and overdiagnosis when protocol driven patient outcomes are deployed. Allow me to state with metaphysical certitude that sepsis alerts will not improve patient mortality at 90 days. It has just become an exercise in wack a mole EHR nonsense. There is this thing called clinical assessment at the bedside which should take priority as the mainstay of diagnosis and treatment.
As a night nurse I frequently report a new sepsis alert to our residents. It rarely makes much difference because in most cases we are already doing what we would anyway. They might throw in a few labs.
Well said. I am glad to be long retired when I see stuff like this being taken seriously. A first year medical student will have no trouble diagnosing sepsis from vital signs and clinical examination.
Dr. Locke commented on design weakness but I would be interested in opinions of the effect of Covid on this study as the timeframe would include hospitalizations for a disease not well understood in that timeframe in terms of critical illness management.
For clarity, the adjusted estimate is the correct one to use. Time period can be a confounder when clusters are randomized through time - an inherent weakness of the design. See https://www.bmj.com/content/350/bmj.h391 . But, knowing it's a weakness of the design does not mean it's OK to use the unadjusted estimate (which amounts to just ignoring the weakness). Same reason it'd be to ignore a confounder in in a plain retrospective study. The critique is that it's possible they did not adequately adjust for this confounder (temporal trends) and that the design does not inherently protect against it like a standard RCT would.
Also worth noting that on the absolute effect size - 0.5%ish using the correct effect estimate - uses all patients hospitalized as the denominator, not all patients in whom the alert triggered. (There is an interest detail that outcomes seemed to improve in the other patients - worrisome). All this is to say - when appraising the "clinical significance" of the alert ... if a popup being active really can keep 1 out of every 200 patients from dying in the hospital - that's mighty important.
Yes, it's possible that those people all die in the 75 days after discharge.. but it's not super likely, and all evidence is limited. A fixation on '90-d mortality or nothing' is a big reason trials of acutely ill patients have stagnated - there's only so much attributable risk an intervention can modify, and we too often erroneously toss interventions with detectable marginal benefits. Care of acutely ill adults is mostly about improving on the margins, not slam dunks - there just aren't many.
So interesting. Thank you for this article. Retired inpatient coder, and love to read your Substack. Survived bacterial meningitis last year. Very fortunate to be alive due to the wonderful care from Vanderbilt.
Another example of “ statistical significance” being meaningless. Another example to consider ARR vs RRR. Most drugs we use today ( think statins in primary prevention) fail to result in clinically meaningful results.
The scary fact..."Median age was 59". Seems like a very young average age to be in the hospital. It would be nice to be able to trust medically related articles in well known journals. That will be a long, long road for me to travel. But it is an interesting study.
The JAMA editorialist is the one who invented the idea that you can diagnose sepsis from EHR. He’s also one of the Sepsis-3 fathers.
The essence of Sepsis-3 is that sepsis is a disease (?) that can be diagnosed by a prognostic score (SOFA). Then they used EHR to validate the “diagnosis by prognosis” construct in large datasets.
I call it “The Angus Method”. Dr Angus is the JAMA editor. He is personally invested in the idea. It explains why this paper has such a strange spin.
I comment about it in this chronicle and in many other posts. I invite you.
Thank you again for highlighting the worrying state of critical care research.
“I think treatment of critical illness should always be measured against 90-day-mortality (and analyzed by RR rather than HR).”
Benefits/advantages to analyzing 90d mortality by RR rather than HR: I would welcome any further insight/nuance anyone may be able to provide here!
Thank you for your opinion, Dr. Cifu!
This article also has confused me a lot.
But I will not agree with you about crude ARR. In this design it doesn't mean much, and indeed it was negative: crude mortality was higher in screening group than in control (3.2 vs. 3.1%). Adjusted RR is primary outcome in designs like this where calendar time is important covariate. I was really stunned by graphical abstract where both crude risks and adjusted RR are reported. At first glance they seem incompatible, unless you dive deep into the design.
And adjusted absolute risk reduction doesn't seem clinically non-significant to me: 0,85 aRR with ~3% baseline risk translates into 0.5% absoulte risk reduction, which means NNT ~200 (for all inpatients, not only sepsis patients!).
Still there are limitations (inherent to cluster RCT), and I'm personally sceptical about eHR alerts etc. Especially their external validity: wil eHR alert work in the context of other multiple and boring alerts? How alert fatigue can be assessed and influence the effectiveness? And I think that the most important argument is the absence of obvious reasons why normal (patient-level) ranomization was not performed -- it was completely feasible and will preclude most of the methodological critique.
“well trained doctor”….such an irony given the deep DEI wokness entrenched in most if not all medical schools and most staff….we sure would need some sort of objective analysis of the potential to discover sepsis…at times a very challenging condition. If detected early you could limit to delayed effects and future mortality. Reminds me of those who overlooked rupture appendix which had wall itself off. In this time of immune compromise from the covid vac it becomes even more critical to be tested rather than wait until a seasoned MD/DO puts their hands on the patient. The question about in-hospital vs overall mortality is s good one…by using in-hospital mortality you are only taking responsibility for actions at your hospital….and the transfer MO to wash your hands of others who may or may not be your well trained doctor. I believe all healthcare studies should have a simple question added to the intake…covid vac status…maybe a better indicator whether sepsis is on the differential. JAMA studies…well they were often Faucci mouthpieces…like the New England Journal….dropped both!
OMG, this sepsis stuff is the bane of the Hospitalist’s existence! First of all, you know you’re in trouble when you can’t even agree on a definition. Then a predictable trade-off exists between under and overdiagnosis when protocol driven patient outcomes are deployed. Allow me to state with metaphysical certitude that sepsis alerts will not improve patient mortality at 90 days. It has just become an exercise in wack a mole EHR nonsense. There is this thing called clinical assessment at the bedside which should take priority as the mainstay of diagnosis and treatment.
As a night nurse I frequently report a new sepsis alert to our residents. It rarely makes much difference because in most cases we are already doing what we would anyway. They might throw in a few labs.
Well said. I am glad to be long retired when I see stuff like this being taken seriously. A first year medical student will have no trouble diagnosing sepsis from vital signs and clinical examination.
JAMA has been pathetic for over a decade now and needs to be abandoned.
Dr. Locke commented on design weakness but I would be interested in opinions of the effect of Covid on this study as the timeframe would include hospitalizations for a disease not well understood in that timeframe in terms of critical illness management.
For clarity, the adjusted estimate is the correct one to use. Time period can be a confounder when clusters are randomized through time - an inherent weakness of the design. See https://www.bmj.com/content/350/bmj.h391 . But, knowing it's a weakness of the design does not mean it's OK to use the unadjusted estimate (which amounts to just ignoring the weakness). Same reason it'd be to ignore a confounder in in a plain retrospective study. The critique is that it's possible they did not adequately adjust for this confounder (temporal trends) and that the design does not inherently protect against it like a standard RCT would.
Also worth noting that on the absolute effect size - 0.5%ish using the correct effect estimate - uses all patients hospitalized as the denominator, not all patients in whom the alert triggered. (There is an interest detail that outcomes seemed to improve in the other patients - worrisome). All this is to say - when appraising the "clinical significance" of the alert ... if a popup being active really can keep 1 out of every 200 patients from dying in the hospital - that's mighty important.
Yes, it's possible that those people all die in the 75 days after discharge.. but it's not super likely, and all evidence is limited. A fixation on '90-d mortality or nothing' is a big reason trials of acutely ill patients have stagnated - there's only so much attributable risk an intervention can modify, and we too often erroneously toss interventions with detectable marginal benefits. Care of acutely ill adults is mostly about improving on the margins, not slam dunks - there just aren't many.
So interesting. Thank you for this article. Retired inpatient coder, and love to read your Substack. Survived bacterial meningitis last year. Very fortunate to be alive due to the wonderful care from Vanderbilt.
Another example of “ statistical significance” being meaningless. Another example to consider ARR vs RRR. Most drugs we use today ( think statins in primary prevention) fail to result in clinically meaningful results.
The scary fact..."Median age was 59". Seems like a very young average age to be in the hospital. It would be nice to be able to trust medically related articles in well known journals. That will be a long, long road for me to travel. But it is an interesting study.
Very important post Dr Cifu. Thank you.
The JAMA editorialist is the one who invented the idea that you can diagnose sepsis from EHR. He’s also one of the Sepsis-3 fathers.
The essence of Sepsis-3 is that sepsis is a disease (?) that can be diagnosed by a prognostic score (SOFA). Then they used EHR to validate the “diagnosis by prognosis” construct in large datasets.
I call it “The Angus Method”. Dr Angus is the JAMA editor. He is personally invested in the idea. It explains why this paper has such a strange spin.
I comment about it in this chronicle and in many other posts. I invite you.
Thank you again for highlighting the worrying state of critical care research.
https://thethoughtfulintensivist.substack.com/p/five-centuries-of-the-angus-method?r=20qrtz
Looking forward to diving into your work.
Ty. Appreciate you doing the deep dive.