I'm a family physician. A 86 year old care home resident with moderate dementia I care for had a major fall leading to a severely swollen arm. She was admitted over 3 days as some debate as to whether fracture or not- eventually decided not. During admission she was found to be in Atrial Fibrillation so discharged on NOAC. The HASBLED score gives her only one point ie >65. No account taken of advanced frailty and numerous falls including the cause of the current admission.
To me as a old fashioned doc of 30 yrs clearly looked wrong- felt obliged to speak to her son but we agreed NOAC to be stopped as tiny benefit potential and clearly significant risk.
An error on my part and I forgot to update the care home and they continued NOAC- 3 days later
developmed intractable nose bleeding and had to attend Accident and Emergency. The hospitals action .... must continue NOAC but have 14 days of tranexamic acid....
Any clinicians on here please tell me if I'm wrong but I struggle to think of this as anything other the crass, unthinking tick box medicine.
For ELAN, I see no major practical difference in composite poor outcomes: 29 out of 1000 vs. 41 out of 1000. That means 1930 out of 2000 (96.5%) had good outcomes with either treatment arm. What could matter, however, is if the incidence of death was overwhelming in either treatment arm.
Here is the very simple Bayesian analysis. Let x be the true fraction of patients who will have a primary (bad) outcome under a given protocol. N patients are given that protocol, P have the primary outcome, N-P do not. Assuming a uniform prior for x (that is, we have no idea what x is going in), the "posterior" probability that x has a particular value is p(x) = C x^P (1-x)^(N-P), where C = (N+1)!/(P!(N-P)!). This is what we now know about what x is likely to be (for a specific protocol), given the data that we now have.
Here are the plots of p(x) vs x for N=1000, P=29 (blue) or P=41 (orange):
Now we could quantify the differences between the two curves in various fancy ways, but just look at them. Again, the x axis is the true fraction of patients who will have a bad outcome. Do you want to be given the blue protocol or the orange protocol? Is there really any question in your mind?
The shape is necessarily single-peaked in a simple yes/no case like we have here, and is approximately gaussian once you have decent statistics.
Here's what happens in this case with N=100, P=3 (blue) or P=4 (orange), what you would have likely gotten as a result with 100 rather than 1000 patients per protocol:
Now there's still a slight preference for the blue protocol, but not nearly as pronounced. Clearly we need more data.
And again this is all Bayesian methodology 101, extremely simple, easy to understand, and quickly gives the subjective results you need, without making your brain hurt or allowing for p-hacking.
"The shape is necessarily single-peaked in a simple yes/no case like we have here, and is approximately gaussian once you have decent statistics."
🤷♂️ it is only "necessarily" uni modal if the population from which you sample is not heterogeneous. But that is almost never the case in real life medicine, even if yes/no. For example, gender.
Once your model gets at all complicated, the amount of prior component probabilities you must account for becomes overwhelming.
I'm not unsympathetic but we must be circumspect. Bayesian approach is not a panacea. Nor is frequentist approach what is "holding back" so-called evidence based medicine.
Statistical thinking is necessary but insufficient.
This is completely irrelevant to the question John posed:
"In multiple centers, slightly more than 2000 patients were randomized to either early or later starting of the anticoagulant drugs. A primary outcome occurred in 2.9% of the patients in the early arm vs 4.1% in the later treatment arm. The absolute risk reduction was 1.2%. The relative risk reduction was 30%—expressed as an odds ratio of 0.70. The question for you is what to make of these results."
I have answered the question. Looking at the Bayesian posterior probability curves shows instantly and viscerally that you should start anticoagulants early.
Now if you want to ask about a specific subpopulation (which John did not), we need the N and P numbers for the subpopulation for the early and late protocols, and then the curves for that subpopulation will again necessarily be unimodal. But since N will be smaller, they will overlap more. That's all there is to it.
John says "A Simple Study May Herald a Big Change in Evidence-Based Medicine", and I am mystified as to how this can be. They do some fancy-pants frequentist statitics in the appendix, but I don't see how it improves on the simple Bayesian analysis.
1. Your idealized model doesn't really have anything to do with the study either.
2. If you think "eyeballing" Bayesian posterior probability curves is how EBM and medical science should work, well I am going to disagree.
3. I probably should have just left your dogmatic support of Bayesian methods and dogmatic critique of frequentists go since this is distracting from what Dr. M might have to share with us.
Re 2, we could of course compute the probability that x_early is less tha x_late; in this case it's 93%. Now we have to make a subjective judgment whether a 93% chance of a better outcome is worthwhile.
Let's say I have been taking mechanical samples from a bin of white and red beads. And what I find is that my estimate is that 4.1% of the beads are red.
Somebody comes along as says she has a magic bead selecting paddle which repeals red beads (which we don't want). She takes 1000 samples with her magic paddle and gets 2.9% red. Does she really have a magic paddle? I say not so far as I can see because:
0.0029 is not < 0.041 - 3 * sqrt (.041*(1-.041)/1000)
But because there is also no evidence of more red beads, I have no objection to her using her magic paddle and maybe we will eventually see that her paddle is better at repelling red beds and we might in future tests gain additional evidence regarding whether there could be special cases where the magic paddle could case some harm.
Your comparison of medical studies to running races, with statistics as the judge, is an apt metaphor that underscores the complexities of interpreting study results. I am looking forward to your upcoming discussions on evidence-based medicine. Your work is a valuable contribution to the ongoing dialogue on improving patient care and outcomes. Thank you for your dedication to sharing these important insights with the wider community. Please keep up the excellent work.
I've always been frustrated at the way most medical research is analyzed and reported - only relative risk reductions, not absolute risk reductions. Also annoying is the failure to talk about selection bias in clinical trials (e.g., disqualifying those who had AE's in Phase 1 and/or 2 from inclusion in Phase 3, causing misleading conclusions about the rate of AE's). So your articles on this are refreshing. However, I was surprised to see them from you, Dr. M., because I always thought you were a big statin fan (maybe from your Medscape articles, or I just assumed it), including for primary prevention where the ARR is only 1-2% over 5 years, less than the incidence of AE's. So I went back to some of your old articles on your blog, and discovered your position is much more nuanced. Now I see that you recognize the benefit varies based on one's absolute risk to begin with. I would like to see you write more about this in the future with statins. For example, is a CAC score a better indicator of absolute risk than calculators that rely heavily on lipid profiles? Does this vary by age? Also, if one starts taking a statin, this tends to increase the CAC score because it turns the soft plaque to calcified plaque, supposedly making it less dangerous, although this is controversial. What are your thoughts on that, and how useful is a CAC score once one starts on a statin? Since cholesterol is not a very good predictor, especially in seniors, and if the CAC score's usefulness is nullified by statins, how should one assess risk in seniors on statins? Most doctors just want to get that cholesterol down really low, but is that useful, especially in seniors? So many questions! :) Thanks!
Sobshrink, that's something I was not aware of, regarding the disqualification of those with AEs from phase 3 trials. Although, wouldn't there be a small risk of bias when they are pulling from fresh populations?
But the job of the ethical physician is not to convince, it's to present facts in a non-biased way so that the patient can make their own decision. Since the RRR is often misleading, it should only be used as a supplement to the ARR, IMO. But perhaps your point was that most doctors prefer to "convince" ?!
In my former field it sure is (addiction and pain medicine) - there are many who think adding bupe to the water would be a great idea. Same with opioids for CNCP which seems to be coming back around again.
I recently had an exchange with a doctor here about opioids, and was surprised at how much he advocated them, given some of his other more evidence-based comments. Also, my husband recently had surgery, and the doctor INSISTED he fill his Rx for oxy, even though I told him RCT's clearly demonstrated that ibuprofen+tylenol are more effective than opioids. They profess to use only EBM, so what's the deal?! My mother's GI tract was destroyed by a prescribed opioid that she only took per doctor's instructions. UGH! :( BTW, I now have chronic pain, but I will NEVER take an opioid!
Yeah, and I think that was sort of my point at my first comment. I’ve had chronic pain, my whole life, prescribed opioids for 18 years, I have been off of them for the past 24. Ask me which life is better? 😉. They also used to say you can’t get addicted when treating pain. So with addiction and ORT, the can gets kicked further down the road, and keeps a low bar low for the recovering person.
Congratulations on getting off them, but I'm sorry life still sucks - I hear you. I told the other doctor to find a non-addictive, non-harmful pain killer. I keep waiting! We Boomers need something! I'm about to try noninvasive vagal nerve stimulation. Even if it doesn't reduce my pain, maybe at least I'll be less stressed about it! :)
I am a physician and I'm just getting my feet wet in clinical research. Your aubstack has allowed me to have more confidence questioning some research papers I read. Thank you for your work. It keeps me inspired.
The results described for the THAPCA trial seems backwards: The survival rate for therapeutic hypothermia is specified as 8% less (12% vs. 20% standard care), yet the article implies that there was a massive benefit from the intervention. Please explain or correct.
Not buying it until I can see the entire studies. None of these drugs can beat the positive effects of high potency cayenne solutions which are extremely cheap to use and with no risk of side effects.
I'm a family physician. A 86 year old care home resident with moderate dementia I care for had a major fall leading to a severely swollen arm. She was admitted over 3 days as some debate as to whether fracture or not- eventually decided not. During admission she was found to be in Atrial Fibrillation so discharged on NOAC. The HASBLED score gives her only one point ie >65. No account taken of advanced frailty and numerous falls including the cause of the current admission.
To me as a old fashioned doc of 30 yrs clearly looked wrong- felt obliged to speak to her son but we agreed NOAC to be stopped as tiny benefit potential and clearly significant risk.
An error on my part and I forgot to update the care home and they continued NOAC- 3 days later
developmed intractable nose bleeding and had to attend Accident and Emergency. The hospitals action .... must continue NOAC but have 14 days of tranexamic acid....
Any clinicians on here please tell me if I'm wrong but I struggle to think of this as anything other the crass, unthinking tick box medicine.
For ELAN, I see no major practical difference in composite poor outcomes: 29 out of 1000 vs. 41 out of 1000. That means 1930 out of 2000 (96.5%) had good outcomes with either treatment arm. What could matter, however, is if the incidence of death was overwhelming in either treatment arm.
thanks for this; we look forward to the next installment
Here is the very simple Bayesian analysis. Let x be the true fraction of patients who will have a primary (bad) outcome under a given protocol. N patients are given that protocol, P have the primary outcome, N-P do not. Assuming a uniform prior for x (that is, we have no idea what x is going in), the "posterior" probability that x has a particular value is p(x) = C x^P (1-x)^(N-P), where C = (N+1)!/(P!(N-P)!). This is what we now know about what x is likely to be (for a specific protocol), given the data that we now have.
Here are the plots of p(x) vs x for N=1000, P=29 (blue) or P=41 (orange):
https://i.postimg.cc/bNzMJt9G/rates.jpg
Now we could quantify the differences between the two curves in various fancy ways, but just look at them. Again, the x axis is the true fraction of patients who will have a bad outcome. Do you want to be given the blue protocol or the orange protocol? Is there really any question in your mind?
Maybe. I am not unsympathetic.
But what if the shape of the distribution is not how you imagine it.
The shape is necessarily single-peaked in a simple yes/no case like we have here, and is approximately gaussian once you have decent statistics.
Here's what happens in this case with N=100, P=3 (blue) or P=4 (orange), what you would have likely gotten as a result with 100 rather than 1000 patients per protocol:
https://i.postimg.cc/tCzQyVYT/rates2.jpg
Now there's still a slight preference for the blue protocol, but not nearly as pronounced. Clearly we need more data.
And again this is all Bayesian methodology 101, extremely simple, easy to understand, and quickly gives the subjective results you need, without making your brain hurt or allowing for p-hacking.
"The shape is necessarily single-peaked in a simple yes/no case like we have here, and is approximately gaussian once you have decent statistics."
🤷♂️ it is only "necessarily" uni modal if the population from which you sample is not heterogeneous. But that is almost never the case in real life medicine, even if yes/no. For example, gender.
Once your model gets at all complicated, the amount of prior component probabilities you must account for becomes overwhelming.
I'm not unsympathetic but we must be circumspect. Bayesian approach is not a panacea. Nor is frequentist approach what is "holding back" so-called evidence based medicine.
Statistical thinking is necessary but insufficient.
This is completely irrelevant to the question John posed:
"In multiple centers, slightly more than 2000 patients were randomized to either early or later starting of the anticoagulant drugs. A primary outcome occurred in 2.9% of the patients in the early arm vs 4.1% in the later treatment arm. The absolute risk reduction was 1.2%. The relative risk reduction was 30%—expressed as an odds ratio of 0.70. The question for you is what to make of these results."
I have answered the question. Looking at the Bayesian posterior probability curves shows instantly and viscerally that you should start anticoagulants early.
Now if you want to ask about a specific subpopulation (which John did not), we need the N and P numbers for the subpopulation for the early and late protocols, and then the curves for that subpopulation will again necessarily be unimodal. But since N will be smaller, they will overlap more. That's all there is to it.
John says "A Simple Study May Herald a Big Change in Evidence-Based Medicine", and I am mystified as to how this can be. They do some fancy-pants frequentist statitics in the appendix, but I don't see how it improves on the simple Bayesian analysis.
Let's see what Dr. M says next week.
1. Your idealized model doesn't really have anything to do with the study either.
2. If you think "eyeballing" Bayesian posterior probability curves is how EBM and medical science should work, well I am going to disagree.
3. I probably should have just left your dogmatic support of Bayesian methods and dogmatic critique of frequentists go since this is distracting from what Dr. M might have to share with us.
Re 2, we could of course compute the probability that x_early is less tha x_late; in this case it's 93%. Now we have to make a subjective judgment whether a 93% chance of a better outcome is worthwhile.
Let's say I have been taking mechanical samples from a bin of white and red beads. And what I find is that my estimate is that 4.1% of the beads are red.
Somebody comes along as says she has a magic bead selecting paddle which repeals red beads (which we don't want). She takes 1000 samples with her magic paddle and gets 2.9% red. Does she really have a magic paddle? I say not so far as I can see because:
0.0029 is not < 0.041 - 3 * sqrt (.041*(1-.041)/1000)
But because there is also no evidence of more red beads, I have no objection to her using her magic paddle and maybe we will eventually see that her paddle is better at repelling red beds and we might in future tests gain additional evidence regarding whether there could be special cases where the magic paddle could case some harm.
There is no way to know it is Gaussian with the information you have provided.
Central limit theorem. Non-gaussian corrections to the moments are suppressed by 1/N.
Views in the millions! Congratulations
Bayesian methodology would fix all this. Frequentism is a dreadful ideology.
Meh.
See my other, more detailed comment.
Your comparison of medical studies to running races, with statistics as the judge, is an apt metaphor that underscores the complexities of interpreting study results. I am looking forward to your upcoming discussions on evidence-based medicine. Your work is a valuable contribution to the ongoing dialogue on improving patient care and outcomes. Thank you for your dedication to sharing these important insights with the wider community. Please keep up the excellent work.
I've always been frustrated at the way most medical research is analyzed and reported - only relative risk reductions, not absolute risk reductions. Also annoying is the failure to talk about selection bias in clinical trials (e.g., disqualifying those who had AE's in Phase 1 and/or 2 from inclusion in Phase 3, causing misleading conclusions about the rate of AE's). So your articles on this are refreshing. However, I was surprised to see them from you, Dr. M., because I always thought you were a big statin fan (maybe from your Medscape articles, or I just assumed it), including for primary prevention where the ARR is only 1-2% over 5 years, less than the incidence of AE's. So I went back to some of your old articles on your blog, and discovered your position is much more nuanced. Now I see that you recognize the benefit varies based on one's absolute risk to begin with. I would like to see you write more about this in the future with statins. For example, is a CAC score a better indicator of absolute risk than calculators that rely heavily on lipid profiles? Does this vary by age? Also, if one starts taking a statin, this tends to increase the CAC score because it turns the soft plaque to calcified plaque, supposedly making it less dangerous, although this is controversial. What are your thoughts on that, and how useful is a CAC score once one starts on a statin? Since cholesterol is not a very good predictor, especially in seniors, and if the CAC score's usefulness is nullified by statins, how should one assess risk in seniors on statins? Most doctors just want to get that cholesterol down really low, but is that useful, especially in seniors? So many questions! :) Thanks!
Sobshrink, that's something I was not aware of, regarding the disqualification of those with AEs from phase 3 trials. Although, wouldn't there be a small risk of bias when they are pulling from fresh populations?
Your question on statins is excellent.
RRR is more convincing for the lay person.
But the job of the ethical physician is not to convince, it's to present facts in a non-biased way so that the patient can make their own decision. Since the RRR is often misleading, it should only be used as a supplement to the ARR, IMO. But perhaps your point was that most doctors prefer to "convince" ?!
In my former field it sure is (addiction and pain medicine) - there are many who think adding bupe to the water would be a great idea. Same with opioids for CNCP which seems to be coming back around again.
I recently had an exchange with a doctor here about opioids, and was surprised at how much he advocated them, given some of his other more evidence-based comments. Also, my husband recently had surgery, and the doctor INSISTED he fill his Rx for oxy, even though I told him RCT's clearly demonstrated that ibuprofen+tylenol are more effective than opioids. They profess to use only EBM, so what's the deal?! My mother's GI tract was destroyed by a prescribed opioid that she only took per doctor's instructions. UGH! :( BTW, I now have chronic pain, but I will NEVER take an opioid!
https://www.nsc.org/getmedia/8ecdc0e5-ae58-43e8-b98b-46c205e1c2b2/evidence-efficacy-pain-medications.pdf
Yeah, and I think that was sort of my point at my first comment. I’ve had chronic pain, my whole life, prescribed opioids for 18 years, I have been off of them for the past 24. Ask me which life is better? 😉. They also used to say you can’t get addicted when treating pain. So with addiction and ORT, the can gets kicked further down the road, and keeps a low bar low for the recovering person.
Congratulations on getting off them, but I'm sorry life still sucks - I hear you. I told the other doctor to find a non-addictive, non-harmful pain killer. I keep waiting! We Boomers need something! I'm about to try noninvasive vagal nerve stimulation. Even if it doesn't reduce my pain, maybe at least I'll be less stressed about it! :)
I am a physician and I'm just getting my feet wet in clinical research. Your aubstack has allowed me to have more confidence questioning some research papers I read. Thank you for your work. It keeps me inspired.
The results described for the THAPCA trial seems backwards: The survival rate for therapeutic hypothermia is specified as 8% less (12% vs. 20% standard care), yet the article implies that there was a massive benefit from the intervention. Please explain or correct.
Fixed. Sorry. See new post and editor’s note. Thx for reading so closely
I'd appreciate a reference/link to NEJM article referenced. Thanks.
so what about when individual drugs are evaluated when they are normally given in conjunction with others? Or using unusual dosages?
I get confused when there are conflicting results due to strange choices that don't mirror what is usually prescribed.
It's almost like there is an attempt to prop up propaganda...
That’s bc we don’t see pharma sponsored studies that “fail”
Not buying it until I can see the entire studies. None of these drugs can beat the positive effects of high potency cayenne solutions which are extremely cheap to use and with no risk of side effects.
Looking forward to next week: "Therefore, no statistical hypotheses as to superiority, inferiority, or noninferiority were tested."
Looking forward to next week discussion. Check whether you flipped the 12 and 20…. Should be 20 in low temp and 12
In not cooled
Thanks JH. Correction made.
What resource do you recommend for physicians who want to get better at critically interpreting trial results?
I was also wondering about that , thanks for the correction
I was wondering about that...thanks!
Sensible Medicine…
we can only hope.
Thank you.