Millions of people have AF and take oral anticoagulation. It's one of Medicine's most evidence-based treatments. It would take strong data to upend this treatment.
So at 4 months about 1 in 5 patients who received Watchman had a device leak which is a known risk factor for thrombus formation and stroke. About 5% had an actual thrombus noted on the device at follow up. Before jumping on the bandwagon for this industry supported study, please answer the following questions:
1. What to do for the patients with leaks? Another invasive procedure with the inherent risk? Resume anticoagulation when we told the patient that being able to avoid anticoagulation was the reason for the implant in the first place? If we resume anticoagulation then for how long?
2. What to do for a patient found to have a thrombus on the device? Resume anticoagulation and if so for how long? Where are the large follow up trials showing how to deal with this problem? What frequency of follow up imaging is needed? Again, any large trials showing the best approach to follow up imaging? I seem to remember that effective anticoagulation generally should lower the risk of major cardioembolic events to < 1% per year in most. Yet the risk of thrombus formation on the Watchman is close to 5% at less than 1 year?
3. With the so called "shared decision making" are these risks as noted above clearly presented to most patients? If the patient develops one of these complications then what is my medical legal risk ?
Boston Scientific has been chomping at the bit for years to advance the Watchman device as something available to ALL patients with AF---not just those who can't take an anticoagulant. Just think of the increase in market share and potential profits for the investors.
The core argument you make is fair enough but I want to pick at it one one point you make.
The key point from Dr. Mandrola regarding the latest trial of atrial appendage closure devices (CHAMPION-AF) is that with very primary outcomes in the trial, claiming that that the closure device achieved non inferiority based on not exceeding an absolute difference of 4.8% is lenient. The event rate was actually a smidge worse for the closure device than medical management, and with a prior study (CLOSURE-AF) showing an "much worse", it's not a good time to be crowing in favor of devices like the Watchman. My biostatistical colleague and I agreed with this point, when we reviewed this point today.
However, there is a "bone to pick" with you when you argue for testing a relative noninferiority margin for super rare events. I'll quote you here so everyone knows what you suggest:
"Now we tip-toe into fractions. Sorry. 4.8% higher than 12% is equal to 40% or 1.4. Translation: the relative risk margin is 1.4. In most if not all drug noninferiority trials, the authors set out NI margins in both absolute and relative terms. CHAMPION AF authors did not do this. Their noninferiority margin was just 4.8% in absolute terms"
Put another way, there is a chance that the device was 1.6 times worse, in relative terms, than traditional anticoagulation, and that would be outside the relative odds ration of 1.4. Again, that's why you are faulting the study (fair enough).
But there is a problem in trying to worry about relative risks for rare events. Once a bad outcome is super uncommon, then the relative risk hardly matters to most ordinary people. If you tell me I could receive a convenient treatment super rare bad outcomes, I might say "I don't care if the relative risk is 1.6 when the difference is between 1 in 1000 and 1.6 in 1000."
Those aren't the actual numbers in CHAMPION-AF of course, but it's probably the case that relative risk margins often don't matter much with very rare bad outcomes. Of course I appreciate your skepticism of industry here!
On your 'bad practice' point John. I have been trying for decades to stop students/people saying that if two options perform equally on a criteria we can drop that criteria from the analysis ('noise' as you call it). A criteria should be included on the basis of its importance and the greater its importance the greater the importance of keeping it in, whether or not all options perform equally on it. The more important the criteria on which the two available options perform equally well the closer the verdict is to a toss-up between them, and it should be accepted as such. CV death seems a pretty important criteria to me,.
Maybe I’m mathematically incorrect.. if you told the patient as part of shared decision making- we have two items for cost comparison one cost - $10,000 and the other is $13,999. The way the studies work they consider both those prices to be the same. The device not only didn’t cost less than the standard therapy. It could cost $3999 more before it would be considered more expensive.
Thank you for once again navigating so deftly through the dense fog that can permeate clinical trial design and complicate interpretation of the data derived.
As a stroke doc who participated as a PI in SPAF I, II, and III, I was complicit in helping to establish the effectiveness of anticoagulation (AC) for preventing stroke in patients with AF. It is the nature of clinical practice that I cannot identify patients in whom AC has prevented a disabling cardioembolic stroke but do see patients who have experienced significant and even fatal complications of AC. As such, I would welcome the Watchman or any other device that provided AF patients the efficacy of AC without the associated bleeding risk. Unfortunately, we have yet to identify and confirm the utility of such a device.
A closing word about AC…
Especially since the arrival of the DOACs, AC for AF has evolved to become routine… perhaps even a bit reflexive. Not good– reflexive medicine can be harmful. Not all AF patients require AC, and even the DOACs convey risk.
In a study we presented at the 2024 European Academy of Neurology meeting, of 117 consecutive AF patients seen in a general neurology clinic, 37% were calculated to have an annual risk of a significant AC-related bleeding complication over 1 % greater than their annual risk of stroke if off AC.
Perhaps this does represent one clinical circumstance where shared decision-making may be appropriate.
Upper bound CI of watchman allowed to be 4.8% worse…than a control arm with an event rate of 4.8%. It’s incredible that NEJM would publish a paper that is allowed to suggest that “up to 100% worse” or “up to 2X event rate” is something that is “non-inferior”. It turned out to be up to 1.66X worse, which still stretched the credulity of the concept.
At least the accompanying editorial demonstrated restraint and did not participate in the cheerleading. It also mentioned that the protocol was designed by sponsor, and data was analyzed by sponsor. Maybe NEJM should have added an “advertorial” disclaimer instead of the “original research” label to the manuscript.
John - do you believe that a patient who is truly not a long-term anticoagulation candidate and who has significant risk for stroke - does the watchmen add benefit over hope?
PS. I have a post that will publish in a few weeks about shared decision-making and the information asymmetry between patients and doctors. The main point is that the phrase “shared decision-making” often is a more palatable way of telling patients what the physicians already has decided. I don't say that cynically. It's just what happens when the phrase gets tossed around and put in guidelines without thinking carefully about what it actually requires. I then discuss the SHARE approach and what sharing decisions really requires. Thanks, John for this post.
As our friend and brilliant methodologist Sanjay Kaul: commented elsewhere “Good trial design balances feasibility with rigor; when it’s designed to deliver a win, it signals more about clever protocol writing and trial design than breakthrough efficacy.” I am ashamed of the NEJM for publishing a paper on a study as flawed as this one - all the points you made, but especially, excluding protocol related bleeding, open label design, the excess of people with PAF who statistically did much better with OAC, and not required the authors to discuss the excess of strokes and the details of the non-inferiority bounds, as you laid out. Gregory M. Marcus's editorial was an insightful breath of fresh air and I highly recommend it.
So at 4 months about 1 in 5 patients who received Watchman had a device leak which is a known risk factor for thrombus formation and stroke. About 5% had an actual thrombus noted on the device at follow up. Before jumping on the bandwagon for this industry supported study, please answer the following questions:
1. What to do for the patients with leaks? Another invasive procedure with the inherent risk? Resume anticoagulation when we told the patient that being able to avoid anticoagulation was the reason for the implant in the first place? If we resume anticoagulation then for how long?
2. What to do for a patient found to have a thrombus on the device? Resume anticoagulation and if so for how long? Where are the large follow up trials showing how to deal with this problem? What frequency of follow up imaging is needed? Again, any large trials showing the best approach to follow up imaging? I seem to remember that effective anticoagulation generally should lower the risk of major cardioembolic events to < 1% per year in most. Yet the risk of thrombus formation on the Watchman is close to 5% at less than 1 year?
3. With the so called "shared decision making" are these risks as noted above clearly presented to most patients? If the patient develops one of these complications then what is my medical legal risk ?
Boston Scientific has been chomping at the bit for years to advance the Watchman device as something available to ALL patients with AF---not just those who can't take an anticoagulant. Just think of the increase in market share and potential profits for the investors.
Hey — I came across your writing and really liked how you think.
I’m exploring something similar from a different angle — writing about human behavior through a system design lens (like debugging internal patterns).
Just started publishing on Substack. If you ever get a moment to read, I’d genuinely value your perspective.
Also happy to support your work — feels like there’s an interesting overlap here.
Awesome analysis.
LAAO has more robust evidence than Afib ablation in the general Afib population (not in specific subgroups). Argue against that, please, Dr Mandrola.
Perfect analysis
The core argument you make is fair enough but I want to pick at it one one point you make.
The key point from Dr. Mandrola regarding the latest trial of atrial appendage closure devices (CHAMPION-AF) is that with very primary outcomes in the trial, claiming that that the closure device achieved non inferiority based on not exceeding an absolute difference of 4.8% is lenient. The event rate was actually a smidge worse for the closure device than medical management, and with a prior study (CLOSURE-AF) showing an "much worse", it's not a good time to be crowing in favor of devices like the Watchman. My biostatistical colleague and I agreed with this point, when we reviewed this point today.
However, there is a "bone to pick" with you when you argue for testing a relative noninferiority margin for super rare events. I'll quote you here so everyone knows what you suggest:
"Now we tip-toe into fractions. Sorry. 4.8% higher than 12% is equal to 40% or 1.4. Translation: the relative risk margin is 1.4. In most if not all drug noninferiority trials, the authors set out NI margins in both absolute and relative terms. CHAMPION AF authors did not do this. Their noninferiority margin was just 4.8% in absolute terms"
Put another way, there is a chance that the device was 1.6 times worse, in relative terms, than traditional anticoagulation, and that would be outside the relative odds ration of 1.4. Again, that's why you are faulting the study (fair enough).
But there is a problem in trying to worry about relative risks for rare events. Once a bad outcome is super uncommon, then the relative risk hardly matters to most ordinary people. If you tell me I could receive a convenient treatment super rare bad outcomes, I might say "I don't care if the relative risk is 1.6 when the difference is between 1 in 1000 and 1.6 in 1000."
Those aren't the actual numbers in CHAMPION-AF of course, but it's probably the case that relative risk margins often don't matter much with very rare bad outcomes. Of course I appreciate your skepticism of industry here!
Yes agree but approximately 5% is not rare, correct?
Thanks for this analysis. As an AF patient it’s good information.
On your 'bad practice' point John. I have been trying for decades to stop students/people saying that if two options perform equally on a criteria we can drop that criteria from the analysis ('noise' as you call it). A criteria should be included on the basis of its importance and the greater its importance the greater the importance of keeping it in, whether or not all options perform equally on it. The more important the criteria on which the two available options perform equally well the closer the verdict is to a toss-up between them, and it should be accepted as such. CV death seems a pretty important criteria to me,.
Maybe I’m mathematically incorrect.. if you told the patient as part of shared decision making- we have two items for cost comparison one cost - $10,000 and the other is $13,999. The way the studies work they consider both those prices to be the same. The device not only didn’t cost less than the standard therapy. It could cost $3999 more before it would be considered more expensive.
Thank you for once again navigating so deftly through the dense fog that can permeate clinical trial design and complicate interpretation of the data derived.
As a stroke doc who participated as a PI in SPAF I, II, and III, I was complicit in helping to establish the effectiveness of anticoagulation (AC) for preventing stroke in patients with AF. It is the nature of clinical practice that I cannot identify patients in whom AC has prevented a disabling cardioembolic stroke but do see patients who have experienced significant and even fatal complications of AC. As such, I would welcome the Watchman or any other device that provided AF patients the efficacy of AC without the associated bleeding risk. Unfortunately, we have yet to identify and confirm the utility of such a device.
A closing word about AC…
Especially since the arrival of the DOACs, AC for AF has evolved to become routine… perhaps even a bit reflexive. Not good– reflexive medicine can be harmful. Not all AF patients require AC, and even the DOACs convey risk.
In a study we presented at the 2024 European Academy of Neurology meeting, of 117 consecutive AF patients seen in a general neurology clinic, 37% were calculated to have an annual risk of a significant AC-related bleeding complication over 1 % greater than their annual risk of stroke if off AC.
Perhaps this does represent one clinical circumstance where shared decision-making may be appropriate.
Upper bound CI of watchman allowed to be 4.8% worse…than a control arm with an event rate of 4.8%. It’s incredible that NEJM would publish a paper that is allowed to suggest that “up to 100% worse” or “up to 2X event rate” is something that is “non-inferior”. It turned out to be up to 1.66X worse, which still stretched the credulity of the concept.
At least the accompanying editorial demonstrated restraint and did not participate in the cheerleading. It also mentioned that the protocol was designed by sponsor, and data was analyzed by sponsor. Maybe NEJM should have added an “advertorial” disclaimer instead of the “original research” label to the manuscript.
A triumph of technology over reason.
John - do you believe that a patient who is truly not a long-term anticoagulation candidate and who has significant risk for stroke - does the watchmen add benefit over hope?
What about the patient who flatly refuses OAC - that's the patient I'm concerned about. Is LAAC better than nothing?
When I started training in the 70s, the NEJM was not a medical journal but “the Bible“ – now it’s the “Enquier“ -(sensational rag )Ben Hourani MBA
Agree. It was NEJM, The Lancet, and JAMA as a distant 3rd. I noticed the NEJM and The Lancet going "woke" in the 1990s. Sad, very sad.
What an excellent analysis. Thank you.
PS. I have a post that will publish in a few weeks about shared decision-making and the information asymmetry between patients and doctors. The main point is that the phrase “shared decision-making” often is a more palatable way of telling patients what the physicians already has decided. I don't say that cynically. It's just what happens when the phrase gets tossed around and put in guidelines without thinking carefully about what it actually requires. I then discuss the SHARE approach and what sharing decisions really requires. Thanks, John for this post.
As our friend and brilliant methodologist Sanjay Kaul: commented elsewhere “Good trial design balances feasibility with rigor; when it’s designed to deliver a win, it signals more about clever protocol writing and trial design than breakthrough efficacy.” I am ashamed of the NEJM for publishing a paper on a study as flawed as this one - all the points you made, but especially, excluding protocol related bleeding, open label design, the excess of people with PAF who statistically did much better with OAC, and not required the authors to discuss the excess of strokes and the details of the non-inferiority bounds, as you laid out. Gregory M. Marcus's editorial was an insightful breath of fresh air and I highly recommend it.