Such reviewers have the negative effect (as expressed by some readers) of reinforcing distrust in medicine in general. Which is not true. There are tremendous advances in treatment of many diseases that relieve suffering and prolong life.
However, all successful fields are susceptible to malign and manipulation. Until and unless we have state funding for all trials, industry is involved, publications will necessarily be biased. Negative trials form the industry are generally buried or terminated early; data tend to be contorted for a positive spin. The ‘journal industry’ should be called out for irresponsible publications such as this. Good job, Dr. Mandrola.
Have no fear! Label yourself as a HAPPY CYNIC and move on. The world needs people who ask good questions and understand that a dark side exists in all professions.
A deep personal thanks. I saw repeat Boston Scientific ads on LinkedIn for this procedure and it caught my attention for very personal reasons as a loved one was recently diagnosed with Afib. The ad seemed to imply this procedure treated Afib. It took a very close read to understand they were only claiming to reduce the need for blood thinners, and then in an almost passing remark, the procedure and device they were selling might not be appropriate if the patient was doing well on blood thinner. Then I read your article how biased and flawed this study was. Thank you. I have blocked their ad and downgraded my thinking on Boston Scientific, although I love their robotic dog. No pooper scooper needed, unlike their Afib ad!
If AI could make these treatment decisions (and save the decision inputs), confounding effects from observational studies could be removed. The decision-to-treat could be reverse-engineered and accounted for. So there is hope for the future.
My pleasure. Lately, I've taken on the hobby of studying bias and its cousin, errors of informal logic. I'm still rather a neophyte, but I try to expand my knowledge base whenever it arises. A specific case, such as this posting by John, serves me as a teachable moment.
The authors did take into account immortal time bias. How easy it is to disqualify them without really taking the time to read and understand the article. These sentences are all evidence that they did take it into account (according to https://jamanetwork.com/journals/jama/fullarticle/2776315 which describes these strategies):
-Time 0 for the LAAO and anticoagulation groups was defined as coincident with the LAAO procedure for the patient receiving LAAO in each matched pair"
-After matching, a subset of patient pairs (31 of 4085 women [0.76%] and 44 of 5378 men [0.82%]) had an anticoagulation death date before the corresponding LAAO procedure date and were excluded from outcome analysis.
-Second, it was expected that survival bias would affect the assessment of outcomes between groups. So, after the removal of data of patients who died within one year of the index date, outcomes were recalculated within the matched groups.
I do agree that the conclusion is inadequate as this observational study cannot be used to support such a recommendation.
This corruption of medical science seems to be widespread. From the FDA and CDC and various academic infectious disease departments during the pandemic to the onslaught of direct to consumer pharmaceutical ads today.
What can we do to build back the integrity and morality of our profession?
First, build a firewall to keep boundaries in place between medical scientific practice and for-profit companies...
Stop direct to consumer pharmaceutical ads.
We can start by reaching out to our state and federal representatives for legislative action. I would be heartbroken to learn that my colleagues who are specialty and subspecialty train have lost their moral commitment to put the patient's interests first and to think critically about any intervention or treatment that they recommend.
As a Dutch GP MD I'm inspired by you and the team of sensible medicine, to improve outcome for patients, by improving my decision-making proces, by adequately recognizing flaws and strengths in clinical reasoning and studies.
Sensible Medicine and VP's essays and reviews teach critical analysis and reasoning in an accessible, transparent and educational way. A must read for all students and colleagues. I hope eyes will be opened and it will induce a change on micro- (individual pt-doctor care), meso- (guidelines, literature etc) and macro level.
Great appraising John. Right up my street as you know!
I wanted to thank you for citing the Catalogue of Bias entry on immortal time bias. I’m chuffed to see it being used for exactly the reason it should be - educating folk on what biased and unbiased research looks like.
As finally turned Marcia Angell, this is all about, and only about, following the dollars. And the government dollars, extorted from each of us under penalty of incarceration, are as corrupt, and perhaps even more corrupt, than the institutional dollars. At least we understand what the companies want prima facie.
This remains hopeless until other, better ways are found for driving/financing research. I published extensively early in my career -- some relatively important papers that changed the way medicine was practiced and put into place and are still followed, and some, shall we say, less important. But in all cases, none of this was funded by anyone. I begged/borrowed what I needed, got tiny grants from the department that came from practice funds, and that was how things worked. I was paid a pittance but I was in medicine for making health better -- not getting a pharma/government granted/coerced sinecure. And I think most of the research was appreciably better.
No research seems to be even contemplated today until "grant funding" is first discussed, decided and achieved. Much of the research is aimed at what will be funded rather than what will be important to patients. (The example in this article is illustrative.) Just reading the disclosures is enough to discredit any conclusions.
There are still rules for granting continuing education credit that someone who profits from or who has been funded by what they are doing cannot give lectures granting CE credit on those topics. Sadly, this appears to be the last place where conflicts of interest actually "do" something.
Again, money corrupts absolutely. Your cynicism will have to join mine and those of many others until the funding is changed. If Covid has taught us anything, changing it to the "government" will only make it worse -- now we have no accountability and extorted money -- so even the modest restrictions perhaps imposed by business models are not there.
The brain power of this stack would be well spent proposing a different model for running research without all the impossible-not-to-corrupt funding going on.
This is why even with heart problems, I have no faith in cardiology. I do not believe the heart drug trials are true RCT's with true placebos and being double blinded. I do not believe cholesterol has anything to do with creating heart problems. I do not believe statins are safe. Stents and angioplasty are not the surest options as you guys want us to believe.
My choice of survival products include hawthorn berry, nattokinase, olive leaf, high potency 180 KHU cayenne tincture and various other supplements. Also there are numerous heart healthy foods.
Surgery, except in certain cases, and drugs should be the last line of defense since they will never cure the cause of heart problems. There is always a reason for all heart difficulties and that is beyond the scope of modern medicine to investigate the problem.
I am so grateful to you and others who take the time to notice and report these flawed studies. I find this work to be so hopeful. Previously, I looked at studies pertinent to my field, did a cursory review and generally accepted the conclusions as ‘fact’. This trend (since crazy 2020) in criticism and sharing is a boon to the integrity of medicine (and politics in general).
There are still so many who see the promise of humble but objective science. They are on the alert now in ways that they weren’t before. I don’t think truth and “good” always win. But in our individual lives, nothing else is tolerable.
Hope comes when the number of Individuals who can’t tolerate lies becomes large and strong.
Thank you for this, Doc. This is a great Howard Beale ("Network") moment: "I'm mad as hell, and I'm not going to take this any more!" Me, too; all of us, probably. What do you want to do now?
It's easy to make a diagnosis, to figure out (or at least imply) a root cause analysis. We all learned how to do that by internalizing a model of pathophysiology. We learned how to question the anatomy and physiology, do an appropriate test (maybe get more history, maybe do a detailed and focused physical, maybe order that MRI that everybody seems to want these days), and lay out an appropriate, testable therapy. It's easy to "doctor", but it's hard indeed to heal.
Your diagnoses seem to include poor understanding of statistics, (potentially) inappropriate goals for some humans involved, a possible lapse in the Great Tradition of a classic journal as well as its associated parties. What has the response been by the other parties? You did the history, you performed an intricate, beautiful physical (you should have been a neurologist!), you have at least implied the right therapy. Now what?
I could think of a bunch of ways to use your article: a class in rhetoric, a class in healthcare policy, a grad class in research design, a class in economics. A basis for a Master's thesis if not a dissertation. But this paper, which in its way is maybe as important as the Ionnadis "Why most published research...." (2005) article, depending on your OWN peer review of course, ought to be the challenge for every medical student, every tired and frenzied resident (we were, so I'm guessing that nothing has changed), every clinical and even basic science researcher, and every administrator in healthcare who has forgotten what the mission is... who has forgotten why there are doctors.
And those of us who are doctors - MD, PhD, DO, DC, whatever - need to remember that the origin of the word "doctor", a word which we too often toss around as a way to make a buck and more than a buck, is in fact from the Latin "docere", i.e., "to teach". Doctors teach. We teach sick people, and sick researchers, and sick journals, and sick medical schools, how to be well, how to 'regain function' so the inflammation disappears.
Doc, you have us. You have me, for sure. Like John Perry, I'm ready to become a paid member and demonstrate my solidarity although I might not live to see your therapy be tested and proved. But I am betting that your rhetorical pose is more than just theater: you have genuine despair, and so do many of us. Well, one way out of despair is action. Cure. Effecting change. What can we do? What should we do? One thing for sure: it is time to stop lamenting "Oh cursed spite, that ever I was born to set it right."
Thank you, Dr. Mandrola, for taking the time to explain this so well in terms a lay person can understand. And thank you for your courage in posting this. Because of pay walls, I (like other members of general public), can't read the full text of the study you criticize so effectively. So that's yet another reason I'm grateful to you for giving us a summary and interpretation which is readily accessible to the public. I feel that the best way I can show my support is by becoming a paid subscribe, so that's what I've just done. Thank you again!
Excellent piece and you bring up two excellent reasons for cynicism.
First, your excellent dissection reveals how this is marketing disguised as science.
Second, the fact that there is no scientific forum in which this can be published is a mark against science. We do discuss studies like these at journal clubs and, as you allude, we keep track of authors' reputations in our heads. Should there be some sort of public reckoning of reputations rather than the whispered discussions in back hallways at each institution? I think about how the computer science community has Stack Overflow where users can gain or lose reputation points (https://stackoverflow.com/help/whats-reputation). It seems like science needs something like this where excellent critiques like this one can be shared and you can earn reputation points for it while disingenuous critiques (or ad hominum attacks - ahem, VP) can get your reputation score downgraded.
Such reviewers have the negative effect (as expressed by some readers) of reinforcing distrust in medicine in general. Which is not true. There are tremendous advances in treatment of many diseases that relieve suffering and prolong life.
However, all successful fields are susceptible to malign and manipulation. Until and unless we have state funding for all trials, industry is involved, publications will necessarily be biased. Negative trials form the industry are generally buried or terminated early; data tend to be contorted for a positive spin. The ‘journal industry’ should be called out for irresponsible publications such as this. Good job, Dr. Mandrola.
Have no fear! Label yourself as a HAPPY CYNIC and move on. The world needs people who ask good questions and understand that a dark side exists in all professions.
A deep personal thanks. I saw repeat Boston Scientific ads on LinkedIn for this procedure and it caught my attention for very personal reasons as a loved one was recently diagnosed with Afib. The ad seemed to imply this procedure treated Afib. It took a very close read to understand they were only claiming to reduce the need for blood thinners, and then in an almost passing remark, the procedure and device they were selling might not be appropriate if the patient was doing well on blood thinner. Then I read your article how biased and flawed this study was. Thank you. I have blocked their ad and downgraded my thinking on Boston Scientific, although I love their robotic dog. No pooper scooper needed, unlike their Afib ad!
If AI could make these treatment decisions (and save the decision inputs), confounding effects from observational studies could be removed. The decision-to-treat could be reverse-engineered and accounted for. So there is hope for the future.
My pleasure. Lately, I've taken on the hobby of studying bias and its cousin, errors of informal logic. I'm still rather a neophyte, but I try to expand my knowledge base whenever it arises. A specific case, such as this posting by John, serves me as a teachable moment.
The authors did take into account immortal time bias. How easy it is to disqualify them without really taking the time to read and understand the article. These sentences are all evidence that they did take it into account (according to https://jamanetwork.com/journals/jama/fullarticle/2776315 which describes these strategies):
-Time 0 for the LAAO and anticoagulation groups was defined as coincident with the LAAO procedure for the patient receiving LAAO in each matched pair"
-After matching, a subset of patient pairs (31 of 4085 women [0.76%] and 44 of 5378 men [0.82%]) had an anticoagulation death date before the corresponding LAAO procedure date and were excluded from outcome analysis.
-Second, it was expected that survival bias would affect the assessment of outcomes between groups. So, after the removal of data of patients who died within one year of the index date, outcomes were recalculated within the matched groups.
I do agree that the conclusion is inadequate as this observational study cannot be used to support such a recommendation.
I am curious. For you, how much AF is “enough” AF to warrant treatment, absent symptoms and moderate, at most, risk of CVA.
This corruption of medical science seems to be widespread. From the FDA and CDC and various academic infectious disease departments during the pandemic to the onslaught of direct to consumer pharmaceutical ads today.
What can we do to build back the integrity and morality of our profession?
First, build a firewall to keep boundaries in place between medical scientific practice and for-profit companies...
Stop direct to consumer pharmaceutical ads.
We can start by reaching out to our state and federal representatives for legislative action. I would be heartbroken to learn that my colleagues who are specialty and subspecialty train have lost their moral commitment to put the patient's interests first and to think critically about any intervention or treatment that they recommend.
As a Dutch GP MD I'm inspired by you and the team of sensible medicine, to improve outcome for patients, by improving my decision-making proces, by adequately recognizing flaws and strengths in clinical reasoning and studies.
Sensible Medicine and VP's essays and reviews teach critical analysis and reasoning in an accessible, transparent and educational way. A must read for all students and colleagues. I hope eyes will be opened and it will induce a change on micro- (individual pt-doctor care), meso- (guidelines, literature etc) and macro level.
Great appraising John. Right up my street as you know!
I wanted to thank you for citing the Catalogue of Bias entry on immortal time bias. I’m chuffed to see it being used for exactly the reason it should be - educating folk on what biased and unbiased research looks like.
As finally turned Marcia Angell, this is all about, and only about, following the dollars. And the government dollars, extorted from each of us under penalty of incarceration, are as corrupt, and perhaps even more corrupt, than the institutional dollars. At least we understand what the companies want prima facie.
This remains hopeless until other, better ways are found for driving/financing research. I published extensively early in my career -- some relatively important papers that changed the way medicine was practiced and put into place and are still followed, and some, shall we say, less important. But in all cases, none of this was funded by anyone. I begged/borrowed what I needed, got tiny grants from the department that came from practice funds, and that was how things worked. I was paid a pittance but I was in medicine for making health better -- not getting a pharma/government granted/coerced sinecure. And I think most of the research was appreciably better.
No research seems to be even contemplated today until "grant funding" is first discussed, decided and achieved. Much of the research is aimed at what will be funded rather than what will be important to patients. (The example in this article is illustrative.) Just reading the disclosures is enough to discredit any conclusions.
There are still rules for granting continuing education credit that someone who profits from or who has been funded by what they are doing cannot give lectures granting CE credit on those topics. Sadly, this appears to be the last place where conflicts of interest actually "do" something.
Again, money corrupts absolutely. Your cynicism will have to join mine and those of many others until the funding is changed. If Covid has taught us anything, changing it to the "government" will only make it worse -- now we have no accountability and extorted money -- so even the modest restrictions perhaps imposed by business models are not there.
The brain power of this stack would be well spent proposing a different model for running research without all the impossible-not-to-corrupt funding going on.
This is why even with heart problems, I have no faith in cardiology. I do not believe the heart drug trials are true RCT's with true placebos and being double blinded. I do not believe cholesterol has anything to do with creating heart problems. I do not believe statins are safe. Stents and angioplasty are not the surest options as you guys want us to believe.
My choice of survival products include hawthorn berry, nattokinase, olive leaf, high potency 180 KHU cayenne tincture and various other supplements. Also there are numerous heart healthy foods.
Surgery, except in certain cases, and drugs should be the last line of defense since they will never cure the cause of heart problems. There is always a reason for all heart difficulties and that is beyond the scope of modern medicine to investigate the problem.
I am so grateful to you and others who take the time to notice and report these flawed studies. I find this work to be so hopeful. Previously, I looked at studies pertinent to my field, did a cursory review and generally accepted the conclusions as ‘fact’. This trend (since crazy 2020) in criticism and sharing is a boon to the integrity of medicine (and politics in general).
There are still so many who see the promise of humble but objective science. They are on the alert now in ways that they weren’t before. I don’t think truth and “good” always win. But in our individual lives, nothing else is tolerable.
Hope comes when the number of Individuals who can’t tolerate lies becomes large and strong.
Thank you for this, Doc. This is a great Howard Beale ("Network") moment: "I'm mad as hell, and I'm not going to take this any more!" Me, too; all of us, probably. What do you want to do now?
It's easy to make a diagnosis, to figure out (or at least imply) a root cause analysis. We all learned how to do that by internalizing a model of pathophysiology. We learned how to question the anatomy and physiology, do an appropriate test (maybe get more history, maybe do a detailed and focused physical, maybe order that MRI that everybody seems to want these days), and lay out an appropriate, testable therapy. It's easy to "doctor", but it's hard indeed to heal.
Your diagnoses seem to include poor understanding of statistics, (potentially) inappropriate goals for some humans involved, a possible lapse in the Great Tradition of a classic journal as well as its associated parties. What has the response been by the other parties? You did the history, you performed an intricate, beautiful physical (you should have been a neurologist!), you have at least implied the right therapy. Now what?
I could think of a bunch of ways to use your article: a class in rhetoric, a class in healthcare policy, a grad class in research design, a class in economics. A basis for a Master's thesis if not a dissertation. But this paper, which in its way is maybe as important as the Ionnadis "Why most published research...." (2005) article, depending on your OWN peer review of course, ought to be the challenge for every medical student, every tired and frenzied resident (we were, so I'm guessing that nothing has changed), every clinical and even basic science researcher, and every administrator in healthcare who has forgotten what the mission is... who has forgotten why there are doctors.
And those of us who are doctors - MD, PhD, DO, DC, whatever - need to remember that the origin of the word "doctor", a word which we too often toss around as a way to make a buck and more than a buck, is in fact from the Latin "docere", i.e., "to teach". Doctors teach. We teach sick people, and sick researchers, and sick journals, and sick medical schools, how to be well, how to 'regain function' so the inflammation disappears.
Doc, you have us. You have me, for sure. Like John Perry, I'm ready to become a paid member and demonstrate my solidarity although I might not live to see your therapy be tested and proved. But I am betting that your rhetorical pose is more than just theater: you have genuine despair, and so do many of us. Well, one way out of despair is action. Cure. Effecting change. What can we do? What should we do? One thing for sure: it is time to stop lamenting "Oh cursed spite, that ever I was born to set it right."
What's the next step?
Thank you, Dr. Mandrola, for taking the time to explain this so well in terms a lay person can understand. And thank you for your courage in posting this. Because of pay walls, I (like other members of general public), can't read the full text of the study you criticize so effectively. So that's yet another reason I'm grateful to you for giving us a summary and interpretation which is readily accessible to the public. I feel that the best way I can show my support is by becoming a paid subscribe, so that's what I've just done. Thank you again!
Excellent piece and you bring up two excellent reasons for cynicism.
First, your excellent dissection reveals how this is marketing disguised as science.
Second, the fact that there is no scientific forum in which this can be published is a mark against science. We do discuss studies like these at journal clubs and, as you allude, we keep track of authors' reputations in our heads. Should there be some sort of public reckoning of reputations rather than the whispered discussions in back hallways at each institution? I think about how the computer science community has Stack Overflow where users can gain or lose reputation points (https://stackoverflow.com/help/whats-reputation). It seems like science needs something like this where excellent critiques like this one can be shared and you can earn reputation points for it while disingenuous critiques (or ad hominum attacks - ahem, VP) can get your reputation score downgraded.