what type of study, healthy skepticism, and a balance of the magnitude of benefit versus risk is essential. The fact that some of the reanalysis were done by initial investigators does not give me great comfort that their assessment is less likely to be biased. Otherthan that can’t think of a way to add to your thoughts.
May I also recommend the following paper? They crowdsourced the analytic methods of a data set looking at penalties in soccer. The findings are intriguing. Similar to your use of the cited work above, this is one of the cited papers in my "Limitations of Evidence Based Medicine" lecture to the residents.
I agree that reanalysis which “confirms” prior results adds to robustness of initial findings.
However, if reanalysis finds “different “ results, given that it’s entirely retrospective post hoc in nature, I’m not sure it is anything more than hypothesis-generating. Esp when it’s done by original authors. I would also be very curious as to how many different analyses were performed in order to find the sweet spot of generating a different result.
Of the 13 studies in “table 3”, 11 found some expansion of initial results (from initial negative to some version of positive, or from positive to a more expansive positive). I would take those with large grains of salt.
However, I do agree that more availability of data from studies at least for independent review and confirmation would be a net good.
Imagine a clinical trial 100% free from big pharma interference, totally independent and conducted with no game playing. Then there would be no need for revisions and guessing. Whatever the results are, that's what the results are.
Of course this a fictional setting as we know that the ONLY purpose of any drug trial is to make all drugs look good so that new and more expensive options can be brought to market. The goal is NEVER better health choices, but more costly drugs even though they may essentially be harmful and worthless. There is no question as to how this game is played by big pharma.
I’m not in medicine, but this, among so many other eye openers since 2020 has me doubting the validity of much of modern medicine. The worst part is, doctors rely on and fully believe in these studies and they will double down if a patient questions the integrity of the studies, or offers another viewpoint regarding care. Example: I have questioned a cardiology practitioner about a taking a recommended statin because there is a ton of conversation and some studies around ldl and total cholesterol not being good markers of heart disease. She barked, “stop watching television!” I kindly left it at that. PS. I don’t watch television…too many pharma ads. 😁
I take zero medications. Feel good except for TMJ pain which affects my sleep, but I am trouble shooting it. So many potential causes. Going to check out airway obstruction next. Dentists are getting into this business big time.
I really don’t fault the doctors. They don’t have the time to review studies. The system forces them to rely on the dictates of the standard of care. I feel like their hands are tied. They are codependent.
I stopped going to the doctor because I don’t trust the standard of care anymore. I reserve medical care mostly for emergencies. I am hopeful that changes will be made to the system. Until then, I am discovering alternative care to be of value. That, and getting the basics right: nutritious food, clean water, no chemicals/preservatives/additives/seed oils, etc, improved sleep, sunshine, movement/exercise, Vitamin D blood level over 50ng/ml, magnesium, no tobacco or alcohol, good oral care, good relationships. Sounds hokey to many, but getting the base components of health established goes a long way.
I don’t drink coffee either but I’ve read arguments of it being healthy and many arguing it’s unhealthy. It’s probably a mix of both. I just don’t want it to stain my teeth. And organic, mold free coffee is expensive. I have to prioritize my organic food budget.
Even if medical journals require that all authors make their de-identified data and codes available, there is no incentive for most researchers to reanalyze a study unless they don't care about advancing their career. Publications help advance careers. Unlike economic journals, medical journals rarely if ever publish replication papers. Incentives matter.
I believe that we have a better chance to get replication papers published in medical journals now than ever. Jay Bhattacharya is an advocate for the Open Science movement. He has been nominated to be the Director of NIH. (My fingers are crossed.) Senator Bill Cassidy, a gastroenterologist and author of the white paper "NIH In the 21st Century: Ensuring Transparency and American Biomedical Leadership" will be Chairman of the Senate's HELP Committee. The NIH is in the process of revising their policies as required by a bill passed in Congress in the 80's. John Ioannidis, Brian Nosek, Tom Mroz, and I met with Kathryn Bell, who co-authored Cassidy's white paper, about many of the issues that Sensible Medicine has raised.. My ask to Kathryn was for Cassidy to help the NIH to incentivize medical journals to publish replication papers. I believe that this may be the right time, the right place, and the right people to accomplish Sensible Medicine and its readers' goals. If you can help Senator Cassidy, Jay, and the many others who are working to advancie the Open Science movement, please do so. You are sorely needed.
The call for making data widely accessible is clearly correct. The main findings of the re-analysis are only surprising in how seldom there was a reversal. The authors and commentators missed a key point. When you do not use best statistical practices and principles in designing an RCT’s statistical analysis plan, expect more diversity of opinions about the analysis. See https://fharrell.com/post/principles for a start.
One continuous "burr under my saddle blanket" has been the damned trick that uses a particularly odious "shortcut" in lots of RCT -- viz. Composite Outcomes.
This is so concerning! If “evidence” changes depending on how you do the statistics, then is this evidence at all? I think, prior to publication, journals should ask the authors to see if their analysis holds up when looked at it a different way.
One problem is that much of medical researcher's data can be identified. The UK has found a way to de-identify data that is identifiable. They have offered to help us, but so far we have not taken them up on their offer. Sigh. Also it can be hard for authors to give their data and codes to independent researchers in an easy to use format. The American Economic Association has hired a data editor, Lars Vilhuber, to simplify the method for researchers to make their data and codes more easily accessible to QUALIFIED researchers. To be published in the top five economic journals requires that authors make their de-identified data and codes available to independent investigators. I have been told of only one medical journal requires this.
Pfizer's vaccine trial data is so garbled and fraudulent that reanalysis literally can't even be done. Tell Vinay to read these and show them to Dr. Makary as the new FDA comissioner. This product needs to be pulled from the market. Pfizer needs to be sued into oblivion. Doctors need to be in some way reprimanded.
Edit: To clarify, just a few of things that have been found:
- imbalanced comorbidities (i,e, not a randomized trial)
- massive unblinding on several accounts (i.e. not a double-blind trial)
- two different products were tested, the latter of which only in a couple hundred people, with a worse side effect profile, and that's what hte public got (i.e. literally didn't test what they said they tested)
- tested the unvaccinated for covid way more (should not happen if blinded)
- disappeared hundreds of patients
- reclassified side effects as covid cases
- supressed serious injuries
- unnatural time distribution of deaths (i.e. not possible without doing funny business)
- and so much more...
Litmus test in progress for the guys at Sensible Medicine... If they can't wake up, then who the hell can.?
I surely do agree that the Blowtorch of Truth should be ignited and then aimed squarely at the Pfizer "trials" that were ballyhooed from the start. It would be supremely useful to finally see a firm, fair, and full-court-press exegesis of the bullshit that was used by that company to hoodwink many rubes, hicks, and yokels like me.
There is actually a viable communication channel to the top levels of regulators right around the corner. I muse that the bottleneck in that process may just be getting Vinay to read the comments! =)
I support greater scrutiny through full publication of all source data, as you suggest — and thorough post-study analysis by impartial, scientifically-oriented third parties along the lines of the Cochrane Group.
Or should we automatically accept the “coincidence” that 80% of all studies sponsored by a drug company or a device maker display favorable results for the sponsors’ products?
Your post is a reminder, no matter
what type of study, healthy skepticism, and a balance of the magnitude of benefit versus risk is essential. The fact that some of the reanalysis were done by initial investigators does not give me great comfort that their assessment is less likely to be biased. Otherthan that can’t think of a way to add to your thoughts.
May I also recommend the following paper? They crowdsourced the analytic methods of a data set looking at penalties in soccer. The findings are intriguing. Similar to your use of the cited work above, this is one of the cited papers in my "Limitations of Evidence Based Medicine" lecture to the residents.
https://journals.sagepub.com/doi/10.1177/2515245917747646
I agree that reanalysis which “confirms” prior results adds to robustness of initial findings.
However, if reanalysis finds “different “ results, given that it’s entirely retrospective post hoc in nature, I’m not sure it is anything more than hypothesis-generating. Esp when it’s done by original authors. I would also be very curious as to how many different analyses were performed in order to find the sweet spot of generating a different result.
Of the 13 studies in “table 3”, 11 found some expansion of initial results (from initial negative to some version of positive, or from positive to a more expansive positive). I would take those with large grains of salt.
However, I do agree that more availability of data from studies at least for independent review and confirmation would be a net good.
Imagine a clinical trial 100% free from big pharma interference, totally independent and conducted with no game playing. Then there would be no need for revisions and guessing. Whatever the results are, that's what the results are.
Of course this a fictional setting as we know that the ONLY purpose of any drug trial is to make all drugs look good so that new and more expensive options can be brought to market. The goal is NEVER better health choices, but more costly drugs even though they may essentially be harmful and worthless. There is no question as to how this game is played by big pharma.
Elon and Vivek need to establish a platform to channel federal employees who wish to speak up as Whistleblowers.
And to do that now.
Elon and Vivek may wish to inform the Whistleblowers that they have a generous number of positions available to those who come forward now:
Those who are committed to making America great again, to making America healthy again, and to making the federal government transparent again.
I’m not in medicine, but this, among so many other eye openers since 2020 has me doubting the validity of much of modern medicine. The worst part is, doctors rely on and fully believe in these studies and they will double down if a patient questions the integrity of the studies, or offers another viewpoint regarding care. Example: I have questioned a cardiology practitioner about a taking a recommended statin because there is a ton of conversation and some studies around ldl and total cholesterol not being good markers of heart disease. She barked, “stop watching television!” I kindly left it at that. PS. I don’t watch television…too many pharma ads. 😁
I take zero medications. Feel good except for TMJ pain which affects my sleep, but I am trouble shooting it. So many potential causes. Going to check out airway obstruction next. Dentists are getting into this business big time.
I really don’t fault the doctors. They don’t have the time to review studies. The system forces them to rely on the dictates of the standard of care. I feel like their hands are tied. They are codependent.
I stopped going to the doctor because I don’t trust the standard of care anymore. I reserve medical care mostly for emergencies. I am hopeful that changes will be made to the system. Until then, I am discovering alternative care to be of value. That, and getting the basics right: nutritious food, clean water, no chemicals/preservatives/additives/seed oils, etc, improved sleep, sunshine, movement/exercise, Vitamin D blood level over 50ng/ml, magnesium, no tobacco or alcohol, good oral care, good relationships. Sounds hokey to many, but getting the base components of health established goes a long way.
I don’t drink coffee either but I’ve read arguments of it being healthy and many arguing it’s unhealthy. It’s probably a mix of both. I just don’t want it to stain my teeth. And organic, mold free coffee is expensive. I have to prioritize my organic food budget.
Thank you!
Even if medical journals require that all authors make their de-identified data and codes available, there is no incentive for most researchers to reanalyze a study unless they don't care about advancing their career. Publications help advance careers. Unlike economic journals, medical journals rarely if ever publish replication papers. Incentives matter.
I believe that we have a better chance to get replication papers published in medical journals now than ever. Jay Bhattacharya is an advocate for the Open Science movement. He has been nominated to be the Director of NIH. (My fingers are crossed.) Senator Bill Cassidy, a gastroenterologist and author of the white paper "NIH In the 21st Century: Ensuring Transparency and American Biomedical Leadership" will be Chairman of the Senate's HELP Committee. The NIH is in the process of revising their policies as required by a bill passed in Congress in the 80's. John Ioannidis, Brian Nosek, Tom Mroz, and I met with Kathryn Bell, who co-authored Cassidy's white paper, about many of the issues that Sensible Medicine has raised.. My ask to Kathryn was for Cassidy to help the NIH to incentivize medical journals to publish replication papers. I believe that this may be the right time, the right place, and the right people to accomplish Sensible Medicine and its readers' goals. If you can help Senator Cassidy, Jay, and the many others who are working to advancie the Open Science movement, please do so. You are sorely needed.
p.s. I misspoke about Jay. Washing Post reported that he is the top candidate for NIH Director. Trump has not made his final decision. My apologies.
The call for making data widely accessible is clearly correct. The main findings of the re-analysis are only surprising in how seldom there was a reversal. The authors and commentators missed a key point. When you do not use best statistical practices and principles in designing an RCT’s statistical analysis plan, expect more diversity of opinions about the analysis. See https://fharrell.com/post/principles for a start.
Thank you for all your hard work and for supporting my efforts to practice better medicine.
One continuous "burr under my saddle blanket" has been the damned trick that uses a particularly odious "shortcut" in lots of RCT -- viz. Composite Outcomes.
the expression: "blinded by the science" comes to mind.
This is so concerning! If “evidence” changes depending on how you do the statistics, then is this evidence at all? I think, prior to publication, journals should ask the authors to see if their analysis holds up when looked at it a different way.
What are some of the challenges to open data?
One problem is that much of medical researcher's data can be identified. The UK has found a way to de-identify data that is identifiable. They have offered to help us, but so far we have not taken them up on their offer. Sigh. Also it can be hard for authors to give their data and codes to independent researchers in an easy to use format. The American Economic Association has hired a data editor, Lars Vilhuber, to simplify the method for researchers to make their data and codes more easily accessible to QUALIFIED researchers. To be published in the top five economic journals requires that authors make their de-identified data and codes available to independent investigators. I have been told of only one medical journal requires this.
Tolerance for uncertainty. Respect for complexity. Humility. Diligence. Honesty.
The hallmarks of good science.
Thanks for showing how it's done John!
Speaking of reanalysis...
https://openvaet.substack.com/p/pfizerbiontech-c4591001-trial-audit
https://wherearethenumbers.substack.com/p/anomalous-patterns-of-mortality-and
Pfizer's vaccine trial data is so garbled and fraudulent that reanalysis literally can't even be done. Tell Vinay to read these and show them to Dr. Makary as the new FDA comissioner. This product needs to be pulled from the market. Pfizer needs to be sued into oblivion. Doctors need to be in some way reprimanded.
Edit: To clarify, just a few of things that have been found:
- imbalanced comorbidities (i,e, not a randomized trial)
- massive unblinding on several accounts (i.e. not a double-blind trial)
- two different products were tested, the latter of which only in a couple hundred people, with a worse side effect profile, and that's what hte public got (i.e. literally didn't test what they said they tested)
- tested the unvaccinated for covid way more (should not happen if blinded)
- disappeared hundreds of patients
- reclassified side effects as covid cases
- supressed serious injuries
- unnatural time distribution of deaths (i.e. not possible without doing funny business)
- and so much more...
Litmus test in progress for the guys at Sensible Medicine... If they can't wake up, then who the hell can.?
I surely do agree that the Blowtorch of Truth should be ignited and then aimed squarely at the Pfizer "trials" that were ballyhooed from the start. It would be supremely useful to finally see a firm, fair, and full-court-press exegesis of the bullshit that was used by that company to hoodwink many rubes, hicks, and yokels like me.
There is actually a viable communication channel to the top levels of regulators right around the corner. I muse that the bottleneck in that process may just be getting Vinay to read the comments! =)
And a lawsuit was required to get the FDA to release the data. Criminals.
I support greater scrutiny through full publication of all source data, as you suggest — and thorough post-study analysis by impartial, scientifically-oriented third parties along the lines of the Cochrane Group.
Or should we automatically accept the “coincidence” that 80% of all studies sponsored by a drug company or a device maker display favorable results for the sponsors’ products?