If the early stopping rule for a trial is “Stop the moment the treatment is better by a ‘statistically significant amount’, then we are guaranteed to create some false positives. Because looking at any trial over time, it’s results form a random walk. And random walks will often go on excursions to one side that are different than where they finally end.
Add to that the possibility that the time for “good effects” and “bad side effects” to manifest can be different due to biological issues.
It would be nice if governments could find a way of continuing trials on all drugs to find out benefits and bad effects.
Mining for information in government health organisations springs to mind. The NHS in UK is one of the biggest organisations in the world but somehow they can't bring themselves to look.
Big Pharma are terrified that it will show up harms and positive effects for out of patent medicines.
New purposes for old drugs cost them a fortune. Examples such as methotrexate for arthtitis instead of new drugs such as baricitinib = cents versus $1000.00+ per packet.
Imagine the loses if ivermectin had been allowed instead of experimental vaccines with dodgy side effects.
Remember that for a medicine to be effective it has to have effects. None are harmless. You hope you have a good effect not a bad effect when you take them.
Stopping the SARS-CoV-2 mRNA RCTs (unblinding and crossing over early) is a great example of the high price of not completing trials due to the supposed benefit of the intervention.
Now reliant on observational data, we lack the ability to arrive at firm numbers for efficacy, safety and frequency of certain more common adverse events. The damage done by not completing this trial, and never following up with another once the COVID dust settled a bit, plays no small part in the bickering and lack of trust in these processes and the products they validate.
I would agree with most all of the comments below that reach the common sense conclusion that early stoppage for "benefits" usually reflects only the biases of the investigators and/or the interests of the funders. A couple of studies you discussed recently were stopped with relatively insignificant differences between the experimental and control groups. They were stopped as soon as they achieved the holy grail of "statistical significance" lest continuation of the trial might reverse that trivial difference.
The more interesting question to me is how often are these early peaks at results planned ahead of time in the study protocol vs a deviation from the study protocol? There have been a lot of books written on the topic, but it seems like early looks at results are often used as an intentional strategy to increase the amount of false positives a trial produces. If I had to bet, I’d say that a lot of these early looks at data represent deviation from study protocol. If this is true, it’s just another way in which a lot of our supposedly great evidence based medicine is really just manipulation and marketing.
OK. Let's create a process ("science") to determine if a "treatment" is safe and effective. Then, let's break that process because a treatment MIGHT be beneficial for a few people, thus preventing us from any confidence about that treatment for the larger population. What science giveth, it can taketh away.
Waiting for the final results of a RCT is hard. The scientists conducting it start acting like kids on a long car trip: "Mommy, are we there yet?"!
Is there even one drug trial of any kind that is 100% honest and not infected with big pharma hubris and arrogance? Yes, thousands of them be we never hear about them because they were all failures. I really couldn't care less since I am an avowed big pharma drug denier. And I don't care if they hire God to market their garbage.
Did these "The group of experts (who) judge" by any chance come from the Pharma company where the meds were manufactured? OR where they Docs that got paid by them?
Maybe the second one wasn't stopped because guess who funded it? BAYER
All voting members of the ARRIVE Executive Committee (JMG, CB, CC, HD, PBG, GH, TAP, PMR, LMR, MT, and GT) received personal fees from Bayer during the conduct of the study. RC is an employee of Bayer and was a non-voting member of the Executive Committee. PMR reports personal fees from Bristol-Myers Squibb. LMR reports personal fees from Novartis, Sanofi, Medtronic, Daiichi-Sankyo, and grant funding from AstraZeneca. MT reports personal fees from Celyad, Janssen Cilag, Kowa, Perfuse Group, and Servier, outside the submitted work.
Not surprised by your “million views per month”. If we could design a proper RCT to discover reasons for your success, I believe we would find a strong correlation with the loss of confidence in our public health institutions and pharma industry. (In addition to the excellence of your writing, of course)
It seems all too easy to forget, in the 'rush to treat' and lack thereof being 'unethical', that those in the placebo arm of a trial are in fact simply maintaining their pre trial status. Leaving placebo arms intact is, to me, and essential part of any trial else we lose the bigger picture. Interrupting the placebo arm throws out valuable comparison information. Participants in any trial know beforehand that they may get placebo, for which they may ultimately be grateful, and to throw the placebo arm out early makes a mockery of the trial.
Isn't it called optional stopping? I. E. Stopping when you get the results that you want. Be it due to "ethical" reasons or not. It still is one of the questionable research practices for a reason. It's playing with randomness to potentially select the false positives.
Doesn't it then become the question for those supposed ethicists, is it ethical to make patients go through trials, and then because of intentional reasons of either their researchers of ethical review board, that the results of the study are massaged towards false positive results? It's potentially wasting patient's time for results that are false. It's better to ha e a trial that is a true negative, which actually gives important information. Than a trial that wasted people time for a career move or a faulty ethical call. Shouldn't questionable research practices be part of the ethical decision making? Wasting people's time with false results is worse than a trial with true negative results.
Kind of like how the original trial for AZT (a failed chemo drug used to treat AIDS) was stopped early, or how more recently Remdesivir (failed Ebola drug) was stopped early for COVID.
From a 1989 article about AZT
"The most serious problem with the original study, however, is that it was never completed. Seventeen weeks into the study, when more patients had died in the placebo group, the study was stopped, five months prematurely, for “ethical” reasons: It was considered unethical to keep giving people a placebo when the drug might keep them alive longer. Because the study was stopped short, and all subjects were put on AZT, no scientific study can ever be conducted to prove unequivocally whether AZT does prolong life."
"In the study that won FDA approval for AZT, the one fact that swayed the panel of judges was that the AZT group outlived the placebo group by what appeared to be a landslide. The ace card of the study, the one that canceled out the issue of the drug’s enormous toxicity, was that 19 persons had died in the placebo group and only one in the AZT group. The AZT recipients were also showing a lower incidence of opportunistic infections.
While this data staggered the panel that approved the drug, other scientists insisted that it meant nothing — because it was so shabbily gathered, and because of the unblinding. Shortly after the study was stopped, the death rate accelerated in the AZT group. “There was no great difference after a while,” says Dr. Brook, “between the treated and the untreated group.”"
Sure seems like the treatment group was being handled differently to keep them alive for the study. I remember reading somewhere that they were receiving weekly blood transfusions to keep them healthier to counteract AZT effects...
If the early stopping rule for a trial is “Stop the moment the treatment is better by a ‘statistically significant amount’, then we are guaranteed to create some false positives. Because looking at any trial over time, it’s results form a random walk. And random walks will often go on excursions to one side that are different than where they finally end.
Add to that the possibility that the time for “good effects” and “bad side effects” to manifest can be different due to biological issues.
Men are exactly WHY men in general don't give preferential meant based on sex.
It would be nice if governments could find a way of continuing trials on all drugs to find out benefits and bad effects.
Mining for information in government health organisations springs to mind. The NHS in UK is one of the biggest organisations in the world but somehow they can't bring themselves to look.
Big Pharma are terrified that it will show up harms and positive effects for out of patent medicines.
New purposes for old drugs cost them a fortune. Examples such as methotrexate for arthtitis instead of new drugs such as baricitinib = cents versus $1000.00+ per packet.
Imagine the loses if ivermectin had been allowed instead of experimental vaccines with dodgy side effects.
Remember that for a medicine to be effective it has to have effects. None are harmless. You hope you have a good effect not a bad effect when you take them.
Stopping the SARS-CoV-2 mRNA RCTs (unblinding and crossing over early) is a great example of the high price of not completing trials due to the supposed benefit of the intervention.
Now reliant on observational data, we lack the ability to arrive at firm numbers for efficacy, safety and frequency of certain more common adverse events. The damage done by not completing this trial, and never following up with another once the COVID dust settled a bit, plays no small part in the bickering and lack of trust in these processes and the products they validate.
I would agree with most all of the comments below that reach the common sense conclusion that early stoppage for "benefits" usually reflects only the biases of the investigators and/or the interests of the funders. A couple of studies you discussed recently were stopped with relatively insignificant differences between the experimental and control groups. They were stopped as soon as they achieved the holy grail of "statistical significance" lest continuation of the trial might reverse that trivial difference.
The more interesting question to me is how often are these early peaks at results planned ahead of time in the study protocol vs a deviation from the study protocol? There have been a lot of books written on the topic, but it seems like early looks at results are often used as an intentional strategy to increase the amount of false positives a trial produces. If I had to bet, I’d say that a lot of these early looks at data represent deviation from study protocol. If this is true, it’s just another way in which a lot of our supposedly great evidence based medicine is really just manipulation and marketing.
OK. Let's create a process ("science") to determine if a "treatment" is safe and effective. Then, let's break that process because a treatment MIGHT be beneficial for a few people, thus preventing us from any confidence about that treatment for the larger population. What science giveth, it can taketh away.
Waiting for the final results of a RCT is hard. The scientists conducting it start acting like kids on a long car trip: "Mommy, are we there yet?"!
Is there even one drug trial of any kind that is 100% honest and not infected with big pharma hubris and arrogance? Yes, thousands of them be we never hear about them because they were all failures. I really couldn't care less since I am an avowed big pharma drug denier. And I don't care if they hire God to market their garbage.
Now do Comirnaty's unblinking.
Did these "The group of experts (who) judge" by any chance come from the Pharma company where the meds were manufactured? OR where they Docs that got paid by them?
Maybe the second one wasn't stopped because guess who funded it? BAYER
Move along... nothing to see here.
Funding: Bayer.
Copyright © 2018 Elsevier Ltd. All rights reserved.
PubMed Disclaimer
Conflict of interest statement
Declaration of interests
All voting members of the ARRIVE Executive Committee (JMG, CB, CC, HD, PBG, GH, TAP, PMR, LMR, MT, and GT) received personal fees from Bayer during the conduct of the study. RC is an employee of Bayer and was a non-voting member of the Executive Committee. PMR reports personal fees from Bristol-Myers Squibb. LMR reports personal fees from Novartis, Sanofi, Medtronic, Daiichi-Sankyo, and grant funding from AstraZeneca. MT reports personal fees from Celyad, Janssen Cilag, Kowa, Perfuse Group, and Servier, outside the submitted work.
Not surprised by your “million views per month”. If we could design a proper RCT to discover reasons for your success, I believe we would find a strong correlation with the loss of confidence in our public health institutions and pharma industry. (In addition to the excellence of your writing, of course)
It seems all too easy to forget, in the 'rush to treat' and lack thereof being 'unethical', that those in the placebo arm of a trial are in fact simply maintaining their pre trial status. Leaving placebo arms intact is, to me, and essential part of any trial else we lose the bigger picture. Interrupting the placebo arm throws out valuable comparison information. Participants in any trial know beforehand that they may get placebo, for which they may ultimately be grateful, and to throw the placebo arm out early makes a mockery of the trial.
I appreciate this substance greatly. Keep them coming!!!
Isn't it called optional stopping? I. E. Stopping when you get the results that you want. Be it due to "ethical" reasons or not. It still is one of the questionable research practices for a reason. It's playing with randomness to potentially select the false positives.
Doesn't it then become the question for those supposed ethicists, is it ethical to make patients go through trials, and then because of intentional reasons of either their researchers of ethical review board, that the results of the study are massaged towards false positive results? It's potentially wasting patient's time for results that are false. It's better to ha e a trial that is a true negative, which actually gives important information. Than a trial that wasted people time for a career move or a faulty ethical call. Shouldn't questionable research practices be part of the ethical decision making? Wasting people's time with false results is worse than a trial with true negative results.
Always learning from you, John. Thanks.
Kind of like how the original trial for AZT (a failed chemo drug used to treat AIDS) was stopped early, or how more recently Remdesivir (failed Ebola drug) was stopped early for COVID.
From a 1989 article about AZT
"The most serious problem with the original study, however, is that it was never completed. Seventeen weeks into the study, when more patients had died in the placebo group, the study was stopped, five months prematurely, for “ethical” reasons: It was considered unethical to keep giving people a placebo when the drug might keep them alive longer. Because the study was stopped short, and all subjects were put on AZT, no scientific study can ever be conducted to prove unequivocally whether AZT does prolong life."
"In the study that won FDA approval for AZT, the one fact that swayed the panel of judges was that the AZT group outlived the placebo group by what appeared to be a landslide. The ace card of the study, the one that canceled out the issue of the drug’s enormous toxicity, was that 19 persons had died in the placebo group and only one in the AZT group. The AZT recipients were also showing a lower incidence of opportunistic infections.
While this data staggered the panel that approved the drug, other scientists insisted that it meant nothing — because it was so shabbily gathered, and because of the unblinding. Shortly after the study was stopped, the death rate accelerated in the AZT group. “There was no great difference after a while,” says Dr. Brook, “between the treated and the untreated group.”"
Sure seems like the treatment group was being handled differently to keep them alive for the study. I remember reading somewhere that they were receiving weekly blood transfusions to keep them healthier to counteract AZT effects...
https://www.spin.com/2015/10/aids-and-the-azt-scandal-spin-1989-feature-sins-of-omission/