As George Gooding said, the folks who conducted this trial "have not directly observed or treated any patients, they haven't even directly interacted with anyone at all, much less confirmed any of their actions." Instead of an exceedingly weak trial (as I think both Adam and Vinay would agree) we have a pretty-near-worthless trial.
Unfortunately, we're going to see more of the same. "Virtual" trials (otherwise known as "decentralized" trials) are a growing trend -- and plenty of research funders are hailing them as the wave of the future. Especially drug companies and other commercial funders. Back in 2021 I wrote a guest post for psychiatrist David Healy's blog about Virtual Trials:
At that point, most folks found it hard to believe this was a real trend. But shortly afterwards we learned that this is how many if not most of the key Covid-19 vaccine trials were conducted. The potential for sloppy reporting and even outright fraud seemed large.
But the scariest thing was that research subjects were "reporting" by filling out online questionnaires, making it close to impossible to report the unexpected. All the subjects could really do was to answer the questions “asked” by the online platform. Data regarding effectiveness would be compromised, but data regarding safety would be even worse.
Would love to get the reaction of a few folks with clinical trial experience to the studies described in the last section ("Fraud is not even the biggest problem"). Virtual research is undoubtedly faster and cheaper -- but can it be trusted?
While I realise that trials are expensive to run, an explosion of 'virtual trials' sounds like it would bloat the literature with useless studies (as if that is not already happening). It's great to see Ben taking apart this study from first principles. These days as soon as I read 'self report' I think the study probably tells us nothing useful. Misaligned incentives for researchers (publish or perish) is just churning out a lot of junk science.
I would agree with Ben although there sure isn't much light between what he's saying and what Vinay said.
However, perhaps the more interesting thing is that if one takes the study at face value, the reported 3% risk reduction, even if statistically significant, is actually completely insignificant in practical or public health terms. Further, the reported time period was very short. In reality over a period of say 2 years it is almost certain that one will get some sort of viral upper respiratory tract infection, and over the period of say 5 years it is guaranteed that one will get Covid (which right now is no more than a mild to bad cold). In that light would anybody consider a reduction from 100% risk to 97% risk worth wearing a mask. Now if a mask produced a 90% reduction in absolute risk then we would be talking. Even 50% might be worthwhile, but 3% seriously!
It was my birthday last week. I experienced heavy breathing (married, for a long time…). I didn’t wear a mask (but wife turned off the light). What experimental category for me? 🤔
Wow. As someone with some expertise in statistical analysis (but not medicine), I am appalled that this study ever got out the door, much less past peer review. It's just OBVIOUSLY invalid (for the reasons clearly stated here), and that should have been clear to the authors and the reviewers.
Ben, This is easily the best analysis of the study I have seen and I had many of the same thoughts when I read it. Adam is dying for masks to be proven useful because he "feels" that way. Vinay is dying for any kind of randomized study to be done because (mostly for good reasons) he thinks that "randomization uber alles" is a good approach to research. Your point here that the randomization is true but irrelevant to the conclusions is brilliant and correct.
Maybe you should start a Substack? In any case, thanks for doing this. I will be using this as the definitive piece for demonstrating how useless most medical research is.
Another variable would be social isolation? I have several family members that are very “ Covid cautious”, which means that they meticulously avoid most interactions with others that are not as cautious as they are. If you’re not going out into the public or only going out minimally, I’m pretty sure you’ll show a reduction in infection? ( One has to be “ around” germs to catch one?) I would propose that folks that are willing to faithfully wear a mask at all times might also be more isolative? Giving us an unknown confounding variable?
Sending a single email results in a 3% decrease in some self-reported URI symptoms. I see a whole series of RCTs. What if we send two emails? From two different sources? Does a snail mail result in a larger or smaller reduction than an email? It’s a whole new academic field: the relationship between communication method and virus spread.
There are so many variables to this debate, that we will never know. Intuitively, I don't think masking makes that much difference. And to mask Small children is stupid. We've never done this before. Why now? So it is constantly devolving into political persuasions.
Covid is on the rise again. It is endemic and not pandemic. Which may be an academic distinction.
What is most important and left out of these discussions is the pathogenicity of the virus that almost always becomes less virulent with time. It is interesting that they did quote the absolute difference. Which is small, but then confused the issue with an arcane derivation of p-values.
I agree that this is NOT a study of “masking”. As mentioned in my previous comments, this study design was more akin to Nordicc assessing colonoscopy for colon cancer screening. The parallel is that both involved an “invitation” to a strategy: of obtaining and wearing masks; and of proceeding with a colonoscopy.
I also agree that on its face, studying an intervention at the level of “strategy” seems a bit weird. And I’m very much not a public health practitioner. But I can understand why a study of an intervention as a population strategy has its usefulness, from a policy lens.
But I have the same question about Nordicc as I do about this study. At the end of the day, as the end user, I don’t care so much about the public health strategy. I care much more about the utility of the actual intervention. And this study, much like Nordicc, imo does not give me a really satisfying answer.
I agree that the “infographics as trial summary” can be a disservice at times. And as you’ve rightly emphasized here….the most important part of any paper is the methods section. Any results derived from faulty methodology should be discounted accordingly. And the 2 key phrases in the infographic to me are “pt reported symptoms” and “real world setting”. My heuristic is to consider anything that follows such phrases with large shakers of salt.
I accept your contention that RCT may not be the be-all and end-all. However, I’m not aware of any other type of trial design that can provide an answer on causation. I also acknowledge that the trade-off in RCT can be its external validity and hence generalizability.
Can this even be called a randomized *clinical* trial? They have not directly observed or treated any patients, they haven't even directly interacted with anyone at all, much less confirmed any of their actions.
I can't understand why BMJ would even accept a trial with such a design. It's completely useless.
Excellent!!! More of this please
As George Gooding said, the folks who conducted this trial "have not directly observed or treated any patients, they haven't even directly interacted with anyone at all, much less confirmed any of their actions." Instead of an exceedingly weak trial (as I think both Adam and Vinay would agree) we have a pretty-near-worthless trial.
Unfortunately, we're going to see more of the same. "Virtual" trials (otherwise known as "decentralized" trials) are a growing trend -- and plenty of research funders are hailing them as the wave of the future. Especially drug companies and other commercial funders. Back in 2021 I wrote a guest post for psychiatrist David Healy's blog about Virtual Trials:
https://davidhealy.org/from-virtual-care-to-virtual-research/
At that point, most folks found it hard to believe this was a real trend. But shortly afterwards we learned that this is how many if not most of the key Covid-19 vaccine trials were conducted. The potential for sloppy reporting and even outright fraud seemed large.
But the scariest thing was that research subjects were "reporting" by filling out online questionnaires, making it close to impossible to report the unexpected. All the subjects could really do was to answer the questions “asked” by the online platform. Data regarding effectiveness would be compromised, but data regarding safety would be even worse.
Would love to get the reaction of a few folks with clinical trial experience to the studies described in the last section ("Fraud is not even the biggest problem"). Virtual research is undoubtedly faster and cheaper -- but can it be trusted?
While I realise that trials are expensive to run, an explosion of 'virtual trials' sounds like it would bloat the literature with useless studies (as if that is not already happening). It's great to see Ben taking apart this study from first principles. These days as soon as I read 'self report' I think the study probably tells us nothing useful. Misaligned incentives for researchers (publish or perish) is just churning out a lot of junk science.
I would agree with Ben although there sure isn't much light between what he's saying and what Vinay said.
However, perhaps the more interesting thing is that if one takes the study at face value, the reported 3% risk reduction, even if statistically significant, is actually completely insignificant in practical or public health terms. Further, the reported time period was very short. In reality over a period of say 2 years it is almost certain that one will get some sort of viral upper respiratory tract infection, and over the period of say 5 years it is guaranteed that one will get Covid (which right now is no more than a mild to bad cold). In that light would anybody consider a reduction from 100% risk to 97% risk worth wearing a mask. Now if a mask produced a 90% reduction in absolute risk then we would be talking. Even 50% might be worthwhile, but 3% seriously!
It was my birthday last week. I experienced heavy breathing (married, for a long time…). I didn’t wear a mask (but wife turned off the light). What experimental category for me? 🤔
Wow. As someone with some expertise in statistical analysis (but not medicine), I am appalled that this study ever got out the door, much less past peer review. It's just OBVIOUSLY invalid (for the reasons clearly stated here), and that should have been clear to the authors and the reviewers.
What fraction of medical studies are this bad???
Ben, This is easily the best analysis of the study I have seen and I had many of the same thoughts when I read it. Adam is dying for masks to be proven useful because he "feels" that way. Vinay is dying for any kind of randomized study to be done because (mostly for good reasons) he thinks that "randomization uber alles" is a good approach to research. Your point here that the randomization is true but irrelevant to the conclusions is brilliant and correct.
Maybe you should start a Substack? In any case, thanks for doing this. I will be using this as the definitive piece for demonstrating how useless most medical research is.
Brilliant.
Another variable would be social isolation? I have several family members that are very “ Covid cautious”, which means that they meticulously avoid most interactions with others that are not as cautious as they are. If you’re not going out into the public or only going out minimally, I’m pretty sure you’ll show a reduction in infection? ( One has to be “ around” germs to catch one?) I would propose that folks that are willing to faithfully wear a mask at all times might also be more isolative? Giving us an unknown confounding variable?
Sending a single email results in a 3% decrease in some self-reported URI symptoms. I see a whole series of RCTs. What if we send two emails? From two different sources? Does a snail mail result in a larger or smaller reduction than an email? It’s a whole new academic field: the relationship between communication method and virus spread.
There are so many variables to this debate, that we will never know. Intuitively, I don't think masking makes that much difference. And to mask Small children is stupid. We've never done this before. Why now? So it is constantly devolving into political persuasions.
Covid is on the rise again. It is endemic and not pandemic. Which may be an academic distinction.
What is most important and left out of these discussions is the pathogenicity of the virus that almost always becomes less virulent with time. It is interesting that they did quote the absolute difference. Which is small, but then confused the issue with an arcane derivation of p-values.
I agree that this is NOT a study of “masking”. As mentioned in my previous comments, this study design was more akin to Nordicc assessing colonoscopy for colon cancer screening. The parallel is that both involved an “invitation” to a strategy: of obtaining and wearing masks; and of proceeding with a colonoscopy.
I also agree that on its face, studying an intervention at the level of “strategy” seems a bit weird. And I’m very much not a public health practitioner. But I can understand why a study of an intervention as a population strategy has its usefulness, from a policy lens.
But I have the same question about Nordicc as I do about this study. At the end of the day, as the end user, I don’t care so much about the public health strategy. I care much more about the utility of the actual intervention. And this study, much like Nordicc, imo does not give me a really satisfying answer.
I agree that the “infographics as trial summary” can be a disservice at times. And as you’ve rightly emphasized here….the most important part of any paper is the methods section. Any results derived from faulty methodology should be discounted accordingly. And the 2 key phrases in the infographic to me are “pt reported symptoms” and “real world setting”. My heuristic is to consider anything that follows such phrases with large shakers of salt.
I accept your contention that RCT may not be the be-all and end-all. However, I’m not aware of any other type of trial design that can provide an answer on causation. I also acknowledge that the trade-off in RCT can be its external validity and hence generalizability.
Can this even be called a randomized *clinical* trial? They have not directly observed or treated any patients, they haven't even directly interacted with anyone at all, much less confirmed any of their actions.
I can't understand why BMJ would even accept a trial with such a design. It's completely useless.
What he said. What a joke of a study. Are people actually paid for this 🤔
Read this one!
We all need a laugh from time to time, given the state of the world.
Tom Perry MD
Vancouver, Canada