Adam Cifu and Vinay Prasad are Both Wrong about A New Masking Trial
By Ben Recht
Last week Adam Cifu and Vinay Prasad offered dueling interpretations of a randomized trial of masking. These posts were read by nearly a hundred thousand readers, and you can read them below. One reader, Ben Recht, a Berkeley Professor, thinks both of them are wrong. This is his column.
Ben Recht
This week blessed us with our fourth RCT on the benefits of masking in the community since the start of the COVID-19 pandemic. I’ve been enjoying the heated debate here on Sensible Medicine and on the website formerly known as Twitter. It’s particularly because I respect Drs. Prasad and Cifu so much that I wanted to take the time to offer my argument for why they are both wrong about this study. We should use this study as a case of how not to design medical trials.
The hypothetical masking intervention we’d love to study goes something like this: A person wears a surgical mask out in public. They don’t change anything else about their behavior. The masking results in some effect on their likelihood of catching a respiratory infection.
This is not what Solberg et al. studied.
What was the intervention?
The treatment in this study is not mask wearing. The investigators emailed the trial participants. Half of the participants were instructed to wear masks when out in public. These participants received a gift card code in this email so they could purchase a 50-pack of masks at their local pharmacy. The control group was explicitly instructed not to wear masks. To reiterate, the intervention was an email with instructions and a gift code.
What was the outcome?
The outcome was not illness. After 17 days, the investigators sent the participants a second email asking them to complete an online symptom survey with some questions about behavior. A subject was marked “sick” according to a complicated Excel macro:
“A positive response to having experienced symptoms of a cold or COVID-19, and having experienced fever and one respiratory symptom (stuffy or runny nose, sore throat, coughing, sneezing, heavy breathing); or one respiratory symptom and at least two other symptoms (body ache, muscular pain, fatigue, reduced appetite, stomach pain, headache, loss of smell).”
To reiterate, the outcome was clicking the right boxes in an online survey.
What did we learn?
This study's surrogate treatment and outcome are so many steps away from the causal chain we cared about. We wanted the effects of masking in public on respiratory illness. We got the effects of emailing people instructions to mask on how they respond to symptom surveys. This a huge leap from what we cared about and renders the study useless for informing medical practice or policy.
The virtuality of this experiment design is absurd. The study staff never had in-person contact with any participants. We have zero evidence that anyone ever wore a mask! We just have emails.
My final problem here is a more general issue with trial analysis. Everyone jumps to the topline result presented in the dumb infographic supplied by the BMJ, and no one reads the numerous qualifications that litter the report.
This graphic advertises an absolute risk difference of -3.2%. It is not this simple, and it’s creepily paternalistic that medical journals believe this is helpful science communication.
That -3.2% difference means that of the people who responded to the symptom survey, 9% of the treatment group reported symptoms while 12% of the control group reported symptoms.
But there’s an even bigger signal upstream. Only 80% of the treatment group completed the survey, while 87% of the control group did. That’s a 7% difference. For those who care about this sort of thing, the p-value associated with the symptom difference is less than 0.0001. That is, less than 1 in ten thousand. The p-value associated with the difference in reporting is less than 0.00000000001. That is, less than 1 in one hundred billion.
We must conclude that emailing people instructing them to mask has a much greater effect on responding to follow-up surveys than on admitting cold symptoms. Why did so few people in the treatment group reply to their email? Who knows? We’ll never know. Replace “mask” with any other intervention, and you will see that this trial only induces arguing among partisans. Just throw this study in the bin.
I empathize with the desire to build rigorous evidence bases for all medical treatments, but not every scientific question can be answered by a randomized clinical trial. Just because someone randomized doesn’t mean we learned anything. We don’t need to incorporate every study into our decision-making processes. One can make the argument that most published research findings are false, but some aren’t even useful.
Can this even be called a randomized *clinical* trial? They have not directly observed or treated any patients, they haven't even directly interacted with anyone at all, much less confirmed any of their actions.
I can't understand why BMJ would even accept a trial with such a design. It's completely useless.
Brilliant.
Another variable would be social isolation? I have several family members that are very “ Covid cautious”, which means that they meticulously avoid most interactions with others that are not as cautious as they are. If you’re not going out into the public or only going out minimally, I’m pretty sure you’ll show a reduction in infection? ( One has to be “ around” germs to catch one?) I would propose that folks that are willing to faithfully wear a mask at all times might also be more isolative? Giving us an unknown confounding variable?