Bad Science Needs to Be Called Out

The medical profession loses trust when bad studies are used to promote policy

This will be a short column about a study that should have never been conceived, performed or published.

JAMA-IM published an observational study called:

Excess Death Rates for Republican and Democratic Registered Voters in Florida and Ohio during the COVID-19 Pandemic.

The conclusion reads: (emphasis mine)

In this cross-sectional study, an association was observed between political party affiliation and excess deaths in Ohio and Florida after COVID-19 vaccines were available to all adults. These findings suggest that differences in vaccination attitudes and reported uptake between Republican and Democratic voters may have been factors in the severity and trajectory of the pandemic in the US.

In just 6 days, it has garnered more than 73,000 views and been covered by 55 news agencies. NPR’s title is representative of most of the media coverage.

My original draft included a tweet from a leader in public health. Her comment indicated that she had made causal conclusions from the study. I decided against singling one person out. Instead, Altmetric keeps the receipts. The link goes to a list of 4603 tweets, which went to more than 29 million followers.

---

This is a terrible turn for the medical profession.

The first problem is technical. Vinay Prasad covers the details on his Substack.

I won’t repeat the many limitations of this paper. The simple fact is that it is an observational study looking back at associations. Even the highest quality non-random comparisons suffer from substantial threats of confounding factors. It borders on anti-science to make causal inferences from highly biased studies. Experts in public health who amplified this study surely know that.

The authors find an association between higher death rates based on voter registration but then strongly suggest that vaccine denial was the cause. Yes, of course, they include the usual ‘limitations’ section buried at the end of the paper. But their feelings are clear.

Such foolish leaps of causation are common in the medical literature. But this is worse because it is nakedly political.

The authors implicate vaccine denial but they did not have vaccine records. They did not control for basic things like socio-economic status, BMI, diabetes.

That leads me to the second problem: that this analysis was conceived. These are Yale scientists. Why would they spend time and effort on a project that had no chance of reliably answering a question? And. They would have been unlikely to write this manuscript if the results were different.

You can’t get a PhD and not know that confounding factors shred the ability to infer causality. Yet they did it anyways.

This saddens me because a normal person, a neutral person, a person who wants to believe in the use of evidence to make better decisions will think: wait, if top scientists from elite schools are doing such studies, how can I believe anything in the scientific process? People used to think academics join the academy to discover truth.

The third problem comes in the judging and publishing of science. Current thinking holds that editorial and peer review provide confidence in a study’s result. The knock on preprint servers is that they aren’t “peer-reviewed.”

Yet this paper made it through review at one of the top medical journals. JAMA-IM is my favorite journal because it routinely published studies that challenged the power centers of medicine. These were often studies that did not draw media attention.

Now it owns an obviously politically-motivated flawed study that they knew would get attention. The paper doesn’t include an editorial, so, if any of the editors come across this column, Vinay and I are available to write one.

My purpose in writing this is not to rehash COVID-19 debates.

My message is a plea to colleagues to resist the urge to conceive, perform and publish studies like this. And to leaders of medical science not to make the mistake of inferring causality so publicly.

If you want to tweet out a study like this why not take the opportunity to teach the public how to appraise it?

A broken observational study about a cardiac device is one thing. A broken study that favors policy positions destroys trust in our profession.

Doctors and scientists should be like the best judges—neutral. Otherwise no one will believe us when we get the correct answers.

JMM

---

Our goal here at Sensible Medicine is to be like that neutral judge. When studies are good, we say so. When studies are bad, we see learning opportunities. We appreciate your support. Feel free to disagree with me/us in the comment section.