The Study of the Week Looks Back and Asks One Question
What is going on with medical science?
I am worried about the state of medical science and our journals.
At the risk of sounding naive, I want to believe that people who choose academic medicine do so to answer important scientific questions.
I’ve traveled the world and met academics. Many of them impress me as serious people trying to advance the field.
But that observation is hard to square with some of the studies that make it into big journals—and then into the lay press.
Today let’s do a bit of meta-analysis of recent examples.
Example 1:
Last week, Professor James McCormack described a subgroup analysis of a trial studying a new cholesterol-lowering drug called bempedoic acid.
A subgroup analysis of a trial involves comparing two types of patients within the main trial group—for instance, diabetic/non-diabetic, male/female, older than 55/younger than 55, etc.
JAMA published the subgroup analysis of primary/secondary prevention patients. But. But. They only presented the data for the primary prevention arm—which happened to have the stronger results.
Here is the thing: everyone knew this was a problem. The authors of the paper knew it. The peer reviewers and editors and statisticians also knew it. But they did it anyways. And more than 80 media outlets covered the study. One small caveat: JAMA editors at least allowed a critical editorial.
Example 2
I wrote yesterday about the MONITOR HF trial of an expensive device placed in the pulmonary artery to help manage patients with heart failure.
The trial was designed without a proper placebo arm. As an open-label trial, one group knew that they got the procedure and fancy device and the other group knew that they did not. The primary endpoint was subjective: how patients score their health on a questionnaire.
Here is the thing: everyone knew this was wrong. The authors knew that a subjective endpoint required proper blinding. The editors and peer reviewers also knew it. But there it is, published in the top journal, The Lancet.
Along with that fatally flawed study, was a glowing editorial from an author who declared strong financial relationships with the company that makes the device.
Example 3
This winter, I wrote about the ATLAS trial comparing the subcutaneous ICD to a regular transvenous ICD. The S-ICD is a more expensive device that is heavily marketed to doctors in my field.
Get this. To compare these two devices, the authors chose an endpoint with 5 bad outcomes from the implant. But. But. Four of the five endpoints could only occur in the transvenous ICD arm.
When the positive results came out, so did the marketing materials.
Here is the thing: the authors of this study come from one of the world’s top research organizations. They knew. The editors and peer reviewers of the Annals of Internal Medicine also knew the fatal design flaw. But there it is, published in a prominent journal.
Example 4
Of all journals, we expect the utmost from the New England Journal of Medicine. They hold the loftiest reputation. One reason is that they rarely publish observational studies that attempt to make causal claims.
But in November of 2022, the NEJM published an observational comparison of masking in schools. This was an attempt to compare—in retrospect without randomization—two groups of individuals. I have written often about the problems with this technique, namely, without randomization, you can’t tell if the groups were similar.
Here at Sensible Medicine, Tracy Beth Hoeg, MD, PhD, offered a point-by-point rebuttal of the limitations of this paper. Instead of this sort of critical appraisal, NEJM published a supportive editorial.
In this case, it is possible that the authors actually thought their analysis was correct. But surely the neutral scientists at NEJM knew otherwise. And if so why did they publish such a flawed analysis?
I could go on with more examples. Let me close with two big thoughts.
First is that NEJM, JAMA, The Lancet and the Annals of Internal Medicine are the top journals in all of medicine.
If such biased and flawed papers can not only reach this level, but do so with outsized conclusions and glowing editorials, what are consumers of medical evidence to think of the adjudication process of science?
We all agree that someone or some process has to judge science. But if this can happen, how can we trust all the other papers in these journals? The system seems broken.
The second thought is worse.
In most (if not all) of these examples, the authors knew the fatal flaw in their study but ignored it.
I may be wrong; I hope I am wrong, but if they knew the design and analysis was flawed and passed it off anyways, then…whoa. I don’t want to go there. It makes me extremely sad.
I am happy for anyone in the comments to help me feel better about our current climate of medical evidence. Or suggest a better way.
I would also urge you to think past pure financial conflicts of interest. Money is obviously a driver of conflicts, but it can’t be the only issue.
As an Engineer I have frequently been appalled by the poor levels of evidence, weak analysis, and over the top promotion of medical interventions. I used to hold my medical colleagues in high esteem and rarely checked their work. My journey began with dietary claims when I discovered the poor or outright false statements on that subject. During the pandemic I began reading medical papers. I was stunned by the obvious attempts to denigrate somewhat effective and cheap treatments for the virus. I was still deluded in thinking that the vaccines had been properly tested. But over the intervening months I have learned just how artificial and erroneous the initial reports were. I also discovered how mendacious the authors were. They knew that they were lying about effectiveness and were definitely obscuring the abysmal safety profile of these vaccines. I no longer trust the medical community in general. I trust a few doctors who I know to be honest. But now I check everything.
That feeling you have is cognitive dissonance. It must be very difficult for those in the medical field to admit that a lot of what their chosen profession is doing and has done is nothing but nonsense. There have been many advances in medicine over the last 150 years or so but the incredible proliferation of pharmaceuticals seems to have had little to no positive benefit on the overall health and well being of the US population. As the pharmaceutical companies have grown and continued to thrive the US population has become more and more unhealthy as a whole and at some point the medical community needs to own up to this failure. Well meaning doctors simply prescribe more and more medications often as treatments for side effects of a previous prescription and before you know it you have a patient with a cabinet full of medications the interactions of which no one could possibly understand. Further causing issues is the fact that almost no one in the medical profession has any clue about nutrition and nutrition "science" has been even worse over the last 100 years than the "science" of pharmaceuticals. If I have an infection, a trauma, cancer, or need a surgery I have full confidence in the medical profession and am grateful for the advances of the last 150 years. If instead I want to remain healthy, understand what to eat, know what supplementation to take, know how to exercise to maintain health, understand what environmental toxins might be effecting me, etc. the medical profession is absolutely abysmal and in fact more likely to give me false or at least uninformed information but deliver that information with complete confidence and often utter disdain for me even asking the questions. Most doctors should stick to treating things like infections and traumas and be humble enough to admit that in large part they are clueless about the rest of it but they don't do this. There seems to be a hubris in much of the medical community that leads them to believe they are just smarter than everyone else even on things they frankly have no clue about.
All of that said I do think this substack is at least an honest attempt to begin to talk about some of these problems and show some of the necessary humility but I don't have much faith in the medical community as a whole taking this approach. Physician, heal thyself and then start to worry about the rest of us.