Evidence based medicine: misunderstood and in decline
A new generation of physicians has forgotten it in the quest to be 'rising stars'.
What is evidence based medicine (EBM)? EBM is a philosophy to make clinical decisions that prioritize the best available evidence over expert opinion, pathophysiologic reasoning, marketing and bluster. Evidence based medicine has never meant slavish devotion to randomized trials, though, if we are talking about implementing an intervention, and, if RCTs are available, EBM will prioritize those results. Evidence based medicine has never ignored patient preferences. Instead, values, preferences and shared decision making are at the core of the philosophy. EBM is not paralyzed by weak evidence— instead it communicates honestly what is known and unknown, and assesses the patient’s preferences. It can take a chance. But when it does, evidence based medicine also means: to advocate—steadily and devotedly— to generate better evidence on questions where uncertainty remains.
Evidence based medicine would not oppose public health officials advising people to wear masks, but it would ask that those officials conduct randomized trials testing whether or not that recommendation helps. The fact the US CDC and NIAID ran 0 such studies would constitute a dereliction of EBM. EBM would not oppose novel treatments for COVID19, though it would advise that those treatments predominantly be given as part of randomized trials. (Like the UK’s RECOVERY studies). EBM would not be cynical about long-covid in adolescents and kids, but it would ask for controlled studies and longitudinal follow up to tease apart symptoms from the virus vs. from living through two distorted, restricted, tumultuous years. Recently, a paper did just this, and was sobering: finding no link between covid and the symptoms. In fact, most of the controlled evidence is sobering (sorry, Atlantic subscribers)
EBM has always faced the same threats. Arrogance. Doctors just know that ivermectin helps, or that masking a 2 year old makes sense ‘cuz physics.’ Often arrogance is paired with anecdote. “I saw a patient recover after getting ivermectin.”
Avarice also plagues EBM. “Everyone should test for COVID19 before parties or gatherings,” says the doctor who is CMO of a covid test company. Do well done randomized studies show that such testing slows the spread? Ignorance is a great threat to EBM. Indeed an outside observer to medicine who doesn’t know about CAST or COURAGE or Auto-transplant for Breast Cancer or the Swan Ganz catheter has difficulty understanding that even smart doctors can be mislead by lower levels of evidence.
Anxiety is a threat to EBM. When we fear for our own safety, we are happy to recommend unproven interventions, particularly on the young, and vulnerable. Especially on our political enemies.
Innovators can also be a threat to EBM. Currently, there are thousands of innovators in artificial intelligence or causal inference. These people do important work, but they can easily be plagued by bias. The biggest bias is that they want to debut their practice as fast as possible.
The target trial is one ‘innovation’ in causal inference. It might be better than traditional observational studies, but whether it rivals RCTs is unknown. The key test is simple: pick 50 RCTs that have been launched and try to predict the results without knowing the results. If the agreement is high, the method is a success. Yet, preliminary efforts show substantial disagreement, and instead of doing this experiment, proponents highlight expert endorsements and selected examples of agreement.
A new generation of doctors has entered the scene, and they have hopelessly confused advocacy and evidence. Huge belief structures in their mind know the ‘right’ thing to do, even when there is no evidence at all. Even if you share their goals, it is not clear that their methods are effective. They know how to ‘reform’ medical eduction and select applicants, but don’t consider that their proposals need some empirical testing. Their world-view constantly conflates morality and evidence— paralyzing real progress.
No one wants children to die from COVID— their logic goes— therefore “only bad people criticize masking— a simple, low cost thing we can do”. Here is where their train derails. We can agree on the goal, but there is simply no good evidence the solution achieves it. Worse, it is divisive. It is clearly political— liberals like to do it, conservatives don’t— and if ineffective (PS likely is) it is a painful distraction from interventions that might have worked (paid sick leave, better backup childcare, focusing on protecting vulnerable, overweight kids).
And, at some point, it becomes clear that all children will get COVID19 anyway. The evidence that vaccinating a health child lowers the risk of death is incredibly poor. The strongest studies are case control studies with controls that fundamentally come from different populations— an eternal and classic fallacy in medical research. Worse, there is zero evidence that vaccinating a health kid who had COVID improves any outcome other than Pfizer sales.
Of course, it is not just COVID19. Many propose universal and repeated genetic sequencing for cancer patients. That costs money, but does universal genetic sequencing improve outcomes? Ans: no data, and previously I have proposed the trial that could assess it.
Others want AI to review colonoscopies in real time with practitioners. Wait, does colonoscopy itself improve overall survival or overall morbidity? Nordic doesn’t look so good. Second, if it does, is it better than flex sig or FIT? Ans: unknown (study ongoing). Finally, adding AI may clip more polyps, but does it improve outcomes? Ans: unknown. Worse, there are no suitable ongoing studies that will shed clarity.
The appetite to call for better studies in medicine is lacking. A screening program creates jobs and revenue, but does it actually help people? It’s not specific to GI. What about lung cancer screening? We have massive programs to corral smokers into CT scanners. Is overall survival improved? NLST showed no OS benefit in updated follow up and worse, the control arm was inappropriate. Nelson has no benefit in OS. The harms and inconvenience are massive. Thousands of researchers devote their lives to implementing and improving CT screening for lung cancer, swallowing buckets of NIH cash, but we have no idea if smokers are actually better off. And there are zero ongoing studies with the ability to ascertain this.
For some cancers, such as multiple myeloma, EBM is dead and buried. Providers keep pushing to treat early, asymptomatic conditions— driving up market share, and putting tens of thousands on anti-cancer drugs costing 600,000 per annum or more. There are no ongoing studies that can test whether this improves survival or quality of life over observation. The two conducted studies are grievously flawed. Yet, the myeloma profession pushes forward on treating pre-cancer like a bobblehead doll on the dashboard— nodding mindlessly— there are even uncontrolled trials of CAR-T opening now. The IRB rubber stamps this madness.
Every day on twitter, I see a junior person in medicine called a “rising star” for presenting a poster or pushing the canonical narrative. Expanding medicine without appropriate evidence is the stuff stars are made of, supposedly. Evidence, after all, means you might fail, and at a minimum, it will take more effort to change practice. If we demanded good evidence to anoint ‘rising stars’ then most of us would never rise above the horizon line.
The battles over masking are being fought in the pages of the New York Times by people who are ignorant of the history of evidence in medicine. Tom Jefferson, a man who joined the Cochrane review in 2006, is locked in debate about masking with those who pressured CDC to recommend masking, including for children, and said nothing when the recommendation went down to 2 years old— and all based on no credible evidence, and with no studies ongoing, which they opposed as unethical. The Cochrane review does not include mannequin studies or retrospective observational studies— because these are not reliable in medicine. Yet, proponents cite these as if they tell us something about the messy reality of policy.
Final point: a wise man recently asked me how often RCTs agree with one another. The answer is they are not perfect, and initial RCTs that show benefit often have benefit eroded in pooled point estimates with more trials. Thus, a future EBM— one that cares about truth— may wish to push beyond initial RCTs, demanding confirmatory ones. Years ago, we did this more often, and elsewhere we explain their value. And I have written papers pushing for confirmatory trials .
Evidence based medicine is delicate, vulnerable, but also profoundly beautiful and rational. It alone is the logical way to implement and pursue costly, invasive, intrusive, and disputed medical and public health interventions. As such, it is a forced move in human development. Even if it withers and dies in the next few years— and it is well on its way to demise— it will be reborn and rekindled in a future moment. It the telos of progress in science. If only the stars could see that.
Very well written, and I completely agree with your perspective. At the tender age of 75 I am still practicing medicine and I intend to keep doing so for the decades to come!
While I agree with Vinay on the whole I think he makes certain classical mistakes, which is surprising given his extensive criticism of so many poorly conducted RCTs involving cancer drugs.
Take Invermectin as an example (and I've no idea whether it's useful or not for COVID). There are RCTs that claim it works (e.g. the Brazil trial) and others (e.g. the RECOVERY trial) that claim it doesn't. However, the UK RECOVERY TRIAL was highly flawed as it included patients up to 14 days from testing positive for COVID. Well under those circumstances of course the trial is going to be negative for Ivermectin as any antiviral really has to be given within 48 hours of the onset of symptoms. Further it is generally the case that any single drug is not useful for a viral illness. So if ohne is going to look at ivermectin or hydroxychloroquine in a trial the treatment arm should comprise the complete care regimen which includes azithromycin or doxycycline, prednisone, zinc, vitamin D, vitamin K2 etc...... Failure to do so in an RCT simply provides no useful information whatsoever.
Likewise, Vinay's comments on colonoscopies is probably flawed because one has to consider the individual and the overall status of the individual. e.g. Take an 85 yr old patient. If that patient has other issues (e.g. cardiovascular disease, etc...) then finding polyps and resecting those in a colonoscopy is unlikely to prolong OS as the patient will likely die from other causes before the colon cancer gets to him/her. however, if the patient is completely fit physically and medically, then a colonoscopy could easily prolong OS. But of course for an 85 yr old, that prolongation could be relatively small since very very few are likely to live beyond 100, let alone say 95. So the issue is far more complicated than Vinay portrays. Similarly if you have a fit 45 yr old where a polyp is found on a colonoscopy, then it is self-evident that resection will prevent the development of colon cancer from that polyp. But of course, continual follow-up is required since other polyps may develop.