How Caring, Informed, Intelligent, and Committed Doctors Can Disagree about COVID Boosters

A Critical Appraisal and Clinical Application

My article last week about the Marsh/Califf JAMA Viewpoint led to some spirited water cooler conversations IRL. I have sworn a dozen times that I was done talking about or writing about COVID but here I am, for the third week in a row... Two weeks ago, there was my mental health, last week there was vaccination, now an “enhanced critical appraisal” of recent article about the COVID vaccine. I guess it is silly to try to avoid the most important topic in medicine so far in the 21^st century.

Most of the feedback I heard was about my Sensible Medicine piece last week was positive. There was the predictable “I can’t believe you’ve bought into this whole vaccine thing” and the “You are shirking your role as a physician by not strongly recommending the COVID booster.” I did get some pushback from one colleague I respect greatly. This person shared an article – and a lot of opinions – with me and got me thinking about how our values, assumptions, and read of the literature interact in shaping our practice. So first, a critical appraisal of a recent Lancet article and then a bit of reflection.

A Critical Appraisal

Undervaccination and severe COVID-19 outcomes: meta-analysis of national cohort studies in England, Northern Ireland, Scotland, and Wales was published in The Lancet on 1/15/2024. It is a well done, observational study of (almost) the entire population of The United Kingdom. Unfortunately, it doesn’t answer the question of whether to support COVID boosters in the winter of 2024 as it compared severe COVID outcomes (hospitalization and death) between 6/1/22 and 9/30/22 in people who were “fully vaccinated” vs. those who were “undervaccinated” as of June 1, 2022.

Design

The authors used electronic health record data to make the comparisons of undervaccinated with fully vaccinated people over five years of age. At the time, fully vaccinated counted as one dose for those aged 5–11 years, two for 12–15 years, three for 16–74 years, and four for 75 years and older. The study is essentially a cohort study for the populations of Northern Ireland, Scotland, and Wales and a case control trial for the English population (with 50 controls per case.)

Recognizing that confounding would be an issue – the vaccinated people would be different from undervaccinated people -- the authors planned for extensive adjustments. The factors included were: age, sex, ethnicity, urban or rural, quintiles of index of deprivation measures and number of Covid risk factors. Other variables are listed on page 4 of the article and details of the COVID risk factors are listed in pages 14-17 of the appendix.

Exposure was measured as doses of vaccine recorded in the medical record. Outcomes were as follows:

A severe COVID-19 outcome was defined as COVID-19 hospitalisation or death. COVID-19 hospitalisation was defined as hospitalisation with … codes for COVID-19 recorded as the first cause of admission and the admission recorded as an emergency in the secondary care records. COVID-19 death was defined as death with … COVID-19 recorded as the underlying cause of death.

(I would have preferred overall hospitalization and overall mortality, but you get you get what you get…).

Results

Given the design, this study was huge, 58.9 million people across the four countries.¹[ii] Not surprisingly the vaccinated and unvaccinated people were quite different. People who were younger, from more deprived backgrounds, were not white, or had a lower number of comorbidities were less likely to be fully vaccinated. To give a flavor, here are some of the differences between groups in the England cohort.

The results showed that undervaccinated people had risks of reaching the endpoints about 1-2 times greater than control for younger groups and 3 times greater for the oldest group. Looking at the difference in outcomes with and without adjustment, there was little change in the older and younger groups. There was qualitative confounding in the 16–74-year-old age group. Below are the main results. Do pay attention to the event rates. These are small for all but the oldest group.

My Conclusion

Going back to 2022, completing what the UK recommended about a year into the pandemic (vs. falling short) reduced your risk of a COVID related hospitalization or death. It is always tricky thinking of absolute risk reduction from observational data but in the highest risk group, you decreased your risk from about 2.4% to about 0.8%. In the low-risk groups, you decreased you risk from almost nothing to a little closer to nothing. Obviously, this was an observational study with all the caveats about confounding but, interestingly, it does not seem like the differences between the vaccinated and undervaccinated groups made that much of a difference. Of course, this is not the only article on the topic. Many others show some benefit. A recent one that Vinay covered on his substack was negative.

So, back to winter 2023/24 where this all began.

As I mentioned above, this article has little to say about my current practice. Most of my patients have gotten at least 3 COVID vaccines and almost all have COVID at least once. If I were “brave” enough to try to extrapolate this 18-month-old observational data to today, I’d expect that boosting would have negligible effect on everybody but the highest risk people. (Certainly, individual risk is only part of the equation with vaccination but, a primary care doctor, I think more about my patient outcomes and less about outcomes over millions of people. I get that I can be criticized for that.) My last post talked about my current approach to boosters. My colleagues is more bullish on boosting and he had some good questions.

Sensible questions:

1. “What the hell do you want, an RCT every 6 months to show that the booster is effective? Next thing you know you’ll be asking for Q 6-month data that parachutes work.”

We have, in fact, argued that evidence should come with an expiration date. Of course, Q 6-month data is asking too much but a 6 month RCT every 2-3 years seems reasonable. I see COVID as being like every other coronavirus. It will continue to mutate, we will always be susceptible, some years will be worse than others, some years the vaccine will be better than others. I would like reliable data that my patients (and I) could use make good decisions.

I am a general internist. Most of what I do, by necessity, does not have a robust evidence base. I extrapolate, reason from pathophysiology, and discuss the possible ways forward with my patients. I wish I could base more of my practice on robust RCTs but that is not what medicine is. I guess this is the art rather than the science. If I am honest, it is probably part of what I love about the practice of medicine.

But, I am unsatisfied relying on art when we are talking about an intervention applied to every American over 65, perhaps twice a year, forever.

2. “We are still early in this. We will get better data. You should be satisfied with observational data for now.”

I really hope we will see better, truly experimental data of COVID boosters. I am not optimistic. The vaccine manufactures have little incentive to run RCTs which might show their boosters are ineffective.²[i] Also, as Vinay Prasad pointed out, it seems that the FDA is satisfied with the status quo.

3. “What is with your hesitancy to accept observational data?”

Doing an observational studies is neither cheap nor easy but it is cheaper and easier than conducting an RCT.³Observational studies, especially in today’s world of advanced statistics, do a pretty good job of predicting the results of RCTs. The problem with observational data is best stated in one of my favorite papers of all time, another JAMA Viewpoint.

Thus, this body of empirical evidence might be summarized quite simply: observational studies are “usually right but sometimes wrong,” and there is no way of knowing when observational study results are reliable. 

OK, my next two articles will have nothing to do with COVID. Maybe I have turned the corner.

1. I am pretty sure that this overestimates the actual number included, as this refers to the overall cohorts while the England analysis was conducted as a case control study.

2. You could argue, with how bad booster uptake has been, that maybe the vaccine makers will eventually calculate that betting on a positive trial will increase rates and is worth funding.

3. Full disclosure, I have conducted neither an observational study or an RCT. I have made a career critiquing other people’s work. Irritating, right?