Foreign-trained vs US-trained doctors

A public health leader recently cited a 2017 BMJ study comparing quality of care between US and foreign-trained doctors. The flaws in the study offer numerous lessons

The debate of the week on social media centered on the role of immigrants working on US visa programs.

Dr Ashish Jha posted on X an observational study in the BMJ that he co-authored, which found that international medical graduates outperformed US doctors.

The flawed study offers many lessons in critical appraisal.

First a note on Dr. Jha, whom I have not met and mean no malice. Few physicians have more academic credentials: an MD, an MPH, an endowed professorship, former Dean of the Brown University School of Public Health, White House COVID-19 response coordinator from 2022–2023 and frequent TV commentator.

Below is a slide I made of his recent post. The hyperlink at the bottom goes to the study. It has 1.2 million views.

The boxes highlight causal language and what he thought of his study.

The goal of the study was to determine whether patient outcomes differed between hospital doctors who graduated from a foreign vs US medical school. The paper is open access.

The authors used a sample of Medicare patients aged 65 and older who were admitted to a hospital between 2011-14 with a medical diagnosis. This was a big database analysis of over 1.2 million admissions from more than 44K general internists.

The main outcome measure was 30-day death, readmission rates, and costs. The authors obviously had to make many adjustments in patient, doctor and hospital characteristics.

The main findings:

Patients treated by international graduates had lower mortality (adjusted mortality 11.2% v 11.6%; adjusted odds ratio 0.95, 95% confidence interval 0.93 to 0.96; P<0.001)

Readmission rates did not differ. Costs were slightly higher for international-trained doctors. I screenshotted the conclusion below:

---

Aaron Carrol, an academic physician and writer cited (without critical appraisal) the results of the study in the NY Times later that year in 2017.

---

Four Problems with the Study

I Analytic Flexibility

The main problem, and one I hope Sensible Medicine readers will quickly see with these types of large database studies, is the issue of analytic flexibility.

We covered this here in April with a study-of-the-week and then podcast with Dr Dena Zeraatkar, from McMaster University in Canada.

Her group showed in a large database study on the effect of meat consumption on mortality that when you consider choice of covariates and statistical plans there were more than 2000 ways to analyze one data source. And, different choices could lead to different results.

Brian Nosek, a social scientist, had shown similar flexibility in his Many Analysts, One Data Set paper. His group gathered 29 different teams of statistical experts to examine one data set of a European football league. They each made different choices; and two-thirds found positive results while one-third found non-significant results.

The authors of the BMJ paper made reasonable choices of covariates and statistical methods. They did sensitivity analyses to sort out biases. On the surface, it looks like a proper paper.

But we know from the work of Drs Zeraatkar and Nosek that different choices could easily lead to non-significant or even opposite odds ratio—especially since the effect size on mortality was only 0.4%.

II Inability to Adjust for All Variables

In any observational non-random comparison study, there exists the possibility that groups differ in important ways. This analysis is no exception. The authors adjusted for a lot of variables, and they did sensitivity analyses. But you can only adjust for what is in data sheets.

Many intangibles factor in medical decisions. This is why randomization is so important. Randomization balances both known and unknown confounding variables.

III Lack of Concordance between Mortality and Readmissions

The authors find what they determine is a robust and clinically meaningful difference in care. Enough to “save American lives” as Dr Jha called it in his post on X. Yet, if the care of US-trained doctors is so bad, why are readmissions similar?

We are taught that readmission within 30-days is a metric for health care quality. It’s a debatable assumption, but you’d expect that if doctoring was causal in patient mortality it would also be so in readmissions. The discordance suggests that the mortality finding is noise rather than signal.

IV The Study Question

The study’s question is problematic.

Anyone who has cared for patients in a hospital knows that many factors affect patient outcomes. The effect of one doctor on one shift can easily be offset by oodles of other things. Specialist consultations, nursing, respiratory and physical therapy, lab, radiology, surgery, to name a few.

Whether your analytic choices lead to finding foreign graduates were better or worse than US-trained doctors, you can tell a story supporting either finding. The authors explained their results by citing international graduates repeat residencies (one in their home country and one in the US). They also found a study reporting that international graduates have higher test scores.

But if the analytic choices led to US-trained doctors having better outcomes, authors could easily find citations to support these findings—though I doubt such a paper would be submitted or accepted for publication. (Publication bias is for another post.)

Summary:

I have seen healthcare and medical education in many countries. I have no reason to doubt the skill of internationally-trained doctors.

But such a study cannot support or refute any differences in care between individual clinicians.

I now believe the complexity of modern-day healthcare has rendered any effort to judge the effect of one (of many) clinicians as pure folly.

We can (and should) study policy or systems through cluster randomization, but not the effect of one clinician.

The BMJ paper is a shining example of how observational studies fail. That a health leader would cite it publicly with causal language tempts me to be cynical about our leadership in Medicine.

---

As we close another year at Sensible Medicine, I want to thank our readers, commenters and subscribers. Vinay, Adam and I will strive to grow and improve this project in independent critical thinking in Medicine. JMM