When Evidence Fails Context

Why Even Good Doctors Miss the Clues That Matter

I think this is a Sensible Medicine First. An insightful post about what ails modern medicine, written by one of my former colleagues, that relates to a work of fiction he is publishing. I would say that we’ve never done product placement before, but I did recently include cigarette ads in a post.

Adam Cifu

One of my favorite studies in all of clinical medicine tells a deceptively simple but deeply important story about the real world of healthcare. In this clever study, trained actor “patients” with audio recorders presented to physicians across Chicago with realistic complaints. In one scripted case, a patient with asthma complained of worsening shortness of breath despite having an inhaler. What he didn’t automatically reveal was his fictitious backstory: he wasn’t actually taking his brand-name asthma medication as prescribed because he had recently lost his job and could no longer afford his expensive inhaler. At a key moment, the actor was trained to drop what should have been an obvious clue:

“Boy, it’s been tough to since I lost my job.”

The correct response, of course, would have been to ask how the job loss was affecting his care, find out that the patient wasn’t actually taking his inhaler, and prescribe a more affordable option. An incorrect response would be to reflexively order expensive lung testing or simply prescribe another inhaler without engaging with the underlying issue.

The results? Only about one in five clinicians in the study provided the right care;¹ most missed the hint entirely. In one striking case, when the actor mentioned he had lost his job, the physician replied, “I’m sorry to hear that. It’s been a rough economy lately. Do you have any allergies?” A classic checkbox question divorced from context.

It’s easy, and by now almost reflexive, to see a study like this and conclude: doctors don’t listen, they’re not empathetic, and they don’t practice evidence-based medicine. Maybe they order more tests because it brings in more revenue, or maybe they just need more training in communication and patient-centered care.

And yes, all of that is at least partially true. A follow-up study showed that recording clinical encounters and giving physicians feedback on missed clues did measurably improve their care. Training and feedback matter. Empathy can be taught. But there’s more to the story.

Because doctors are working inside a system that makes it hard, and at times nearly impossible, to provide exactly the kind of care the patient-actors in the study needed most.

Let’s be clear: most physicians enter medicine for the right reasons, to help people. In my experience, that trend hasn’t decreased; if anything, it’s intensified, especially among younger clinicians. The vast majority don’t look at patients as blank checks. They want to listen and do the right thing. But they’re also human, fallible, and constrained by a system that makes the right thing harder to do than the easy thing.

The doctor’s reflex question, “Do you have any allergies?” is awful, but it also reflects structural pressures:

1. Time pressure and productivity metrics. Clinical encounters are squeezed into 10–15 minute slots because we pay for healthcare by the yard. Even in salaried academic or hospital jobs, physicians are constantly reminded of their RVU targets — a crude measure of how many patients they see and how many tests and procedures they generate. With limited time and relentless expectations, it’s no surprise corners get cut.

2. A bewildering array of quality metrics. Policymakers and regulators, well-intentioned as they may be, have layered on metrics that often have little to do with what matters most to patients. Doctors are prompted to document allergies at every visit, even when they just saw the patient last week and aren’t prescribing anything new. To satisfy these metrics, clinicians must ask a scripted series of questions that, in aggregate, transform care into a transactional checklist.

3. Misaligned incentives. Doctors are human and, like all humans, they respond to incentives. We appreciate a Christmas card or a holiday cookie as much as anyone. But in practice, what gets rewarded is productivity, documentation, and narrowly defined quality measures. Being a good listener, finding a more affordable medicine, or even making the right diagnosis—these aren’t what keep you on the good side of performance reviews.

So where does that leave us?

First, we need to reframe the “bad apple” narrative. Yes, there are bad actors in healthcare, just as there are in every profession. But the patterns we see in these studies are best understood not as proof of individual moral failing, but as the system squeezing physician autonomy in favor of throughput and compliance.

Second, we need to examine the constellation of systems that conspire to those results. Evidence-based practice isn’t dying because clinicians are ignorant or callous, it’s eroding by a thousand cuts. Consequently, improving it won’t be a single, dramatic fix. It will require both big and small changes: from paying physicians within the same specialty the same regardless of RVU productivity, to simple logic in electronic health records that only prompt allergy updates when new medications are prescribed.

I actually wove this exact study into my debut novel, A Dangerous Diagnosis. In the story, the protagonist, a physician caught between clinical conviction and system demands, infiltrates a futuristic startup clinic and plays the “allergy patient” to expose a veneer of care without substance. Early drafts made the clinician who missed the clue look stupid. But that was never my intent. It took a dozen revisions to twist it into the essence: the doctor wasn’t an island unto himself. The final version has him failing to notice the clue because he’s too busy reading scripted prompts in the electronic health record and worrying about the videorecording the startup company was making of the entire encounter. That may be an exaggeration, but only to make the point:

We need more evidence-based medicine, but the first patient we need to treat isn’t the clinician. It’s the system itself.

Shantanu Rai is a medical doctor working on the frontlines of America's healthcare crisis. Raised on Bollywood, whodunits, and brainteasers, he now turns to writing and fiction to reveal the unsettling truths behind the exam room curtain. Read more of his writing on Substack.

1. The lead researcher, Saul Weiner, argues that most doctors in the study actually practiced “evidence-based medicine,” but that what they needed was to practice contextual medicine and evidence-based medicine. This is an important distinction, but here, we are just using the term “evidence-based medicine” to be synonymous with good medicine that’s both contextual and evidence-based.