An Easy Way to Spot Bias in Observational Studies

We are excited to share this guest post from Dr. Mohammed Ruzieh from the University of Florida. He dissects a flawed study of stroke prevention in atrial fibrillation

Dr. Mohammed Ruzieh is a cardiologist at the University of Florida. His special interests include syncope, arrhythmias, and sports cardiology. He has conducted numerous outcome studies and has published over a hundred manuscripts, abstracts, and book chapters to date.

I met Ruzieh, as we call him, because he trained with Andrew Foy at Penn State. He’s one of those young faculty members who give me hope about the future of Medicine. Why? Because he excels in what Emerson urged from academics in his American Scholar lecture in 1837: he thinks clearly and not influenced by tradition or history.

Since people like Ruzieh and Foy are teaching young people, we should all feel a bit more optimistic. JMM

---

By Mohammed Ruzieh

Atrial fibrillation is the most common cardiac rhythm disturbance. It’s projected to affect more than 12 million Americans by 2030. The most important threat from AF is an elevated risk of stroke. The absolute increase in stroke risk occurs on a continuum, based on the presences of factors such as age, high blood pressure, diabetes, heart failure and a history of previous stroke.

Doctors debate the exact reasons for this elevated risk, but a likely component is simply stasis of the blood in the fibrillating atria. The atria are supposed to squeeze blood into the ventricles; during AF, the blood passively flows through the atria to the ventricle. When blood sits still, clots can form, which can then travel north to the brain and cause stroke.

Both atria have these appendages, pouches with many crevasses that can act as a place where clots form. A famous imaging and autopsy study found that the left atrial appendage was the most common place that clots form in the heart.

Doctors then had an idea: what if we could close off that appendage? Industry then did what industry does: they fired up the bioengineers and made a plug that doctors could put in the appendage. Look how simple it is:

Once in place, the idea held that the sealed appendage would no longer be a source of clots and stroke. Another bonus would be that patients would no longer need drugs that block coagulation—so-called anticoagulants.

(Editor’s note: we don’t call these drugs blood thinners because they don’t actually thin the blood.)

I will abbreviate this procedure pLAAO, or percutaneous left atrial appendage occlusion.

Is pLAAO a good idea?

It sure sounds good. But there are at least four caveats.

First, the left atrial appendage is not the only source of stroke in patients with AF. Second, patients have to take aspirin indefinitely after pLAAO. In a famous trial, called AVERROES, which compared aspirin to the common anticoagulant, apixaban, the authors found similar bleeding risks. This finding dispelled the notion that aspirin has (substantially) lower bleeding risk compared to apixaban.

Third, inserting a device confers a risk for major procedural complications. Finally, any foreign body in the heart can increase the risk that clots form on the device.

What do we know so far?

Not much. That’s because the trials of pLAAO have limitations. The first of which is their small sample sizes, which result in uncertainty regarding signal vs noise. Another problem was that the trials that led to approval of the device tested pLAAO vs warfarin. The device did not prove better than warfarin, and we now know that warfarin is inferior to the newer anticoagulants. Perhaps the main limitation, though, is that most patients who currently receive the device would have been excluded from these trials because they are poor candidates to take anticoagulants over the long-term.

Now to the Study:

This was a nonrandomized comparison study published in the journal JACC-Interventions. The goal was to assess outcomes in patients with AF who had pLAAO using a device called Amulet vs those who had therapy with standard anticoagulants.

The authors used data from two sources. One was the Amulet registry from 17 countries, and the other was a group of patients in two Danish registries who had AF and were treated with the newer anticoagulants.

Their outcomes of interest were bleeding, stroke, all-cause mortality, and cardiovascular mortality.

The results were impressive: pLAAO significantly reduced bleeding, all-cause mortality, and cardiovascular mortality, although the one outcome you’d expect the device to excel in—ischemic stroke—was the same in each group.

How confident are you in these results?

Bias in observational studies is not always easy to catch. This is because authors often do statistical adjustments to try and match the non-random groups.

But in this case, the Kaplan-Meier curves hold an important clue about why these results are biased.

Red line is pLAAO and blue line is anticoagulation

This image shows that the curves start to separate at day zero. This means that pLAAO started saving lives from the day of the implant.

Which is impossible given that AF is a chronic condition. Furthermore, even if pLAAO reduced mortality, it would do so by reducing strokes or bleeding.

In this study, stroke rate did not change with pLAAO and it took 6 months for any signal of lower rates of major bleeding.  

How could mortality be reduced this early?

The only explanation is selection bias. You might wonder how this could be, given that the authors used propensity-matching to make the comparison groups similar?

The problem: even the most advanced matching processes rely on data from a spreadsheet. But a clinician—not randomization—made the choice to use or not use an intervention.

Some of the factors influencing that decision, such as age, high blood pressure, and diabetes can be recorded in a spreadsheet. But other factors cannot. For example, if a patient is frail, or has dementia, or difficulty taking medications, they are less likely to receive an anticoagulant and their mortality is higher not because they did not receive the device but because of their frailty or dementia.

We call this confounding, and it is a nearly existential problem when trying to make inferences about causation from nonrandom samples.

(Editor’s note: we say this time and time again on Sensible Medicine: randomization balances both known and unknown factors. That is why it’s so important.)

Here is an example of a Kaplan-Meier curve that comports with a plausible effect of a drug. Carvedilol is a beta-blocker that reduces the workload of a failing heart.

This KM curve comes from one of the seminal randomized trials and you can see that separation took many months to years. This is exactly what you’d expect because it takes time for the drug to exert its effect.

Summary

The key point is to examine when the Kaplan-Meier curves separate.

If you see that an intervention of chronic condition reduces mortality very early in observational studies then the answer is likely bias wherein healthier patients get one intervention.

---

Editor’s Note: Here at Sensible Medicine, we welcome well-argued columns of less than 1000 words, even if we disagree with the argument.