Understanding Benefits of a Treatment

The Study of the Week explores multiple ways to express benefits of an intervention. It turns out there is some degree of nuance.

Here is the headline:

Study finds <device> reduces the rate of a <really bad outcome> by 40%.

The headline is true. And the 40% reduction reached statistical significance.

This sounds like a winner, right?

The lesson of this column centers on communication of benefits. The 40% reduction is a relative risk reduction. This does not take into account the absolute event rates of the outcome. Another way to express benefits is absolute risk reduction.

Let me show you an example.

A few years ago, Medtronic funded a 7000-patient study to test whether an antibiotic-impregnated absorbable envelope would reduce the rate of device infection. They called the study the WRAP-IT trial.

Many readers know that I have been critical of industry, but I laud the company for funding such a trial. Pacemaker or ICD infection is a disaster. The bacteria can track down on the leads, which are in the heart. People can die from this kind of infection.

One of the scariest times for device infection comes when patients have their generators changed due to low battery. Here the leads have been in the body for years, and if they get infected, removal often requires a risky surgical procedure.

If this simple pouch prevented infection, it would be a serious medical advance.

Instead of relying on flawed observational studies, or mere bio-plausibility, Medtronic funded a trial, led by a Professor Khaldoun Tarakji, from Cleveland Clinic.

I like the WRAP-IT trial because it was elegant. About 3500 patients got the envelope and about 3500 did not. The primary outcome was serious device infection. That’s it. No composite or surrogate endpoints. No fancy statistical plan.

---

WRAP-IT delivered positive results. Positive, yes, but also interesting.

The main results:

There were 25 infections in the envelope group vs 42 in the control arm. That is an incidence rate of 0.7 percent vs 1.2 percent. (Astute readers, I hope, have noticed that device infection is a quite rare event. In the control arm, ≈ 99% of patients do NOT get an infection.

The difference in infections can be expressed in two ways.

The relative risk reduction is a proportion. It’s the difference between the control and experimental arms divided by the control arm. It calculates to about 40%. The hazard ratio approximates that.

Keeping it simple, we could say that patients in the envelope arm were 40% less likely than those in the control arm to get an infection.

But. But.

This does not incorporate the low event rates. If you are a patient, and you had to pay for the envelope yourself, you’d want to know the absolute risk reduction.

We calculate that by subtracting the two event rates. 1.2 – 0.7 is 0.5%. That is a pretty small difference. Much less than the headline of 40% reduction.

Some in medicine like to express the absolute risk decrease as a number needed to treat or NNT. How many patients do we have to treat to reduce one bad outcome.

We find that by taking the inverse of 0.5%. Here 1/0.005 is 200.

In other words, we have to treat 200 patients with the envelope to prevent one infection. 199 patients get the same result—no infection—whether or not we use the envelope.

Here comes the kicker.

The envelope costs approximately $1000. The figure shows that it costs about $200,000 with this envelope to prevent one infection.

The specific take-home is that this device reduces the rate of infection by 40%, but because infection rates are low, the absolute risk reduction is small. Since it is expensive, and most patients get the same result—no infection—it is a pricy intervention.

---

I don’t know the right answer about using the envelope. That is not the point of the post. Some have argued that device infections are so terrible and expensive, that the cost of the envelope is justified. Maybe.

The main point of this post is that when you read a medical study be on the alert for both ways to communicate efficacy.

You can have a substantial relative risk reduction that leads to a very small absolute risk reduction.