Is the FDA too lax?

Here is a list of 5 devices that all reached FDA approval with extremely dubious evidence. I expanded a list from Dr. Sanjay Kaul.

Vinay started with this:

Professor Frank Harrell responded

As has often been the case, hyperbole limits the value of some of your opinions Vinay. To say that the FDA is a rubber stamp is ludicrous. Sure there are shortcomings as with any organization but sponsors know they can't get approval without a heck of a lot of work.

I don’t know. The laxity of the FDA has lately surprised me—especially for devices. Let’s detail a list of 5 examples that support Vinay’s POV.

---

Dr. Sanjay Kaul is one of the smartest trial appraisers. Here is a thread on a few of the FDA device approvals. (I will address Sam’s comment at the end.) I will use his list, expanding on each of his short statements.

I Left Atrial Appendage Occlusion

Dr Kaul starts with Watchman, which is one of the left atrial appendage closure devices. The idea is to prevent stroke and reduce bleeding by plugging the appendage. LAAO has evolved into a common and heavily marketed procedure.

Here is Kaul’s comment:

I can add to this that the PROTECT trial, which sits in the literature as a trial that met its noninferiority endpoint (Watchman NI to warfarin), did not pass FDA muster due to internal validity concerns. FDA mandated PREVAIL. But, get this: PREVAIL did not find noninferiority in its first co-primary endpoint of stroke, systemic embolism or cardiovascular death. Despite that, FDA approved the device for a group of patients who were not included in those two trials.

For the PREVAIL trial, the first composite coprimary endpoint of stroke, systemic embolism (SE), or cardiovascular/unexplained death did not achieve noninferiority (posterior probability for noninferiority = 88.4%)

---

II MitraClip (Transcutaneous Edge-to-Edge repair)

Here is Dr. Kaul again:

Re the EVEREST 2 trial:

The primary composite end point for efficacy was freedom from death, from surgery for mitral-valve dysfunction, and from grade 3+ or 4+ mitral regurgitation at 12 months. (Editor’s note: this is an endpoint where the higher number is better.)

The results: At 12 months, the rates of the primary end point for efficacy were 55% in the percutaneous-repair group and 73% in the surgery group (P=0.007)

Secondary MR: What about clipping the mitral valve for primary MR? This occurs in patients with heart failure.

FDA approved this indication despite two trials with opposite results. The COAPT trial (industry-sponsored) was highly positive but the MITRA-FR (government funded) trial was totally neutral.

Proponents say COAPT chose patients who were most likely to benefit. Ok. Maybe. I am a cardiologist and it’s hard for me to look at the Table 1 in these trials and think the patients are that much different.

---

III Renal Denervation for high blood pressure.

This was a recent FDA approval. There are two companies. One makes an ultrasound device, the other a radiofrequency device. The idea is to ablate nerves in the kidney to lower blood pressure.

Here is Kaul again:

My translation is that the Recor device (ultrasound) approval was very weak. But at least the FDA advisory committee voted favorably. Here is the most recent meta-analysis.

While unadjusted between-group difference in daytime ambulatory SBP was similar at 6 months, the baseline and medication-adjusted between-group difference at 6 months was -3.0 mmHg (95%CI: -5.7,-0.2; p=0.033) in favor of uRDN+AHT

You did not read that incorrectly. It’s 3. As in 3 mmHg. And all of this data is at 6 months. No RDN device has (placebo-controlled) data beyond 6 months.

Kaul also notes that the newly approved radiofrequency device had an actual null trial. And the advisory committee voted unfavorably. Here were the results of the SPYRAL HTN ON meds trial.

In the primary efficacy analysis, there was no significant difference observed across treatment groups for 24-hour ambulatory systolic BP at 6 months (Bayesian treatment difference: −0.03 mm Hg [95% Bayesian credible interval: −2.82 to 2.77 mm Hg];

The RF device did have one trial, called SPYRAL HTN OFF meds that did show a modest (≈ 4 mmHg) reduction in BP at 3 months.

Here were headlines this week. They actually quoted a doctor in the story.

Renal denervation may turn out to be the lowest value, least proven procedure ever to come to market. Millions of people have high blood pressure. It could be a cost-efficacy disaster.

---

IV The Absorbable Stent

Dr. Kaul writes:

This may have been one of the worst FDA approvals in recent years. The idea behind the absorbable stent is that after it absorbs, there is no more metal left to potentially clot off. It was a brilliant idea.

Here were comments from the FDA advisory committee. Ralph Brindis, MD, "I do believe this is a novel breakthrough technology for patients undergoing PCI.”

Hardly a year passed and Abbott pulled the stent off the market because doctors stopped using it. Why? Longer term data (3 years) found higher rates of poor outcomes.

---

V Invasive Monitoring for Heart Failure

Dr. Kaul mentions a device called CardioMEMS, which is a paperclip-shaped device that doctors place in the pulmonary artery. The grand ideas is to transmit cardiac data (pressures, etc) to doctors so that they can better manage patients.

FDA originally approved the device for use in a very narrow category based on one trial. The real flub came later when FDA expanded its use in a much broader group of patients based on a trial called GUIDE HF.

The trial compared patients managed with the device vs those managed with normal means. The primary endpoint was a composite of death and total heart failure events (hospitalization for heart failure or urgent heart failure visit).

Here were the results:

There were 253 primary endpoint events (0·563 per patient-year) among 497 patients in the hemodynamic management group (treatment group) and 289 (0·640 per patient-year) in 503 patients in the control group (hazard ratio [HR] 0·88, 95% CI 0·74–1·05; p=0·16).

You can read as easily as me. It did not result in a statistically significant difference. Not even really that close. Did they accept defeat? No. They did not.

Instead, they presented a COVID-19 sensitivity analysis wherein they subdivided groups based on whether they were enrolled in the trial before or during COVID-19. (Think smaller groups = more noise).

Here they found their positive p-value, barely.

In the pre-COVID-19 impact analysis, there were 177 primary events (0·553 per patient-year) in the intervention group and 224 events (0·682 per patient-year) in the control group (HR 0·81, 95% CI 0·66–1·00; p=0·049)

Boom. P-value now 0.049.

It turns out the device performed less well during the pandemic. That’s weird, isn’t it? Weird because you’d expect a device that transmits data wirelessly and minimizes outpatient visits would overperform during the pandemic.

I’ve written on this story:

---

Conclusion

Let’s conclude by addressing Sam’s comment:

I disagree strongly. I hope to have persuaded you with these five recent examples that FDA can be very lax and approve devices with extremely dubious supporting evidence. Except for the absorbable stent, all others are (or will be) costly devices. (Why doctors embrace these devices is a topic for another post or book).

I believe (and I think Drs Prasad and Kaul agree) that medical innovation can be amazing. This is the best time to be a patient.

Yet, we should require our wealthy partners in industry to fund studies that define a group of patients that will benefit. And we should then also require our regulators to protect the public by forcing new innovations to pass a proper bar before being allowed on the market. Ideally, we’d expect doctors to be better at critical appraisal.

Real innovations will pass that test. Dubious ones will not. JMM