Can AI tell you who needs an ICD?

Five considerations for a New Nature Paper (Obermeyer, et al) claiming to find a better way to place ICDs to avoid SCD

Here is something different for the Study of the Week. This week, Joe Marine and Vinay help me sort out a complex AI paper. JMM

---

Implantable cardiac defibrillators (ICDs) can save lives if a person goes into a shockable rhythm (VT or VF), which would otherwise progress to impaired perfusion and death (termed sudden cardiac death, SCD), but instead, the ICD shocks them back to sinus rhythm.

Some models can also terminate VT by administering rapid ventricular pacing to abort the problematic rhythm — so-called anti-tachycardic pacing (ATP).

Currently, most ICDs are placed when a patient’s left ventricular ejection fraction is less than 30-35% (after 3 months of optimal medical therapy). Other less common indications include hypertrophic cardiomyopathy and inherited arrhythmia syndromes such as long QT.

Key Problem: Despite three decades of placing ICDs, many people still die of SCD without getting an ICD, and most people who get an ICD die without it ever doing anything — it never goes off.

There are then 2 important research questions: How can we identify groups who may benefit from ICDs (who don’t currently receive them), and how can we identify people who currently receive ICDs but don’t need them?

In a new Nature paper, five authors claim to have identified a new ECG risk marker that may do the former.

At the outset, let us be clear: If you claim to have found a new group that may benefit from ICD placement, the definitive way to prove it is to randomize participants to ICD placement or no intervention and show an all-cause mortality benefit.

The authors have not yet done that, though they say they will in an ongoing study. Importantly, the control arm of such a study should not be banned from receiving an ICD, but should be permitted to do so the moment they have an indication (low EF, inducible VT, etc).

In the absence of an RCT, one can try to construct a chain of causality. The authors attempt to do this; here, we outline 5 considerations that warrant further exploration.

1. “Sudden death” occurs for many reasons (massive pulmonary embolism, aortic rupture, brain bleeding, etc). You need to identify people with sudden cardiac death due to an arrhythmia that the ICD can stop. There are only two: rapid VT or VF and asystole (bradycardia). The former is far more common. The authors do this by relying on death certificate data. But being coded as a sudden cardiac death on a death certificate does not mean it was a sudden cardiac death that could have been stopped with an ICD. For instance, the most common cause of VF is ST elevation MI. Here the actual cause of death would is coronary occlusion—the ICD may shock the VF but if the coronary artery is not opened, death occurs. The authors anticipate this objection and performed a complex analysis linking their prediction method to either sudden death or just VT/VF without SCD in a Swedish dataset. It was a nice attempt but even they admit serious limitations to their methods.

2. You have to be careful to train the AI tool on naive EKGs and not on the byproducts of medical interventions aimed at someone whom the doctor is worried about the risk of sudden cardiac death. The Nature paper does not have pure, naive EKGs. As an example, one of the early attempts at AI famously trained the algorithm to recognize abnormal blood smears. Slides from around the office were used. The algorithm learned that dust on the coverslip was a predictor of normal blood smears, as abnormal slides received more views from residents. Similarly, you do not want to train the algorithm for SCD on EKG markers that occur if you already have an ICD in place.

   In the paper, 7% of patients who died of sudden cardiac death (by death certificate) died with an ICD in place. ICDs were present in 5% of people at the time of baseline EKG. It is possible that AI is learning to recognize some artifact of ICD placement — pacing, for instance, or perhaps creation of the ICD pocket alters the EKG in a way we do not fully appreciate — by making providers place the leads in slightly different spots. Pacing is definitely visible on the EKG with nearly every beat. It is not clear that the Nature paper controls for pacing and post-pacing ECG changes — they are not mentioned.

3. You need to know that these rhythms would have been shockable (ATP, a shock, or shock series will restore sinus rhythm). Not all SCD is shockable. Not all VT and VF can be successfully shocked. Cardiac amyloidosis can result in cardiomyopathy, with risk of SCD, but it is also difficult to cardiovert. Dead meat, don’t beat, cardiologists say. The authors of the Nature paper may well identify a group of people prone to VT, but that is not sufficient. They have to have an arrhythmia that can be successfully cardioverted. To overcome this objection, the authors rely on an observational study showing that those at risk of SCD who received ICDs had lower mortality than a theoretically modeled control group — but, like all observational studies, this is at risk of serious residual confounding and guarantee time bias.

4. The people you identify as needing an ICD cannot have a high competing cause of death. Patients need to die of cardiovertible SCD, but not for lots of other reasons. For example, some ICD studies have shown that ICDs don’t improve survival in folks over the age of 75 with low EF — or in patients on dialysis, partly because competing risk is so high.

   The coefficients in the paper show that the authors are finding a new group at much higher risk of competing death than heart failure with reduced ejection fraction. While low EF pts are roughly 3X at risk of all death vs SCD, the new algorithm is roughly 5x the risk of all-cause death. This is a huge problem because competing risk will kill — no pun intended — the potential for your intervention to save lives with the ICD.

Competing risk issues are raised further by looking at a Figure on page 5 of the supplement:

The strikingly high total mortality in the “ECG high-risk group” (25% at 3 months) suggests that the high-risk group contained many subjects who had their ECG taken in the context of a serious/life-threatening illness or imminent death. Curves that separate instantly in retrospective studies often indicate that you are looking at two entirely different groups. No one needs an AI model to tell you that a critically ill patient is more likely to die in the next year than a healthy clinic patient getting a checkup. And no cardiologist would be keen on placing a ten-thousand-dollar device in a person with a high competing risk of death. Clinical context matters.

5. People identified as dying of SCD with the new method can’t later develop a standard indication for an ICD before SCD. In other words, if they were going to die on Friday from VT, and AI found they needed a device on Monday, but the doctor was going to find they needed a device on Tuesday, you didn’t help them. They were still going to get the device before Friday. Just like the control arm of the validation study is not — no ICD, not ever — but ICD when you otherwise develop an indication. It is not clear whether the Nature paper considers this. The algorithm is only of value if it finds a person who will have shockable SCD before they have another reason to have an ICD placed.

Ultimately, the concerns in our analysis are neither damning nor exculpatory. The only way to settle the debate is a prospective randomized trial. Take patients who do not have an indication for ICD, screen them with the new AI tool, and randomize them to ICD now vs ICD, if and only if it is needed in the course of usual care. The endpoint must be overall survival. If we were betting, the high risk of competing death, the possibility that some artifact is being learned, and the chance that patients at high risk may later qualify for an ICD are all reasons to be skeptical that this approach will change practice.

We agree with the authors who write, “many promising predictors in the past have failed to identify patients who benefit from defibrillators.” Failed predictors include microvolt T-wave alternans, baroreflex sensitivity, heart rate variability, and heart rate turbulence. Each of these had a reasonable physiological basis and abundant preclinical evidence suggesting that they would be clinically useful risk predictors, but, at some point in development, failed to show any value in predicting clinical benefit from an ICD.

Modern medicine has made the ICD selection method even harder: it is now clear that sudden death mortality in common conditions like heart failure is declining. A recent paper from Switzerland has also confirmed substantial decreases in the incidence of appropriate ICD therapies over the past two decades. A declining arrhythmia incidence reduces the proportion of people who can benefit from an ICD. In fact, a new trial in Europe called PROFID is retesting the positive results of the 25-year-old MADIT II trial.

There are a number of other questions that should be addressed before even considering exposing patients to the risk of such an ICD trial:

1. Is the new ECG biomarker stable over time in a given patient?

2. Is it reproducible from ECG to ECG from one day to the next?

3. Is it affected by metabolic, acute, illness, position, or other changes?

4. Does it predict risk independently of other known variables such as other cardiac diagnoses, age, heart failure status, other comorbidities, etc?

5. Does it predict risk in stable ambulatory patients and in subjects with no other clinical indication for an ECG?

6. Is the model accurate in people with bundle branch block, paced QRS or widened QRS for other reasons?

7. Is the model affected by lead placement or ECG system used to make the recording?

Two final points:

1. We note that a Washington Post reporter has already posted that the paper is “amazing.” While we would hope a reporter at a major newspaper would have enough in-house expertise to sort out limitations of a medical paper, his enthusiasm is not that unusual—even amongst doctors.

2. While we are eager for the results of a randomized trial, we note that many machine learning or AI-suggested algorithms still lack such verification, despite many years having elapsed since the claim of “discovery.”

In short, the new proposed ECG biomarker approach generates a lot of buzz, but there are open conceptual issues with the paper. Newspapers and physicians accepting the results of science that they either did not or cannot vet is problematic.

One vexing problem with ICDs is that the vast majority of patients implanted with an ICD never use it. They incur all the harms (inappropriate shocks and infection), but none of the benefits. We wonder if an ECG model could predict patients unlikely to benefit from the ICD. Such an algorithm could be used to design an RCT at the time of ICD generator change—the most risky time for patients with the device.