Judging Medical Science

The Study of the Week explores a provocative trial presented at the recent American Heart Association meeting. The message goes to the norms of reviewing science

This is a story about herbs, insects, and judging science.

First the bit about science. It starts with a personal story about the first time I presented results of a study.

About 10 years into private practice, I got an itch to answer a question. It was a crazy idea, because until that time, I had been allergic to research. I had eked through an academic program learning only how to be a doctor.

We reviewed charts and gathered data in our hospital to study an association between a co-variate and a procedural outcome. We put our data into an Excel spreadsheet. We wrote up a paper and it was accepted as an abstract at a big medical meeting.

But here was my great surprise:

Nowhere on the application was a place to upload our source data. They trusted us. I have now come to learn that this is the norm everywhere in medical science.

I thought to myself--studies with huge datasheets--change practice. They influence guidelines and create a standard of care.

But no one checks the data. Everyone trusts the scientists.

You don’t believe me. I understand.

So I made some images of four recent trials published in the New England Journal of Medicine, which requires a data-sharing statement from the authors.

I could have made a hundred of these if I wanted to push on over the last few years.

Now let’s move to the herbs and insects.

At the recent American Heart Association meeting, we learned the results of the CTS-AMI trial, which studied a (super novel) medication to improve outcomes after a heart attack.

The authors presented results during a featured presentation, called a late-breaking trial session. The AHA meeting is one of the world’s most prestigious medical meetings.

Patients with heart attack due to a blocked blood vessel, especially those who present late, still face a high risk of death. This may be due to the poorly understood notion of reperfusion injury, wherein restoring blood flow to an injured heart muscle wreaks havoc. No drugs have yet proven effective in the space. It’s a major problem because lots of people still present late after the onset of a blocked coronary artery.

Let me tell you the results of CTS-AMI trial before I tell you the details.

• Relative to a placebo tablet, the active medication reduced the incidence of a major adverse cardiac outcomes, such as death due to cardiac causes, another heart attack or stroke, by 36%--a reduction that easily met the threshold for statistical significance.

Let’s also say that the CTS-AMI trial enrolled nearly 3800 patients from 124 hospitals. This was no chart review; it was a major randomized trial presented in a featured presentation at AHA.

---

What was this amazing medication? Surely it is in line for emergency use authorization at FDA.

It is a Chinese medicine compound made of insects and herbs called Tongxinluo. And the CTS-AMI was conducted entirely in China.

Tongxinluo has traditionally been thought to have clinical benefits for patients with heart disease and those who’ve had heart surgery. There is even a Cochrane review that found some evidence that it helps patients with chest pain—though all the included trials had limitations of their methods.

At the AHA meeting, the trial discussant, an American academic, expressed surprise.

He wanted more details about the analysis of the data, trial conduct, and the potential ways this strange-sounding capsule worked. He also wanted to see the trial repeated in a Western population.

These were not unusual claims. It’s always helpful to have a full paper along with the presentation, however, most studies presented at meetings do not have a simultaneous publication. These come later.

---

Here’s the part that makes the point of this column. Many of the experts at AHA wanted something else.

They wanted to see the actual angiograms (pictures of the arteries), the ultrasound images and the biomarkers. In other words: they wanted to see the source data.

This sudden burst of skepticism hit a nerve. Wait a moment. Skepticism may be a core tenet of science, but it is, as I’ve shown you, not the norm. Many if not most major trials do not share their source data.

Western doctors do not understand traditional Chinese medicine. We don’t understand how this capsule filled with herbs and insects could reduce cardiac events.

Yet, regarding unknown mechanisms, we don’t understand how catheter ablation of atrial fibrillation works, but AF ablation remains a major economic boon to hospitals and doctors. I’ve heard no calls to review source data of big trials.

I do not know if CTS-AMI would replicate. It’s unlikely. Large effect sizes in initial trials rarely ever hold up.

But if the CTS-AMI trial did replicate, gosh, we will have learned a lot. Not only about a potential new cardiac medicine, but also a new way to judge science.

The sudden desire to review source data got me thinking that perhaps medical science would engender more trust and move faster if all scientists always shared their source data.

I remained surprised that our current norm is that no one but the experimenters sees the source data.

What about you?