The Value of Reanalysis of a Clinical Trial

The study of the week explores a provocative paper in 2014 wherein the authors studied reanalyses of original trials. The findings are sobering.

When a clinical trial is published, people are relieved. The researchers did the work, analyzed the data, published the report and surely feel a sense of completion. Clinicians are relieved because we have more evidence to use in patient care.

The question is how certain we should be about the trial’s result.

In April, I covered the work of Dena Zeraatker, from McMaster University, who elegantly showed that there were thousands of ways to analyze data from observational studies. And, provocatively, some choices led to “positive” results, other choices to “negative” results—of the same data.

I have heard other argue that those were observational studies; there isn’t that much analytic flexibility in clinical trials. I am not sure this is correct.

Let me show you a study that I use in my talks on evidence-based medicine.

Shanil Ebrahim, PhD, working w many co-authors, looked at reanalyses of RCTs. They searched the literature for all published reanalyses. They ended up with 36 published studies.

Most of these reanalyses were performed by authors of the original trials, while 5 of the 36 were done by independent authors.

The reanalyses used different methods from the original trial. These included things like changing the statistical method, or redefining of an outcome, or how to handle missing data, or using intention-to-treat vs on-treatment or a slew of other choices. Table 3 in the JAMA paper outlines all the specific different analytic choices.  

Results:

Thirteen reanalyses (35%) led to interpretations different from that of the original article, 3 (8%) showing that different patients should be treated; 1 (3%), that fewer patients should be treated; and 9 (24%), that more patients should be treated.

Comments:

This paper opened my eyes to the degree of uncertainty in clinical trials. One optimistic way to interpret this study would be to say that two-thirds of the reanalyses upheld the original finding.

I am less optimistic. Instead I focus on the fact that a) there were so few reanalyses published in the literature, and b) the fact that a third of reanalyses could lead to entirely different treatment recommendations is sobering.

You can’t argue that these were new authors playing with data after the fact to get results they wanted—as most of the reanalyses were done by the original authors.

Table 3 in the paper lists the choices made in the re-analyses and most seem quite reasonable.

Drs Harlan Krumholz (now editor-in-chief of JACC) and Eric Petersen (Duke) argued in the editorial that we should have better access to source data when trials are published. I could not agree more.

This paper and editorial were published ten years ago. Over the past decade, clinical trials—at least in cardiology—have struggled to show big gains.

Likely because we have already made many gains in the past, and it’s hard to further reduce residual risk. The result of this plateau in progress has been more complicated trials. Trials that use composite endpoints or complicated analyses such as win ratio. And often the clinical significance or statistical robustness is less than ideal.

To me, the current situation argues even more strongly for open data and confirmation of results via reanalyses.

If a trial’s conclusion could be altered simply by making a change in statistical choices or minor changes in an endpoint, then clinician’s may make different judgements about the value of the intervention.

I realize there are challenges to open data. Yet the positive value of transparency seems to outweigh the negatives.

My view: If you have a great result, it gets even greater when it holds up to reanalysis.

---

I opened Sensible Medicine on a device that wasn’t signed in. It asked me to join 83,000 subscribers. Then I looked at the statistics. SM now earns 1.35 million pageviews per month. This fills me with gratitude. Thank you for supporting our efforts to bring some sense to medical information. Our subscriber model allows us to avoid industry influence. Please consider subscribing. And spreading the word about our project. JMM