A Strong Critical Appraisal of the CHAGASICS Trial in patients with Chagas Cardiomyopathy

Dr. Anis Rassi Jr helped design this trial. He ended up resigning authorship due to flaws in trial conduct and reporting. JAMA-Cardiology refused to publish his concerns.

Sensible Medicine aims to give public voice to critical appraisal that is rejected by medical journals. I see two important messages in Dr. Rassi’s appraisal: one is the specific mistakes that this trial could lead to regarding treatment of patients with Chagas cardiomyopathy but the more important message is what it says about scientific conduct and publishing.

The first section is Dr. Rassi’s explanation of interactions with the journal; the second section is a detailed list of criticisms and errors in the paper. You will learn a lot from his list.

First read the paper in JAMA-Cardiology. JMM

---

The CHAGASICS Trial: A Disturbing Failure of Peer Review at a Leading Cardiology Journal

By Anis Rassi Jr, MD, PhD, FAHA, FACP, FACP, Scientific Director, Anis Rassi Hospital, Goiânia, Brazil

I submitted a letter to the editor regarding the CHAGASICS trial, which was recently published in JAMA Cardiology (October 2024). I was advised to formally submit my concerns in a letter to the editor. This letter was rejected by the Editor-in-Chief .

This prompted me to share my detailed analysis of the CHAGASICS trial and the slides from a recent presentation I delivered at the Brazilian Congress of Arrhythmias. (Ed note: Link at the end of post.) My aim is to shed light on the quality of scientific articles being published.

I am curious to see how JAMA Cardiology will address the numerous errors that it allowed to pass through its review process: Will they maintain the status quo, effectively disregarding these identified discrepancies? Will they implement corrections for the multiple errors identified through post-publication peer review while appropriating the findings, or will the authors issue a retraction, attributing these substantial inconsistencies to mere typographical errors – potentially even eliminating existing commentary regarding these discrepancies from the manuscript?

It would be concerning if these corrections were made without acknowledging that these methodological issues were initially identified in a correspondence that the journal declined to publish.

---

October 23, 2024

To the Editorial Board of JAMA Cardiology,

I write to express my profound concerns regarding the publication of the CHAGASICS trial. While the study aimed to evaluate the role of implantable cardioverter-defibrillators (ICDs) versus amiodarone in patients with chronic Chagas cardiomyopathy, there are many critical errors, which undermine its findings.

The sheer number of methodological flaws, calculation mistakes, misinterpretation of the literature, and omission of crucial data is deeply troubling. These errors are unacceptable and raise serious questions about the peer-review process that allowed this study to be published in its current form.

1. Statistical power and underpowered nature of the study

The trial was initially designed to enroll 1,100 patients, providing 90% power to detect a significant difference between the ICD and amiodarone groups. However, with only 323 patients ultimately analyzed, the study was severely underpowered. Based on the original assumptions (30% mortality in the amiodarone group and a 30% RRR in the ICD group), the study achieved only 46% power, far below the planned 90%. When adapting expectations to reflect the reality of the trial, where a 38.6% mortality rate was observed in the amiodarone group, and maintaining the same 30% RRR in the ICD group, the power increased slightly to 60%.

This drastic reduction in power was not properly acknowledged in the manuscript. Such a large decrease in statistical power limits the study's ability to detect any real differences between the groups. With that in mind, the reported outcomes, such as the reduction in heart failure hospitalizations at three years, and even the questionable reduction in total mortality in the first years of follow-up, are likely to be the result of statistical noise.

---

The following deficiencies assume that CHAGASICS was adequately powered—which it was not.

2. Modified Intention-to-Treat Analysis (ITT)

The study deviated from its planned ITT analysis (see the design paper) by employing a "modified ITT" approach, excluding 15 patients from the amiodarone group and 24 from the ICD group who did not receive their assigned interventions.

This modification, particularly problematic given the already undersized study population, exacerbated imbalances in baseline characteristics, including the distribution of Rassi scores and male gender. Such post-randomization exclusions violate the principles of ITT, which exist to preserve the comparability between groups. The authors did not acknowledge this significant deviation from standard practice.

3. Miscalculation of medication usage percentages

Another striking mistake is the miscalculation of medication usage percentages at the end of follow-up. The authors erroneously reported a decrease in the use of heart failure medications, when, in fact, their use increased.

This error occurred because the authors calculated percentages based on the total number of patients (323) rather than the appropriate group sizes (157 for ICD and 166 for amiodarone). For example, the use of beta-blockers at the end of follow-up was reported as 38.1% in the ICD group, but the correct percentage, based on 123 out of 157 patients in the ICD group, is 78.3%. This error distorts the interpretation of medication usage in the study. All other medication percentages reported at the end of follow-up are similarly incorrect.

The statement in the paper, “We acknowledge that despite explicit protocol instructions regarding medical therapy, a low use of angiotensin-converting enzyme inhibitors, angiotensin II receptor blockers, and β-blockers may have influenced HF death and hospitalization end points" (eTable 7 in Supplement 2), is incorrect.

Far from showing a decrease, the medication utilization increased over the study period, which aligns with the expected clinical course of these patients.

How can such a basic mistake in arithmetic go unnoticed by the author, co-authors, reviewers, and editors?

4. Questionable reporting of heart failure hospitalizations

The reporting of heart failure hospitalizations is plagued with contradictions. The authors report a reduction in hospitalizations at three years in the ICD group, while simultaneously reporting an increase in heart failure deaths in the same group over six years. This is clinically nonsensical. No prior studies have shown that ICDs reduce heart failure hospitalizations—on the contrary, ICDs are often associated with increased hospitalizations due to the progression of heart failure, particularly when frequent shocks are involved (Goldenberg I et al). What is more, the increase in heart failure deaths only became apparent after four years, yet the authors inexplicably chose to analyze heart failure hospitalizations at the three-year mark, excluding all the subsequent data.

This selective reporting is not only misleading but also methodologically suspicious. As it is reported, I question whether patients were simply left to die at home from heart failure, which would explain the supposed reduction in hospitalizations.

5. Inconsistencies in adverse event reporting

The reporting of ICD-related adverse events contains errors. In eTable 3, which reports the number of patients receiving appropriate and inappropriate ICD shocks, the sums do not add up. For instance, the total number of shocks reported (81) cannot be accounted for by the breakdown of appropriate (48) and inappropriate (19) shocks. Moreover, the number of patients who received shocks (56) is inconsistent with the sum of patients receiving appropriate and inappropriate shocks (40 + 13). The same discrepancies occurred with the reporting of antitachycardia pacing therapy.

This is not a minor error. It demonstrates a fundamental lack of rigor in reporting basic data. How can yet another obvious arithmetic error pass peer review? This calls into question the overall accuracy of the data presented in the study.

6. No Information about amiodarone use in the ICD group and my resignation from authorship.

There is no mention of whether patients in the ICD group received amiodarone during the study, despite it being commonly co-prescribed in such scenarios. Was amiodarone truly not used, or was this data simply omitted? Transparency on this issue is critical.

If amiodarone was used in the ICD group, crossover becomes a major problem, and the comparison becomes one of amiodarone versus amiodarone + ICD, not just ICD versus amiodarone.

The lack of clarity on this crucial point raises significant doubts about the study’s integrity and conclusions.

As one of the original designers of the CHAGASICS study, I have firsthand knowledge from the study authors that most patients in the ICD group did receive amiodarone as the study progressed. This important fact, which is missing from the publication, needs to be clarified. Without it, the interpretation of the results becomes questionable. It is fundamental to mention that as a lead investigator, I was involved from the study's inception through its completion. The study's primary objective was to determine whether the Rassi score could guide treatment selection between ICD and amiodarone for patients with Chagas cardiomyopathy, with all-cause mortality as the primary endpoint.

Given my concerns regarding the planned presentation and interpretation of the study findings, I withdrew my authorship before the manuscript's submission to the journal. This was my own decision, to ensure that my critique could not be seen as stemming from being excluded from the study. (Editor’s note: Dr Rassi Jr is second author on the design paper.)

It is also important to note that the presentation of results appears to favor the Rassi score as a useful tool for guiding ICD use in primary prevention for Chagas disease. However, the study lacks the statistical power and methodological rigor to support such a conclusion. This is particularly concerning given that validation of the Rassi score for this purpose would represent a major advancement in the field of Chagas disease management and could significantly influence clinical decision-making. Making such claims without adequate supporting evidence could lead to inappropriate clinical applications of the score.

7. Errors in the Discussion Section

Several critical errors in the discussion misrepresent key studies. The authors incorrectly state that the SCD-HeFT trial had a follow-up of 4.5 months, when it was actually ten times that duration—45.5 months. This is a significant mistake, as the trial's long-term outcomes are central to understanding the impact of ICDs on mortality.

Another mistake: the authors confuse two distinct meta-analyses. The phrase, “A systematic review and meta-analysis of 13 retrospective studies including 1041 patients” refers to the meta-analysis conducted by Rassi et al. However, their subsequent statement,

“…suggested that ICD implantation, compared with amiodarone, was not associated with a lower all-cause mortality rate. The reported annual mortality rate was 9.7% for ICD and 9.6% for amiodarone”

pertains to a different study, conducted by Carmo et al. This confusion is ridiculous and inexcusable, and it weakens the discussion's credibility.

8. Illogical inclusion of necessity of a pacemaker as a secondary outcome

Another critical flaw in the study is the inclusion of “pacemaker requirement” as a secondary outcome, a decision that was imposed by the first author. I strongly opposed this inclusion because it made no sense, as all patients in the ICD group already have pacemaker backup as part of their ICD implantation.

No previous study has ever included this as a meaningful endpoint, as it is redundant and irrelevant in this context. Furthermore, if one considers hospitalization for any reason, the ICD group would naturally have higher hospitalization rates because ICD implantation itself requires hospital admission.

9. Lack of event adjudication

The study deviated from its original protocol, which specified that clinical events would be adjudicated by an independent committee. This did not occur, which is particularly concerning given that the three secondary outcomes reported as favorable to ICD (reduction in heart failure hospitalizations, sudden cardiac death, and pacemaker requirement) are subjective.

In an open-label study, independent adjudication is essential to ensure unbiased reporting of such outcomes.

10. Misleading and inappropriate conclusions

The study was underpowered for its primary endpoint of all-cause mortality. While no difference was found between ICD and amiodarone for total mortality, this result is inconclusive due to the high risk of false negative results (Type II error).

While some secondary endpoints suggested benefits of ICD (reduced sudden cardiac death, heart failure hospitalizations, and pacemaker requirement), these positive findings should be interpreted with extreme caution given the study's low statistical power and increased risk of false positive results in secondary analyses (Type I error). A larger, adequately powered study would be needed to definitively determine both the mortality benefit and the reliability of the observed secondary outcomes.

Even setting aside the issue of statistical power, the conclusions presented by the authors are problematic for several reasons:

1. While the ICD did reduce sudden cardiac deaths, this advantage was completely offset by an increase in heart failure-related mortality, resulting in no net effect on cardiovascular or overall mortality.

2. The reported reduction in heart failure hospitalizations is both methodologically flawed and clinically questionable, as it was reported only at the 3-year mark, and not addressed for the full 6-year follow-up.

3. ICD therapy is typically associated with higher hospitalization rates due to complications, device-related issues, and shocks--factors not sufficiently accounted for in the analysis.

11. Clinical relevance and patient considerations

While the study attempts to demonstrate the potential benefit of ICDs, the real-life implications, as derived from the data, are far less promising. A patient receiving prophylactic ICD implantation faces a higher chance of hospitalization, either for the ICD implantation itself or for related complications. They also have a 36% chance of receiving an inappropriate or appropriate distressing shock, and the ICD is unlikely to prevent their overall death, especially from heart failure.

Ironically, avoiding the ICD might result in a more dignified death due to sudden cardiac arrest rather than due to a prolonged decline from heart failure, which the ICD cannot prevent. And, nearly all patients in the ICD group still require amiodarone (my own guess), complicating the narrative of ICD efficacy presented by the authors.

12. The role of lobbying for ICDs in Brazil

In Brazil, there has been a strong lobby advocating for the use of ICDs in the primary prevention of Chagas disease-related complications.

Given the authors' spin (manipulation of language to distort findings by highlighting purported benefits of the experimental treatment (ICDs) despite statistically nonsignificant differences in the primary outcome), this trial risks being selectively cited to support this initiative.

13. My Suggestions for Publishing Studies like CHAGASICS

Underpowered studies like CHAGASICS can still be published, but only if certain rules are followed to ensure credibility and scientific rigor.

• Extreme transparency must be maintained, with clear disclosure of the study's limitations.

• There must be an extensive discussion of statistical power, with the low power of the study highlighted and its implications openly discussed.

• No efficacy claims should be made in underpowered studies.

• Data should be presented as preliminary, with suggestions for how it might be useful in future studies.

• Researchers should discuss the lessons learned from the study and how they can improve the design of future trials.

The CHAGASICS study did not meet these criteria. The lack of transparency, underreporting of important data, and flawed statistical analysis severely challenge the validity of its conclusions.

For these reasons, I strongly believe that the editorial board should reconsider the study’s inclusion in JAMA Cardiology and question whether a study with this many critical issues should remain in the scientific literature.

14. Should this study be retracted?

Given the multitude of these errors, I contest whether this study meets the standards for publication.

In fact, this may well represent one of the most problematic publications ever produced in the field of Chagas disease research.

I respectfully urge the editorial board to re-evaluate this study, and if necessary, consider retracting it to maintain the high scientific standards for advancing medical knowledge.

Editor’s Note: Attached is a hyperlink to the slides from Dr. Rassi’s lecture recently at the Brazilian EP congress.

As always, we appreciate your support. Sensible Medicine exists because of your generosity.