Dr. Anis Rassi Jr helped design this trial. He ended up resigning authorship due to flaws in trial conduct and reporting. JAMA-Cardiology refused to publish his concerns.
I feel compelled to respond to Dr. Anis Rassi Jr's thoughtful critique of the CHAGASICS study, published in Sensible Medicine. As a co-author of the original study, I must acknowledge that his criticisms are mostly valid and raise serious concerns specially about the study's conclusions.
The most fundamental issue, which I wholly agree with, is the severely underpowered nature of the study. With only 323 patients analyzed instead of the planned 1,100 (or 256 deaths), the study achieved merely 60% power (assuming a 38.6% mortality rate in the amiodarone group and a 30% relative risk reduction in the ICD group). This dramatic reduction in statistical power renders the study inconclusive for both primary and secondary outcomes. The current conclusion stating that "ICD did not reduce the risk of all-cause mortality" is inappropriate given this limitation. Instead, we must acknowledge that the study's severe lack of power precludes any definitive conclusions, even regarding the apparent lack of reduction in overall mortality.
I also concur with concerns about the modified intention-to-treat analysis, which violated the original study design by excluding 39 patients who didn't receive their assigned interventions. This deviation from standard practice in my view remains a controversial issue, despite being supported by the Reference 29, and possibly also by its previous paper by Dr Anna Dossing and her colleagues (Dossing A, Tarp S, Furst DE, Gluud C, Beyene J, Hansen BB, Bliddal H, Christensen R. BMJ Open. 2014 Sep 26;4(9):e005297. doi: 10.1136/bmjopen-2014-005297. Interpreting trial results following use of different intention-to-treat approaches for preventing attrition bias: a meta-epidemiological study protocol.). At any rate, I would favor the insertion of this aspect on the topic of Limitations of the paper. Because, in the case of CHAGASICS, these post-randomization exclusions violate the fundamental principles of ITT that was originally devised for the analyses. Since the number of exclusions affected differentially the two groups, it might have contributed to unbalance the comparability between them. I guess the calculations were done following both strategies. Therefore, if, as postulated by a conclusion based on Dr Dossing’s paper, results were similar with the two methods, why should the Statistician have not adopted the most orthodox ITT method?
Regarding medication usage, it is pristine that there were significant errors in the percentage calculations. The paper requires correction to show the actual percentages, and the incorrect statement about low medication use influencing heart failure outcomes must be withdrawn. Furthermore, I agree that the reporting of heart failure hospitalizations at three years, while excluding subsequent data, was methodologically flawed. This selective reporting, combined with the contradictory increase in heart failure deaths, would warrant the exclusion of any conclusions about this secondary endpoint.
Another crucial point concerns the use of amiodarone in the ICD group. I believe the study database would confirm that most patients in both arms were indeed receiving amiodarone. This means that, contrary to the original conception, CHAGASICS effectively compared amiodarone alone versus amiodarone plus ICD, a distinction that significantly impacts the interpretation of results.
The inclusion of pacemaker requirement as a secondary outcome was, as Dr. Rassi Jr notes, inappropriate given that ICDs inherently include pacemaker functionality. This endpoint should be eliminated from the analysis. Additionally, while the lack of event adjudication was acknowledged in the paper, I agree that this limitation should be emphasized particularly regarding the secondary endpoints of hospitalization, sudden death, and pacemaker requirement, which are more susceptible to interpretation bias in open trials.
Regarding statistical interpretation, I share Dr. Rassi Jr's concern that both Type I and Type II errors may have occurred. The study's lack of power means that real effects of ICD might have gone undetected, while apparent differences in secondary outcomes could be unreliable due to statistical noise.
Based on these significant limitations, I must agree that no claims for regular ICD implantation in primary prevention for patients with chronic Chagas cardiomyopathy can be validated by this study. This is, in my view, the most detrimental potential consequence of the paper, as it is published, consists in that these inconclusive findings might be used to support ongoing lobbying efforts for expanded ICD use in Brazil.
Various technical errors in the paper, including incorrect citations and data presentation, would require correction through an errata or corrigenda in JAMA Cardiology.
In conclusion, I appreciate Dr. Rassi Jr bringing these important issues to light. The scientific community deserves transparency about the limitations of our research, and I believe acknowledging these shortcomings serves the greater goal of advancing medical knowledge through rigorous scientific inquiry.
The scientific courage demonstrated by Prof. Marin-Neto in his thorough response to the methodological concerns raised about the CHAGASICS trial deserves our highest recognition. In today's academic environment, where acknowledging study limitations often carries professional risks, his commitment to scientific truth stands as a beacon of academic integrity.
His meticulous critique highlights key issues in the study, including severe underpowering, a compromised intention-to-treat analysis, unjustified emphasis on misleading secondary endpoints, concealment of crucial data, ridiculous confusion and incorrect citation related to well-known literary texts, the egregious spinning of results in the article's conclusions, and unacceptable mathematical errors - such as miscalculations in additions and percentages - that should NEVER pass peer review, especially in a journal like JAMA Cardiology. This level of scholarly rigor is essential, particularly given the study’s potential real-world implications.
While the manuscript carefully avoids explicit recommendations for ICD implantation, several authors of the CHAGASICS trial are already citing this ill-fated study in scientific presentations to support broader ICD use in Chagas disease. This concerning development highlights how fundamentally spurious research can be weaponized to influence clinical practice, particularly troubling given the resource constraints in regions where Chagas disease is endemic.
The conspicuous silence from other co-authors in the face of substantive criticism raises disturbing questions about scientific accountability. Equally troubling is the editorial process from JAMA Cardiology, which not only allowed publication of a study with such profound deficiencies but subsequently rejected a detailed letter to the editor documenting these important shortcomings. This rejection of legitimate scientific discourse through established academic channels forced this crucial debate into alternative forums.
How does JAMA Cardiology propose to address these issues now? A simple erratum seems woefully inadequate given the magnitude of the methodological errors identified. The journal's peer review process requires serious examination when such compromised research receives the imprimatur of publication while detailed, substantive criticism is summarily dismissed.
The response of Prof. Marin-Neto demonstrates that acknowledging limitations enhances rather than diminishes scientific credibility. His commitment to scientific integrity serves our field's highest purpose - advancing medical knowledge through honest, rigorous inquiry to improve patient care. In the context of a neglected tropical disease affecting predominantly underprivileged populations, such intellectual honesty becomes not just an academic virtue but principally a moral imperative.
It is a perfect teaching opportunity to learn how NOT to make a clinical trial, and Dr. Anis Rassi mastered this class.
Clinical research nudges medical decisions, implying accountability at the highest level. The interventions evaluated by this “trial” is the implantation of a permanent medical device with intended and unintended consequences. It adds to the complexity of researching a socially vulnerable population like Chagas’ patients. This study has implications.
The evidence of flaws is so overwhelming that a collective discussion should seek a justification of how such a report is published, especially in a major medical journal. Such accountability must extend to the institutions that nest studies.
In the best scenario, the flaws were caused by a lack of methodological skills. Being a meaningful physician does not depend on doing research. One should know before practicing it. The scientific art is complex and full of nuances. Physicians who are not sufficiently equipped with skills should avoid the scientific enterprise. Clinical research should be a professional rather than an amateur activity of physicians promoting themselves as publishers in major journals.
Although the authors intended to have done a randomized clinical trial, this label misleads the public to its scientific quality. From a causal inference standpoint, this study has evidence inferior to that of a good observation study. Given all the limitations, it is more compatible with a descriptive study. If framed this way, some of the information could be useful, but not about comparative efficacy.
Dr Rassi’s invitation to an open discussion makes sense. Not a debate restricted to opinion leaders. Instead, the community of physicians who consume science should speak truth to power and advocate for better science and science for the right reasons.
This is a stark example of the extent to which researchers will go to in order to follow the famous adage; “it’s not the money, it’s the MONEY!” As discussed on many occasions, big Pharma, and in this case the equipment manufacturers, and even those who use their wears will do virtually anything to fatten their wallet.
ICD’s are big ticket items and a few more inappropriate insertions based on bad science more than pays for those involved in placing this instrumentation. Medicine is one of the few businesses were one can bend one’s ethics under the guise of science and purported benefit to the health of the population and personally benefit greatly. It saddens me, but I also relish and commend Sensible Medicine’s (subscribed) attempts to bring back honesty and integrity to our profession.
The CHAGASICS trial has significant implications for Chagas disease management - a neglected tropical disease affecting vulnerable populations. The methodological flaws and interpretation errors in the published study risk leading to inappropriate clinical recommendations and resource allocation.
Following publication, I submitted a detailed critique identifying specific methodological and analytical errors to JAMA Cardiology. Despite following the journal's formal process for letter submission, my concerns were rejected without substantive explanation.
This lack of engagement has compelled me to seek alternative platforms to address these discrepancies. As a final opportunity, I invite both the JAMA Editorial Board and the CHAGASICS trial authors to engage in a meaningful discussion of these concerns, particularly given recent presentations interpreting the trial as practice-changing.
JAMA Cardiology's handling of legitimate scientific critique sets a precedent for academic publishing standards. The scientific community deserves transparent discussion of methodological concerns, especially when findings may influence clinical guidelines for vulnerable populations.
Dr. Ionnidis, one of the most respected researchers in the world, told us that 95% of journal articles are junk science. This article is just one example illustrating why "peer reviewed" is no longer a label assuring accuracy in research reports. It reflects badly on JAMA-Cardiology that this study passed review. It is unacceptable that they would refuse to publish the detailed critique.
This paper and Rassi’s assessment of errors would make a great teaching case for medical students and residents (and perhaps for accomplished so-called experts) on all the major points of scientific discovery and rigor.
It is imperative that the publication of comments and questions by peers should not be censored. The scientific community is much worse for the wear with this overt censorship of questioning and observation.
Dear Editor,
I feel compelled to respond to Dr. Anis Rassi Jr's thoughtful critique of the CHAGASICS study, published in Sensible Medicine. As a co-author of the original study, I must acknowledge that his criticisms are mostly valid and raise serious concerns specially about the study's conclusions.
The most fundamental issue, which I wholly agree with, is the severely underpowered nature of the study. With only 323 patients analyzed instead of the planned 1,100 (or 256 deaths), the study achieved merely 60% power (assuming a 38.6% mortality rate in the amiodarone group and a 30% relative risk reduction in the ICD group). This dramatic reduction in statistical power renders the study inconclusive for both primary and secondary outcomes. The current conclusion stating that "ICD did not reduce the risk of all-cause mortality" is inappropriate given this limitation. Instead, we must acknowledge that the study's severe lack of power precludes any definitive conclusions, even regarding the apparent lack of reduction in overall mortality.
I also concur with concerns about the modified intention-to-treat analysis, which violated the original study design by excluding 39 patients who didn't receive their assigned interventions. This deviation from standard practice in my view remains a controversial issue, despite being supported by the Reference 29, and possibly also by its previous paper by Dr Anna Dossing and her colleagues (Dossing A, Tarp S, Furst DE, Gluud C, Beyene J, Hansen BB, Bliddal H, Christensen R. BMJ Open. 2014 Sep 26;4(9):e005297. doi: 10.1136/bmjopen-2014-005297. Interpreting trial results following use of different intention-to-treat approaches for preventing attrition bias: a meta-epidemiological study protocol.). At any rate, I would favor the insertion of this aspect on the topic of Limitations of the paper. Because, in the case of CHAGASICS, these post-randomization exclusions violate the fundamental principles of ITT that was originally devised for the analyses. Since the number of exclusions affected differentially the two groups, it might have contributed to unbalance the comparability between them. I guess the calculations were done following both strategies. Therefore, if, as postulated by a conclusion based on Dr Dossing’s paper, results were similar with the two methods, why should the Statistician have not adopted the most orthodox ITT method?
Regarding medication usage, it is pristine that there were significant errors in the percentage calculations. The paper requires correction to show the actual percentages, and the incorrect statement about low medication use influencing heart failure outcomes must be withdrawn. Furthermore, I agree that the reporting of heart failure hospitalizations at three years, while excluding subsequent data, was methodologically flawed. This selective reporting, combined with the contradictory increase in heart failure deaths, would warrant the exclusion of any conclusions about this secondary endpoint.
Another crucial point concerns the use of amiodarone in the ICD group. I believe the study database would confirm that most patients in both arms were indeed receiving amiodarone. This means that, contrary to the original conception, CHAGASICS effectively compared amiodarone alone versus amiodarone plus ICD, a distinction that significantly impacts the interpretation of results.
The inclusion of pacemaker requirement as a secondary outcome was, as Dr. Rassi Jr notes, inappropriate given that ICDs inherently include pacemaker functionality. This endpoint should be eliminated from the analysis. Additionally, while the lack of event adjudication was acknowledged in the paper, I agree that this limitation should be emphasized particularly regarding the secondary endpoints of hospitalization, sudden death, and pacemaker requirement, which are more susceptible to interpretation bias in open trials.
Regarding statistical interpretation, I share Dr. Rassi Jr's concern that both Type I and Type II errors may have occurred. The study's lack of power means that real effects of ICD might have gone undetected, while apparent differences in secondary outcomes could be unreliable due to statistical noise.
Based on these significant limitations, I must agree that no claims for regular ICD implantation in primary prevention for patients with chronic Chagas cardiomyopathy can be validated by this study. This is, in my view, the most detrimental potential consequence of the paper, as it is published, consists in that these inconclusive findings might be used to support ongoing lobbying efforts for expanded ICD use in Brazil.
Various technical errors in the paper, including incorrect citations and data presentation, would require correction through an errata or corrigenda in JAMA Cardiology.
In conclusion, I appreciate Dr. Rassi Jr bringing these important issues to light. The scientific community deserves transparency about the limitations of our research, and I believe acknowledging these shortcomings serves the greater goal of advancing medical knowledge through rigorous scientific inquiry.
Sincerely,
JA Marin-Neto
The scientific courage demonstrated by Prof. Marin-Neto in his thorough response to the methodological concerns raised about the CHAGASICS trial deserves our highest recognition. In today's academic environment, where acknowledging study limitations often carries professional risks, his commitment to scientific truth stands as a beacon of academic integrity.
His meticulous critique highlights key issues in the study, including severe underpowering, a compromised intention-to-treat analysis, unjustified emphasis on misleading secondary endpoints, concealment of crucial data, ridiculous confusion and incorrect citation related to well-known literary texts, the egregious spinning of results in the article's conclusions, and unacceptable mathematical errors - such as miscalculations in additions and percentages - that should NEVER pass peer review, especially in a journal like JAMA Cardiology. This level of scholarly rigor is essential, particularly given the study’s potential real-world implications.
While the manuscript carefully avoids explicit recommendations for ICD implantation, several authors of the CHAGASICS trial are already citing this ill-fated study in scientific presentations to support broader ICD use in Chagas disease. This concerning development highlights how fundamentally spurious research can be weaponized to influence clinical practice, particularly troubling given the resource constraints in regions where Chagas disease is endemic.
The conspicuous silence from other co-authors in the face of substantive criticism raises disturbing questions about scientific accountability. Equally troubling is the editorial process from JAMA Cardiology, which not only allowed publication of a study with such profound deficiencies but subsequently rejected a detailed letter to the editor documenting these important shortcomings. This rejection of legitimate scientific discourse through established academic channels forced this crucial debate into alternative forums.
How does JAMA Cardiology propose to address these issues now? A simple erratum seems woefully inadequate given the magnitude of the methodological errors identified. The journal's peer review process requires serious examination when such compromised research receives the imprimatur of publication while detailed, substantive criticism is summarily dismissed.
The response of Prof. Marin-Neto demonstrates that acknowledging limitations enhances rather than diminishes scientific credibility. His commitment to scientific integrity serves our field's highest purpose - advancing medical knowledge through honest, rigorous inquiry to improve patient care. In the context of a neglected tropical disease affecting predominantly underprivileged populations, such intellectual honesty becomes not just an academic virtue but principally a moral imperative.
It is a perfect teaching opportunity to learn how NOT to make a clinical trial, and Dr. Anis Rassi mastered this class.
Clinical research nudges medical decisions, implying accountability at the highest level. The interventions evaluated by this “trial” is the implantation of a permanent medical device with intended and unintended consequences. It adds to the complexity of researching a socially vulnerable population like Chagas’ patients. This study has implications.
The evidence of flaws is so overwhelming that a collective discussion should seek a justification of how such a report is published, especially in a major medical journal. Such accountability must extend to the institutions that nest studies.
In the best scenario, the flaws were caused by a lack of methodological skills. Being a meaningful physician does not depend on doing research. One should know before practicing it. The scientific art is complex and full of nuances. Physicians who are not sufficiently equipped with skills should avoid the scientific enterprise. Clinical research should be a professional rather than an amateur activity of physicians promoting themselves as publishers in major journals.
Although the authors intended to have done a randomized clinical trial, this label misleads the public to its scientific quality. From a causal inference standpoint, this study has evidence inferior to that of a good observation study. Given all the limitations, it is more compatible with a descriptive study. If framed this way, some of the information could be useful, but not about comparative efficacy.
Dr Rassi’s invitation to an open discussion makes sense. Not a debate restricted to opinion leaders. Instead, the community of physicians who consume science should speak truth to power and advocate for better science and science for the right reasons.
JAMA has been a pathetic joke for over ten years at this point. Any who doubt this in good faith need only read this excellent post.
This is a stark example of the extent to which researchers will go to in order to follow the famous adage; “it’s not the money, it’s the MONEY!” As discussed on many occasions, big Pharma, and in this case the equipment manufacturers, and even those who use their wears will do virtually anything to fatten their wallet.
ICD’s are big ticket items and a few more inappropriate insertions based on bad science more than pays for those involved in placing this instrumentation. Medicine is one of the few businesses were one can bend one’s ethics under the guise of science and purported benefit to the health of the population and personally benefit greatly. It saddens me, but I also relish and commend Sensible Medicine’s (subscribed) attempts to bring back honesty and integrity to our profession.
Ben Hourani, MD, MBA
The CHAGASICS trial has significant implications for Chagas disease management - a neglected tropical disease affecting vulnerable populations. The methodological flaws and interpretation errors in the published study risk leading to inappropriate clinical recommendations and resource allocation.
Following publication, I submitted a detailed critique identifying specific methodological and analytical errors to JAMA Cardiology. Despite following the journal's formal process for letter submission, my concerns were rejected without substantive explanation.
This lack of engagement has compelled me to seek alternative platforms to address these discrepancies. As a final opportunity, I invite both the JAMA Editorial Board and the CHAGASICS trial authors to engage in a meaningful discussion of these concerns, particularly given recent presentations interpreting the trial as practice-changing.
JAMA Cardiology's handling of legitimate scientific critique sets a precedent for academic publishing standards. The scientific community deserves transparent discussion of methodological concerns, especially when findings may influence clinical guidelines for vulnerable populations.
Dr. Ionnidis, one of the most respected researchers in the world, told us that 95% of journal articles are junk science. This article is just one example illustrating why "peer reviewed" is no longer a label assuring accuracy in research reports. It reflects badly on JAMA-Cardiology that this study passed review. It is unacceptable that they would refuse to publish the detailed critique.
Thank you for your attempts to restore integrity to the field
Well done.
This paper and Rassi’s assessment of errors would make a great teaching case for medical students and residents (and perhaps for accomplished so-called experts) on all the major points of scientific discovery and rigor.
It is imperative that the publication of comments and questions by peers should not be censored. The scientific community is much worse for the wear with this overt censorship of questioning and observation.