Melatonin, Heart Failure, and Churnalism

We have published a series of articles on churnalism, defining it and providing examples. While the actual definition of churnalist is the reprocessing of press releases into new articles, we define churnalism as the careless, incurious reporting of poorly conducted biomedical research. Churnalism trades the real story — why a study is unimportant or proves something other than it contends — for the easy headline. Sensible Medicine began its coverage of churnalism with a series of introductory articles.

Dr. Edward Livingston is back with us today, taking up the churnalism mantle to discuss data that was all over the news.

Adam Cifu

Recently, this was the top medical news story of the day:

“Among adults with insomnia, those whose electronic health records indicated long-term melatonin use (12 months or more) had about a 90% higher chance of incident heart failure over 5 years compared with matched non-users (4.6% vs. 2.7%, respectively).”

Nearly every major news outlet covered this story based on an American Heart Association press release about a presentation to be given at their annual meeting. The science underlying this statement does not justify the conclusion, but how is the average reader supposed to know that? Knowing what to believe or not believe about scientific findings requires some knowledge of how science is done.

The very first step is making an educated guess about something: generating a hypothesis. No hypothesis regarding what it is about melatonin that would cause heart failure was provided. The study justification was “Melatonin supplements are promoted and marketed as a safe sleep aid; however, data demonstrating their long-term cardiovascular (CV) safety are lacking.” Why would someone be concerned about melatonin’s CV safety?

Failure to identify a plausible reason why melatonin causes CV disease should signal to readers that the findings should be ignored. A corollary to this is in the way the study rationale was stated-as a negative-information was lacking. The investigators went fishing in a sea of data hoping to catch something. Because they landed heart failure does not mean heart failure had anything to do with melatonin use. More about that later, but suffice it to say that unless authors state why they believe two phenomena are related, the findings should be ignored.

The second part of any scientific endeavor is the collection of data that will be used to answer research questions. How well the data represent the phenomena being studied is often overlooked when research findings are interpreted. Unfortunately for medical science, it is common for research data to have serious limitations that are often overlooked.

The melatonin research queried a database, TriNteX, that is not a reliable source of research data. A plethora of very large but low-quality data sources exist for medical research purposes. It is hard for non-experts to know what to trust. Because it is more likely than not for medical research to use unreliable data sources, readers of the literature should ignore any research findings that are not associated with reporting of the data source’s quality. Another signal for the non-trained reader of the melatonin articles to not accept the findings is that there was no discussion of what TriNetX is and how its use accurately reflects the universe of patients who take melatonin, who might, in turn, be at risk for developing heart failure.

“Using a large international database (the TriNetX Global Research Network), the researchers reviewed 5 years of electronic health records for adults with chronic insomnia who had melatonin recorded in their health records and used it for more than a year. They were matched with peers in the database who also had insomnia but never had melatonin recorded in their health records. People were excluded from the analysis if they had previously been diagnosed with heart failure or had been prescribed other sleep medications.”

TriNetX is a huge black-box (or cesspool if you prefer) database composed of information extracted from a multinational collection of 220 health care organizations. It is a black box, lacking information about where the data comes from or how reliably it is reported. The patient characteristics of those contributing to the data are unknown. It is not population-representative. Validation of the data within TriNetX is not publicly available. Because one does not know how reliable the data are and how that data might reflect any individual patient, findings derived from it are subject to many limitations.

Databases like TriNEtX also have problems accurately identifying patients with the condition being studied. In the melatonin study, “adults with insomnia” were examined. Patients with insomnia were identified by ICD-10 codes. Diagnostic coding in administrative databases is frequently inaccurate. Thus, it is not known if the melatonin study captured all patients with insomnia or only those with insomnia severe enough to merit ICD-10 coding in the medical record. It is conceivable that insomnia is rarely captured when present, as is the case with dyspnea in bariatric surgery, where almost all patients have it when asked, but it rarely appears in diagnoses in medical encounters.

The study authors conceded that the main exposure, melatonin ingestion, is also unknown. They were able to count prescriptions for the drug, but it is available over the counter. Even if the analysis were restricted to countries with no OTC melatonin, one does not know if patients who are prescribed the drug actually use it and, if they do, how much they take.

What about the enormous press coverage of this flawed study? It demonstrates problems with medical journalism. The modern medical journalist tends to field press releases and write articles based on what is in the release, with little in-depth investigation. Because of the very short attention span of the modern reader, these journalists need to produce content every day. They have little time to fully vet stories.

The internet facilitates the unprecedented exchange of information. Much of that information may be misleading. How to know what to believe is hard. The best chance is to read material on trusted platforms, but these are rare. Previously trusted organizations and journals are failing. I’m betting on new platforms such as Sensible Medicine and others like it to fill the void as trusted sources of information.

When deciding what to trust in science reporting, look for the articulation of a study question, why that question is plausible, how the data represent the phenomenon being studied, and that the data truly represent the phenomenon being studied. Ignore reporting that lacks this sort of discussion or, worse yet, simply wants you to uncritically accept the headline assertion.