What Makes a Good Journal?

I’ve been a bit obsessed with what to do about the quality of journals today. I’ve written about how we can fix them and how we can figure out which ones should be shuttered. Today, we have someone who knows a thing or two about journals answering the question, “What makes a good journal?”

Adam Cifu

I am often asked what makes a good journal — or a similar question — which journal should an author try to publish in, often with the hidden agenda of the person wanting to publish in a high-impact journal. Publishing in a high-impact journal is important for a career. It can lead to increased recognition and the likelihood of promotion and grants. It is also important for the academic’s institution, because most institutional rankings include an assessment of publications. Assessing the quality of a journal includes many factors – let’s start with the most recognized: impact factor

Impact Factor

The impact factor (IF) is a journal’s Framingham Heart Score (FHS). Both were introduced in the 1960s. Like the FHS, the IF is simple to calculate, has strong face validity, and has stood the test of time. The IF is the citations that articles published in a journal collect over 2 years, divided by the number of substantive reports. What are substantive reports – all research reports, even those that are brief, and long reviews, generally more than 1500 words and 10 references. Every other article in a journal (including opinion pieces -- such as editorials, viewpoints, and perspectives -- research letters, or short article types, like guideline synopses, or reviews of statistics) do not count in the denominator. In general, about 2/3 of citations of a journal are generated from original research, and the rest from the so-called value-added material.

There are some important issues about the IF to consider.

1. IFs of journals in some disciplines will be higher than those in other areas simply because there are more journals in those disciplines. Cardiology and oncology journals will outperform pediatrics and ophthalmology ones.

2. The IF of a journal does not speak to the quality of any single article. – There are important articles in journals with low IFs, just as there are poorly conceived and analyzed articles in journals with high IFs.

3. A journal with an IF of 22 is no better than one with an IF of 20. But, when the IF difference exceeds 20-25%, then I begin to consider that one journal has richer content than the other.

4. When choosing where to publish an article, authors should consider more than just the IF. For example, JAMA Pediatrics has an IF of 18, while Pediatrics has an IF of 6.4. Yet Pediatrics is the official publication of the AAP, and virtually every pediatrician in the US has access to it. So, if an author wants to reach the US pediatric community, they may be better off submitting their manuscript to Pediatrics.

5. Recently, a new issue has arisen – what should count as “research”? The European Heart Journal regularly publishes guidelines, which are well cited. This contrasts with Circulation, Journal of the American College of Cardiology, and JAMA Cardiology, which rarely publish guidelines. Guidelines (as well as “big epi” and similar manuscripts) contribute substantially to the IF but some may not consider guidelines to be research.

6. There has been article creep – that is, longer and longer short article types that Clarivate, the company that calculates and releases the IF every Spring, does not classify as substantive (count in the denominator of the IF). Journals simply ensure that the article is not classified as substantive, in part by limiting the number of references.

Other Citation Measures

There are other citation measures such as the H-index, Cite-Score, and Scimago ranking. What is counted in the denominator and the time counted may be different, how citations are weighted may be different, but all these measure the same thing – how often the content of a journal is cited in other journals. Many of the journals with the highest IF also have the highest Cite-Score and Scimago ranking.

Readership

Another important metric for a journal is the number of people who read it. Indeed, one should never submit a manuscript to a journal unless they have read it or at least perused its content. That said, I am not sure anyone reads a journal any longer. Prior to the internet, journals would arrive by mail weekly, bi-weekly, or monthly. They were often identified by unique characteristics: JAMA had art on the cover, NEJM had its table of contents, Lancet had a quote. One could peruse the table of contents and then decide what to read. The world has changed. Now journals arrive in bits and pieces, almost daily, with those bits and pieces collected into an issue periodically. Does anyone ever “read” a journal any longer? But I am sure when an alert pops into an inbox, readers pay more attention to content from journals they like – hence, the 2nd important way to judge a journal – do you like reading it?

Circulation (actual circulation, not the journal)

Journals are far more than publication platforms; they are communication networks. Circulation should now be thought of as the reach of a journal through the distribution of its table of contents or as bits and pieces on the internet and various social media platforms. When I first arrived at JAMA in 2011, we had ~15,000 followers on social media. When I left in 2021, we had more than 1.2 million. Articles published in the leading journals travel around the world instantaneously. In general, print circulation has declined dramatically over the past 10 years, but the “reach” of journals has increased.

Recently, the NIH began to require that studies it sponsors be immediately available on the day of publication. This is like other funders that require that authors publish in open-access journals. The NIH requirement permits depositing the accepted manuscript in PubMedCentral (PMC). One of the major differences between PMC and a journal, and why I do not believe the NIH announcement regarding open access is consistent with the philosophy of open science, is because no one “reads” PMC – it is a repository, whereas journals are communication networks. In addition, the new NIH mandate now creates the possibility of 3 versions of a manuscript: a pre-print, an accepted version, and a published version.

Conclusion

To make more informed decisions and to support transparency, journals should provide readers and authors with data, including the various traditional measures of “quality” beyond the IF. This should include its reach (print circulation, social media activity, and the number of individuals receiving the electronic table of contents), as well as yearly downloads. Authors should also know the so-called turnaround times – how long does it take for a journal to process a manuscript? The turnaround time should be separated into research reports and other content. Because journals often reject manuscripts without review, generally within a few days, the data on research reports should be categorized by the percentage rejected without review and by turnaround times for published manuscripts. Authors need to carefully consider whether it is worth submitting a manuscript to a journal that takes months to reach a decision.

No single measure reflects the quality of a journal or of an individual manuscript. Whether an individual likes reading the journal and its impact factor are good starting points. However, there are other important considerations. For clinicians, value-added content, including opinion pieces, editorials, news, and short articles, may be important. For investigators, the focus might be on traditional research content. Evaluating the quality of a journal — which one to read and where to publish — needs to be carefully considered.

Howard Bauchner, MD, is a Professor of Pediatrics and Public Health at the Boston University Chobanian & Avedisian School of Medicine. He is also a Visiting Scholar at the National University of Singapore. Dr. Bauchner is also the former Editor in Chief of JAMA and the JAMA Network and former Editor in Chief of Archives of Disease in Childhood.

Photo Credit: Jan Kahánek