For anyone who participates in the publishing process – authors, reviewers, and editors – it can be a frustrating and tedious process. A proposal for the future.
AI is likely going to be great for some things (e.g., a first check), but for other things, I bet it will have to be trained very carefully. As noted by others here, many journals are publishing research with conclusions that are not supported by their data/analyses, and publication bias is preventing many corrections from occurring. I can easily see some work being rejected because it doesn't agree with the "consensus", even if the consensus flies in the face of the evidence and logic.
I have gender medicine in mind, where the only accurate published pieces I can think of in NEJM, for instance, are a few letters to the editor (for Chen et al., 2023). Many other NEJM articles instead politicize the issue or say those prioritizing evidence based medicine are engaging in "science denialism" and such, often using flawed logic or unsupported starting points. If AI trains on this, I can see getting more of the same, and thus more misinformed clinicians and patients. Leor Sapir has written NEJM up in City Journal, https://www.city-journal.org/article/new-england-journal-of-medicine-transgender-activists, but it's not just NEJM. Other journals also regularly refuse to touch anything which disagrees with what they believe already, no matter how much evidence you provide, how clearly you identify limitations of the work and guidelines they are relying upon, etc. Wikipedia already is a complete mess in this field as well, as a result.
I read "Blind Spots" and gather this is happening in several areas of medicine, but would be happy if it is not!
I agree the current system is broken, based on the observation that there are some crap studies (poor design, lousy statistical methods, egregious conclusions based on the data) that are being published in (what should be) top flight journals like NEJM.
I sympathize with peer reviewers, who are being asked to spend countless hours in the process, without any monetary or professional gain. They are definitely doing it for the love of the game, but it is just a lot to ask. Big papers get authored editorials. I agree with the suggestion I’ve seen elsewhere that peer reviews should actually be published and the reviewers publicly recognized (and credited with review authorship). It should mean something to develop a reputation as a tough but fair, productive and efficient reviewer whose stamp of approval signals a paper worth reading.
[Neither doctor nor scientist] AI is only as good as its training. Is there a medical AI trained only in medical knowledge, statistics, and grammar? And not malarkey?
My one quibble would be about the pia factor of formatting. I have to believe that all one has to do is say 'hey favorite LLM, please reformat this in the style requested by med journal X'. Why doesn't that work ?
There are some things that LLMs seem particularly well equipped to do. For example, they can be very useful in editing survey questions. Another may be what is described by Drs. Bauchner and Rivara. I heartily agree with the inefficiencies of the current system, but the thing that bothers me the most is the conscious and unconscious bias of human review. Whatever 'bias' that may exist in a bot due to its training (I'm assuming it'll only be as fair as the existing materials allow it to be) will be far less than the bias that exists in any single reviewer (I include myself of course).
Hope you are right! I expect it can be good as a logical critic which one should be able to defend one's reasoning against, which is very useful. But I have seen it come out with some completely crazy stuff, too.
Dr. Bauchner and Dr. Rivara describe the frustrations I have watched keep good clinical cases from ever being published: reformatting for each journal, dozens of boxes to check, and months of waiting only to be rejected. That process is reason enough never to write the case up, and those cases are lost.
I built Rural Hospitalist Case Reports to take a different approach. Physicians share their cases directly. There are no formatting requirements, no word limits, no fees, no rejection, and no waiting. A case is published when the author decides it is ready, under a CC-BY license with a permanent URL. We are also working on assigning DOIs so each case can be formally cited. The idea is that the clinicians who see these cases are in the best position to judge whether they are worth sharing, and readers can judge the rest.
The site is new, rudimentary, and very much a work in progress, so I welcome feedback from anyone willing to try it or share a case:
Thank you--it is going to be interesting!
AI is likely going to be great for some things (e.g., a first check), but for other things, I bet it will have to be trained very carefully. As noted by others here, many journals are publishing research with conclusions that are not supported by their data/analyses, and publication bias is preventing many corrections from occurring. I can easily see some work being rejected because it doesn't agree with the "consensus", even if the consensus flies in the face of the evidence and logic.
I have gender medicine in mind, where the only accurate published pieces I can think of in NEJM, for instance, are a few letters to the editor (for Chen et al., 2023). Many other NEJM articles instead politicize the issue or say those prioritizing evidence based medicine are engaging in "science denialism" and such, often using flawed logic or unsupported starting points. If AI trains on this, I can see getting more of the same, and thus more misinformed clinicians and patients. Leor Sapir has written NEJM up in City Journal, https://www.city-journal.org/article/new-england-journal-of-medicine-transgender-activists, but it's not just NEJM. Other journals also regularly refuse to touch anything which disagrees with what they believe already, no matter how much evidence you provide, how clearly you identify limitations of the work and guidelines they are relying upon, etc. Wikipedia already is a complete mess in this field as well, as a result.
I read "Blind Spots" and gather this is happening in several areas of medicine, but would be happy if it is not!
I agree the current system is broken, based on the observation that there are some crap studies (poor design, lousy statistical methods, egregious conclusions based on the data) that are being published in (what should be) top flight journals like NEJM.
I sympathize with peer reviewers, who are being asked to spend countless hours in the process, without any monetary or professional gain. They are definitely doing it for the love of the game, but it is just a lot to ask. Big papers get authored editorials. I agree with the suggestion I’ve seen elsewhere that peer reviews should actually be published and the reviewers publicly recognized (and credited with review authorship). It should mean something to develop a reputation as a tough but fair, productive and efficient reviewer whose stamp of approval signals a paper worth reading.
[Neither doctor nor scientist] AI is only as good as its training. Is there a medical AI trained only in medical knowledge, statistics, and grammar? And not malarkey?
My one quibble would be about the pia factor of formatting. I have to believe that all one has to do is say 'hey favorite LLM, please reformat this in the style requested by med journal X'. Why doesn't that work ?
I agree with this. AI is a much better reviewer than I am to be honest.
There are some things that LLMs seem particularly well equipped to do. For example, they can be very useful in editing survey questions. Another may be what is described by Drs. Bauchner and Rivara. I heartily agree with the inefficiencies of the current system, but the thing that bothers me the most is the conscious and unconscious bias of human review. Whatever 'bias' that may exist in a bot due to its training (I'm assuming it'll only be as fair as the existing materials allow it to be) will be far less than the bias that exists in any single reviewer (I include myself of course).
"great journals". You keep using those words. I do not think it means what you think it means.
I am not the most experienced in publishing but as both an author and reviewer, I am optimistic about AI in peer review.
Hope you are right! I expect it can be good as a logical critic which one should be able to defend one's reasoning against, which is very useful. But I have seen it come out with some completely crazy stuff, too.
I liked this. Much better than the AI alternative.
Dr. Bauchner and Dr. Rivara describe the frustrations I have watched keep good clinical cases from ever being published: reformatting for each journal, dozens of boxes to check, and months of waiting only to be rejected. That process is reason enough never to write the case up, and those cases are lost.
I built Rural Hospitalist Case Reports to take a different approach. Physicians share their cases directly. There are no formatting requirements, no word limits, no fees, no rejection, and no waiting. A case is published when the author decides it is ready, under a CC-BY license with a permanent URL. We are also working on assigning DOIs so each case can be formally cited. The idea is that the clinicians who see these cases are in the best position to judge whether they are worth sharing, and readers can judge the rest.
The site is new, rudimentary, and very much a work in progress, so I welcome feedback from anyone willing to try it or share a case:
https://rural-hospitalist-case-reports.web.app
Kirk Dodson, MD