Researchers prefer AI for pre-submission article review

Survey indicates that scientists view AI as an aid to improve manuscripts, but reject its adoption as a substitute for human peer review

Rodrigo de Oliveira Andrade

A researcher analyzes scientific data in front of monitors displaying images of the brain. She makes notes on a clipboard.

Researchers prefer to use artificial intelligence (AI) as a self-assessment tool before submitting their manuscripts, but do not view it as a substitute for human peer review, at least in the short and medium term, according to a survey published in February in the scientific journal EMBO Reports.

In the study, the authors invited scientists whose manuscripts had been evaluated by the preprint review platform Review Commons to participate in the survey. Review Commons conducts a peer-review process independent of scientific journals without making editorial decisions on acceptance or rejection following the review.

The participants were then provided with an AI-generated review of the same manuscript produced by the qed platform for comparison.

The platform analyzes the content of articles to identify their main scientific claims and potential gaps in the evidence supporting them, suggesting ways to address these shortcomings—whether through additional experiments or changes to the text that moderate or reformulate certain conclusions.

The authors then asked participants to compare human and AI-generated reviews based on seven quality criteria: ability to critically identify relevant gaps; level of detail regarding aspects addressed in the paper; constructiveness of suggestions; depth of understanding of the research’s objectives and relevance; substantiation of critiques based on data, figures, and scientific literature; clarity of writing; and use of professional and neutral language.

Of the 408 invited scientists, 126 responded to at least part of the questionnaire.

Overall, human peer review was rated higher in aspects related to scientific understanding and the ability to offer relevant insights.

The only exceptions were the criteria of clarity and cordiality, where reviews produced by AI and by humans performed similarly.

Next, the authors asked participants to evaluate different possible uses of AI-generated reviews, allowing them to choose up to three options from a list of nine alternatives.

The most preferred option was for the participants themselves to use the tool to review and refine their manuscripts before submitting them to scientific journals. The second-highest-rated possibility was the use of AI as an aid for human reviewers.

Limitations

On the other hand, there was little acceptance of the idea of scientific journals relying primarily on reviews produced by artificial intelligence, even under human editorial supervision—although the authors acknowledge that this perception may vary depending on the profile of each scientific journal.

Similarly, the possibility of automatically reviewing all preprints on a large scale was not considered an attractive alternative by the participants.

An important limitation of the survey is that participants knew the source of each review, which means that responses may have been influenced by prior perceptions—whether favorable or unfavorable—regarding the use of AI in scientific research.

The small size and specific profile of the sample do not allow for generalizations across the entire scientific community. Nevertheless, the results offer important insights into emerging trends in the use of AI in scientific peer review.

AI as an Auxiliary Tool

Overall, the survey points to a cautious openness among participants toward the use of AI as an aid. In the open-ended comments section of the questionnaire, perceptions regarding the role of AI in the peer review process remained ambivalent.

Some participants said they were surprised by a machine’s ability to “understand” scientific articles at such a sophisticated level.

Others even considered that, in certain situations, reviews produced by AI could surpass those produced by human reviewers.

Many participants, however, argued that AI-based reviews should remain under human supervision and not replace the traditional peer-review model.

Broader concerns were also raised, such as the risk of losing the critical diversity provided by the different experiences and interpretations of human reviewers, as well as a possible “progressive homogenization” of scientific reviews.

Based on the survey responses, the authors argue that AI can help scientists address some of the current problems related to scientific publishing.

“Although still imperfect, these systems offer interesting advantages: they are extremely fast, do not compete with researchers, do not get tired, have ‘infinite patience,’ and are able to analyze figures and supporting materials, and verify statistics systematically,” the authors wrote.

“When properly designed, they can also provide more consistent evaluations,” they noted.

To reap these benefits, however, the use of AI in scientific peer review must adhere to two central principles: a focus on the author—that is, using the tool to help researchers improve their manuscripts—and transparency.

* This article may be republished online under the CC-BY-NC-ND Creative Commons license.
The text must not be edited and the author(s) and source (Science Arena) must be credited.