In April, developmental neuropsychologist Dorothy Bishop noticed strange phrases in a 2021 study of autistic children who interacted with a human-like robot. It described the children as “tainted kids” and “medically introverted,” and it referred to autism as a “disease” and a “mental imbalance.”
“This paper is not only complete gobbledegook, but it also manages to be offensive to autistic people and their families,” wrote Bishop, professor of developmental neuropsychology at the University of Oxford in England, on the post-publication peer-review site PubPeer, where scientists discuss research papers.
The study is one of half a dozen autism studies flagged by the Problematic Paper Screener, an online tool that searches for “tortured phrases” — strangely worded paraphrases of established terms — and other red flags in published work. The phrases likely result from academics trying to circumvent plagiarism detectors by using paraphrasing software, such as SpinBot, to generate synonyms without much thought, says Guillaume Cabanac, associate professor of computer science at the University of Toulouse in France, who created the Problematic Paper Screener.
Cabanac manually waded through the tortured phrases in the robot study after he posted the screener’s findings on PubPeer in March. He confirmed that the author had apparently substituted “man-made consciousness” for “artificial intelligence” and “Alzheimer’s sickness” for “Alzheimer’s disease.” He verified similar oddities in a 2022 study about using machine-learning methods to detect autism early in life.
Alexander Magazinov, a software engineer at Yandex in Moscow, Russia, and a collaborator of Cabanac’s, manually sifted through a 2014 Frontiers in Psychiatry autism paper that has accumulated more than 100 citations to date. The paper was flagged by Cabanac’s tool but contains only one potentially problematic phrase: “super vector machine,” instead of “support-vector machine” — likely an unfortunate typographical error, say several researchers, including Bishop. Spectrum has reached out to study author Robert Hendren, a psychiatrist at the University of California, San Francisco, for confirmation.
The screener flagged three other autism studies — about using machine learning to predict and classify levels of the condition, and chatbots or virtual reality to communicate with autistic children — that still await manual review. Spectrum contacted the corresponding authors of all the autism studies in question.
“I see the Problematic Paper Screener like an informant that would tip off detectives about papers,” Cabanac says, noting that all flagged papers should be screened manually.
T
he screener deemed about 7,650 studies problematic, including more than 6,000 for having tortured phrases. Most papers containing tortured phrases seem to come from the fields of machine learning, artificial intelligence and engineering. Cabanac says he isn’t sure why autism papers are just starting to crop up.Of the six autism studies flagged, five are book chapters. A spokesperson for Wiley-Blackwell, which published one of the flagged chapters, said the company is investigating the chapter in accordance with guidelines from the Committee on Publication Ethics.
And Springer Nature, which published two of the chapters, is investigating a number of reports related to tortured phrases, says Tamara Welschot, head of advisory and assurance in the publisher’s Research Integrity Group.
“We have also developed a tool that assesses manuscripts for signs of artificial manipulation, and this has started to support us in our proactive investigations,” Welschot says. “Tortured phrases may be an indication that there are integrity issues associated with a paper or chapter, although in some cases they can indicate that a paper or chapter is written in a language that is not the author’s primary one.”
The flagged papers highlight the lack of a robust peer-review system and the capacity to quickly correct the literature, says Jennifer Byrne, professor of molecular oncology at the University of Sydney in Australia and the co-creator of one of the Problematic Paper Screener’s detectors that identifies errors in genetic sequences.
“It needs a complete overhaul,” Byrne says, referring to scholarly publishing. She argues that scholarly publishers don’t have enough incentives to make sure their content is solid. “I think that there needs to be some fairly strong backlash from the consumers of the literature to indicate that this is simply not acceptable and that it will not be tolerated.”
Bishop, however, says that papers with tortured phrases are probably not having much of an impact. “I get very concerned with dodgy science that is then taken seriously, where the work is picked up and built on and cited,” she says. “But these things wouldn’t be because they’re so weird and they don’t mean anything.”
