AI is fabricating citations in biomedical studies, researchers find

Artificial intelligence is fabricating references to medical research that does not exist, according to recent findings. 

A recent audit found that, among millions of biomedical papers, more than 4,000 contained citations to non-existent research, according to an article in The Lancet. Such fabricated citations can undermine the clinical guidelines that health care professionals rely on to provide care, said Maxim Topaz, an associate professor at the Columbia School of Nursing and the study’s lead author.

An audit of millions of biomedical papers found more than 4,000 citations to bogus studies, the researchers said in a recent article published in The Lancet. 

Fabricated citations are dangerous because they influence clinical guidelines, which are based on public research that health care professionals follow in providing care, Maxim Topaz, an associate professor at the Columbia School of Nursing and the study’s lead author, told CBS News.

“When those fake references are making it into the literature, they will end up in those guidelines, and that’s how doctors decide how to provide care for you,” he said. “Your doctor could be making decisions around treatment based on studies that never existed.”

Growing problem

Also troubling is that none of the mistakes Topaz and his team identified have been corrected or retracted, and could still be influencing patient care, he said.  

“The rate of fake references showing up in published medical literature is growing,” Topaz added, noting that the number of such erroneous citations has grown 12-fold over the last three years. The fabricated references spanned nearly 3,000 academic papers.

Topaz’s own experience spurred him to investigate the issue. An AI app he was using to help polish one of his own scientific papers inserted a fake citation, he told CBS News. It then slipped through several layers of peer reviews before one sharp-eyed editor caught the phony reference. 

“I was mortified, because I’ve been studying AI for the past 15 years, so if it can happen to me, it can happen to anyone,” he said.

Such mishaps arise when an author asserts a statement of fact and asks AI for a citation, Topaz explained. “In some cases, AI would slip those in, inadvertently,” he said. “You would hope the facts are accurate, but if they are supported by fabricated citations, you don’t know if the ‘facts’ are accurate.”

In some cases, an AI tool will also cite a real author while inventing research and attributing it to that person. Other times, citations were completely fabricated, Topaz said. 

“This is just the tip of the iceberg,” he said, noting that research across other fields could also be subject to the same issues.

Meanwhile, faux AI-generated scientific citations can “look perfectly real,” Topaz added, who emphasized the importance of researchers rigorously fact-checking their work.