Anthropic’s IPO filing lists 93 pages of risks. What could possibly go wrong?

When Anthropic registered for its widely anticipated initial public offering on Monday, the artificial intelligence developer framed its technology as potentially game-changing for all of humanity. A dangerous game. 

Nearly a third of the 335-page securities filing is devoted to outlining not only the routine risks any company might face as it seeks to grow its business, but also the extraordinary risks that Anthropic’s frontier models could pose.  

To be sure, companies planning to go public commonly disclose a sprawling set of risks in their IPO filings, noted Jay Ritter, an IPO expert and professor at the University of Florida’s Warrington College of Business. 

Yet the extensive risk section in Anthropic’s S-1 filing also highlights a financial and operational tension at the heart of AI companies as they pitch themselves to investors. The value of artificial intelligence businesses, which in Anthropic’s case is expected to top $1 trillion, depends on the continued spread of a technology that even its developers acknowledge could cause worldwide harm. 

Read on for a snapshot of the risks Anthropic outlines in its IPO.

“Existential risks” 

Anthropic, which deveoped the Claude chatbot, notes in the IPO filing that its technology could pose “catastrophic or existential risks to humanity,” a stark contrast to the company’s founding mission to build AI for the “benefit of humanity.” 

“The company is asking investors to fund a race to more autonomous AI while acknowledging that capability could outpace safety,” David Hsu, a professor at the University of Pennsylvania’s Wharton School who specializes in technology and entrepreneurship, said in an email. “This is not an ordinary technology risk. It is a governance issue with potentially societal consequences.”

Anthropic admits that maintaining guardrails around its technology could be difficult, noting that AI models could “resist shutdown” and even engage in “blackmail” to persuade humans to “preserve the model’s continued operation or access.”

Models could also take actions that run counter to a user’s original intent, like gaining access to third-party systems, Anthropic said. And as AI models gain greater autonomy, Anthropic said the potential consequences of such “misaligned behavior” could grow more severe. 

Even if humans remain in control of AI, the technology could still be abused. In its IPO document, Anthropic said bad actors could use its models to develop weapons, conduct cyberattacks and engage in other illegal activity.

Self-replicating tech

Anthropic’s reliance on its own technology to produce future AI models — a concept known as “recursive self-improvement,” or RSI — represents another potential vulnerability. Although AI doesn’t yet have that capability, it could come sooner than people are ready for, the company warned earlier this year.

Through RSI, models could replicate flaws or errors in future models, something that would be “difficult to detect or remediate,” Anthropic said in the IPO filing. Indeed, new models could develop so quickly that it may not leave enough time for Anthropic to evaluate and safeguard them, the company acknowledged.

Cyberattacks

Anthropic warned that hackers could gain unauthorized access to the company’s proprietary data, including information about its finances, models and other intellectual property, as well as users’ personal data.

Such threats could come from external sources, such as nation-states or state-sponsored actors, or even company insiders with access to its systems, Anthropic said. For example, bad actors could try to gain employment at Anthropic under a false identity to access proprietary information. Employees could also be pressured to share sensitive data with external actors, Anthropic said.

Legal exposure

Anthropic said it could be sued over harm caused by its AI models. Rival AI developer OpenAI, the company behind ChatGPT, has faced lawsuits over claims that its technology was used to coach people to commit suicide.

At the same time, Anthropic concedes it may not be able to adequately insure itself against such liabilities, given that the insurance industry has never had to write policies covering AI-related risks.

“With the scale of harm that could result from a catastrophic AI-related incident, no single insurer, and potentially the insurance industry in the aggregate, may have sufficient capacity or expertise to underwrite insurance commensurate with such exposure,” Anthropic said in its IPO filing.