Ahead of going public, Anthropic is nearing a $1 trillion valuation, surpassing OpenAI—now valued at $862 billion—to become the world’s most valuable AI company. Not long after that news broke, on Wednesday, Anthropic CEO Dario Amodei published a blog post and policy framework outlining his preferred way for AI companies like Anthropic to be regulated. Not for the first time, he warns that his company’s products promise both an ill-defined set of benefits and potentially catastrophic risks that “could even threaten humanity itself.”
Luckily for us, he has a plan to keep the products that are making him rich from wiping everyone out. Amodei suggests the government should “have the power to block or deter deployment” of models it deems too risky. He calls for frontier large language models to be subjected to technical testing and auditing; workplace protections against AI-related job displacement; coordination among “allied democracies” against “adversaries”; and limits on the use of LLMs’ use in warfare and for domestic surveillance. Anthropic has previously suggested that a pause on frontier model development might be worthwhile, but—for now—impossible, as it might allow “the least cautious actors catch up technologically.” Like OpenAI’s progressive-coded “New Deal,” Amodei’s vision contains plenty of nice-enough-sounding ideas that are unlikely to be implemented so long as Donald Trump is in the White House and our political system is being pumped full of donations from Amodei’s fabulously wealthy, openly reactionary colleagues in Silicon Valley. Amodei, of course, laments the lack of global coordination on these issues, and the disconnect between the scale of the problem at hand and the pace of policymaking: “We now, globally and collectively, need to activate a slow and rickety policy apparatus to deal with risks and opportunities that are going to compound surprisingly quickly from here.”
As a climate reporter, I find Amodei’s admonitions eerily familiar. For decades, scientists have warned about the enormous dangers posed by continuing to burn fossil fuels that deposit greenhouse gases into the atmosphere and warm the planet. Once policymakers seemed to be taking those concerns seriously, the companies whose products have fueled global warming—and that backed efforts to downplay its importance—started to ape those scientists’ warnings, laying out their own plans for a “transition” to a vaguely defined future known as “net-zero.” Ahead of U.N. climate talks in Paris in 2015, for instance, Saudi Aramco, Shell, and other major fossil fuel producers announced their “collective support for an effective global climate change agreement.” Many backed the implementation of a global carbon tax that there was no practical means of implementing, especially given that a foundational premise of what became the Paris climate agreement was that its goals would be nonbinding. The companies poured millions into academic institutions that lent credibility to the idea that fossil fuel companies would play a leading role in the transition to a fossil fuel–free world.
These moves weren’t all cynical theatrics. In the lead-up to U.N. climate talks in Glasgow, in 2021, a few European producers released somewhat plausible-sounding plans to start actually scaling back their oil and gas operations and invest in renewables. That was seemingly out of a fear that governments might actually start requiring them to do that, but also because there were a few greenish areas—like carbon storage—that aligned well with their expertise and core business model. Governments never did enforce a global energy transition, and most of those lofty industry climate plans have been walked back. Throughout that saga, at every level of government, even the allegedly more climate-conscious oil and gas companies continued to lobby against laws and regulatory proposals that weren’t to their liking.
The regulatory proposals from Anthropic and OpenAI are different from polluters’ net-zero plans in meaningful ways. It may be the case that Amodei, at least, really does believe the scary stories he tells about Anthropic LLMs creating biological weapons and defying their creators. Unlike fossil fuel CEOs, Amodei and Altman have been among the loudest voices broadcasting the existential risks their products pose. However, genuine or not, Amodei and Altman’s philosophizing about the allegedly mystical properties of their products enables them to cast themselves as guided by some deeper, more altruistic purpose because of their access to a special kind of knowledge that endows them with a power nobody else has: If they’re the only ones who truly understand the awe-inspiring powers of Claude and ChatGPT, then who else could possibly know how to regulate them and avert dystopia?
The truth is that there’s a business imperative for the likes of Altman and Amodei to avoid talking about the middle ground between the techno-futurist utopia and/or existential threats promised by artificial intelligence—all squishy concepts in their own right. Increasingly advanced large language models may well turn out to be massively important for the businesses that can afford to automate enormous numbers of the entry-level programming jobs and administrative positions. They could at once help advance some genuinely exciting medical breakthroughs and create a generation of kids who never learn to read or think for themselves. Scammers might figure out new ways to trick your grandparents into signing away their life savings as governments automate warfare and make manual tax preparation a thing of the past. These are all transformative developments in their own right. They pose novel dangers that governments ought to take seriously. They do not add up to a new god.
Recent headlines, moreover, lend some credence to the idea that AI developers are on the verge of acting more like normal corporations. Companies who’ve rushed to embrace LLMs are running up unsustainable bills on their token usage, i.e., monetized units of usage that users pay for based on how compute-intensive and numerous the tasks are that they’re asking LLMs to do. Results have been mixed, and, for now, automation remains an expensive and (human), labor-intensive task for many firms. In response to such concerns, OpenAI has signaled that it will start lowering its prices to compete with Anthropic.
The most important difference between oil and gas producers’ climate pledges and AI companies’ recent policy proposals is that the world does not run on large language models. Fossil fuels are the foundation of modernity. As the now monthslong closure of the Strait of Hormuz has shown, abruptly cutting off supplies of coal, oil, gas, and the many products derived from them is economically disastrous. And whereas few people on earth can remember life without fossil fuels, just about everyone can recall—perhaps fondly—a world before Claude and ChatGPT. Despite their best attempts to convince the public and policymakers otherwise, these companies are neither too big to fail nor too magical for regular people to understand.
That’s not a case for pulling the plug so much as for seeing through the religious bromides that Amodei and Altman use to describe their companies. Anthropic and OpenAI’s products should indeed be subject to stringent regulations. Their billionaire CEOs are just that: executives with a financial interest in a regulatory regime that preserves their business model and future earnings. Like their counterparts in the fossil fuel industry, it’s their prerogative to try to convince policymakers and the public that they have a good-faith interest in our collective well-being. But no one should mistake them for philosopher kings building a god. Dario Amodei and Sam Altman have some doctrinal differences, but they are both—above all—wealthy men who want to keep getting wealthier by selling their products however and to whoever they want.
