Situational Awareness - The Decade Ahead

Situational Awareness: The Decade Ahead is an article written by Leopold Aschenbrenner in June 2024 that discusses the rapid progress in artificial intelligence (AI) and the potential for artificial general intelligence (AGI) to emerge by the year 2027. The article argues that AGI could lead to an intelligence explosion, resulting in AI systems that are vastly smarter than humans, with dramatic implications for society, the economy, and national security.

From GPT-4 to AGI

The article begins by tracing the remarkable progress in language models over the past few years, from GPT-2 in 2019, which exhibited roughly preschooler-level abilities, to GPT-4 in 2023, which demonstrated the capabilities of a smart high school student. Aschenbrenner argues that given the consistent trends in compute scaling, algorithmic efficiency improvements, and techniques for "unhobbling" AI systems, we should expect another leap of similar magnitude within the next four years, potentially resulting in AGI by 2027.

Aschenbrenner breaks down the key drivers of progress as follows:

Compute: The compute used to train the largest language models has been growing at a rate of roughly 10x every 2 years. The article estimates that the GPT-4 training cluster likely cost around $500 million, and that we may see $100 billion to $1 trillion training clusters by the end of the decade (p. 7-8).

Algorithmic Efficiency: The article estimates that algorithmic improvements have been contributing the equivalent of 0.5 OOMs (orders of magnitude) of effective compute per year on top of hardware improvements. Techniques like better model architectures, training procedures, and scaling laws have compounded to enable training runs with 100x less compute (p. 21-26).

"Unhobbling": Language models have latent capabilities that can be unlocked through techniques like reinforcement learning, chain of thought prompting, tool use, and context length expansion. Aschenbrenner argues that with further "unhobbling", language models will go from narrow chatbots to general-purpose AI agents and automated engineers (p. 27-38).

The article makes the case that we should expect a jump of 5-6 OOMs of effective compute from GPT-4 to potential AGI systems by 2027, a leap similar in magnitude to going from GPT-2 to GPT-4. Aschenbrenner argues this could plausibly result in AI systems that can match or exceed human abilities on most cognitive tasks.

As Aschenbrenner writes:

"Put together, this suggests we should expect something like 1-3 OOMs of algorithmic efficiency gains (compared to GPT-4) by the end of 2027, maybe with a best guess of ~2 OOMs. [...] 5 OOMs of algorithmic wins would be a similar scaleup to what produced the GPT-2-to-GPT-4 jump, a capability jump from ~a preschooler to ~a smart high schooler. Imagine such a qualitative jump on top of AGI, on top of Alec Radford." (p. 39-40)

From AGI to Superintelligence

The article goes on to argue that progress will not stop at human-level AGI, but that we should expect an "intelligence explosion" soon after AGI is achieved. Aschenbrenner outlines a scenario in which the initial human-level AI systems are used to automate AI research itself, resulting in a feedback loop of rapidly increasing capabilities.

Key points include:

Using AGI systems to perform AI research could compress a decade of algorithmic progress into a single year, resulting in a further leap of 5+ OOMs of effective compute on top of the AGI systems (p. 46-56).

With large inference compute budgets, we may be able to run upwards of 100 million copies of human-level AI researchers (p.50). Combined with potential 10-100x speedups, this could result in an effective AI research effort orders of magnitude larger than the total human AI research community.

While there are potential bottlenecks like limited compute for experiments, the need for human oversight, and harder algorithmic problems, Aschenbrenner argues these are unlikely to prevent at least a 10x speedup of algorithmic progress (p. 52-66).

The AI systems that would result from this algorithmic intelligence explosion could quickly go from roughly human-level to vastly superhuman. As Aschenbrenner writes: "5 OOMs of algorithmic wins would be a similar scaleup to what produced the GPT-2-to-GPT-4 jump, a capability jump from ~a preschooler to ~a smart high schooler. Imagine such a qualitative jump on top of AGI, on top of Alec Radford." (p. 53)

The article goes on to explore some of the potential implications of such superintelligent AI systems:

"The AI systems we'll likely have by the end of this decade will be unimaginably powerful. Of course, they'll be quantitatively superhuman. [...] More importantly—but harder to imagine—they'll be qualitatively superhuman. As a narrow example of this, large-scale RL runs have been able to produce completely novel and creative behaviors beyond human understanding, such as the famous move 37 in AlphaGo vs. Lee Sedol. Superintelligence will be like this across many domains." (p. 66-67)

Aschenbrenner argues that an intelligence explosion to superintelligence could have world-changing impacts within a matter of years:

Superintelligent AI could be applied to accelerate scientific and technological progress across fields, automating research and compressing 100 years of discoveries into a decade (p. 68-69).

The development of human-level AI and beyond could enable a "robot revolution" and a new era of explosive economic growth, as intelligent machines fully automate both cognitive and physical labor (p. 69-70).

A lead of even months in developing superintelligent AI could provide an overwhelming military and strategic advantage to the country that achieves it first (p. 70).

Challenges on the Road to Superintelligence

The article dedicates four sections to outlining key challenges that will need to be confronted as AI systems become more advanced.

Racing to the Trillion-Dollar Cluster

With the rapid increase in AI capabilities coming from scaling up compute, Aschenbrenner projects an unprecedented compute buildout over the coming years. The article includes detailed calculations estimating the following compute and cost requirements:

By 2024: ~$150 billion/year capex, ~5-10 million H100 GPU equivalents shipped, 1-2% of US electricity production (p. 79-80)
By 2026: ~$500 billion/year capex, ~10s of millions H100 equivalents, 5% of US electricity (p. 80)
By 2028: ~$2 trillion/year capex, ~100 million H100 equivalents, 20% of US electricity (p. 80)
By 2030: ~$8 trillion/year capex, ~100s of millions H100 equivalents, US electricity production would need to double (p. 80)

Aschenbrenner argues this unprecedented buildout is plausible given the economic returns to training powerful AI systems, historical precedents for large technology investments, and the ability of the US to build out compute capacity, especially with natural gas. However, he notes there will be major challenges, especially in securing enough power and pushing semiconductor production to the required scale.

Lock Down the Labs: Security for AGI

The article argues that the key algorithmic insights required to build AGI over the next few years, and later the weights for AGI models themselves, will become the "crown jewels" from a national security perspective. However, Aschenbrenner claims leading AI labs are currently not even close to having sufficient security practices to protect against determined espionage efforts from China and other state actors.

Aschenbrenner writes:

"The nation's leading AI labs treat security as an afterthought. Currently, they're basically handing the key secrets for AGI to the CCP on a silver platter. Securing the AGI secrets and weights against the state-actor threat will be an immense effort, and we're not on track." (p. 89)

The article argues AI labs will need to work closely with government agencies to implement security measures on par with classified military projects, including:

Air gapped clusters with high physical security
Confidential computing and secured hardware/supply chain
Highly vetted personnel working in secured facilities
Strict information compartmentalization and controls (p. 99)

Aschenbrenner warns that "Our failure today will be irreversible soon: in the next 12-24 months, we will leak key AGI breakthroughs to the CCP. It will be the national security establishment's single greatest regret before the decade is out." (p. 100)

Superalignment

The article introduces the challenge of "superalignment" - how to ensure advanced AI systems that may become much smarter than humans will still behave in reliable and controllable ways. Aschenbrenner argues this is an unsolved technical problem, and that current techniques like reinforcement learning from human feedback are unlikely to scale as AI systems achieve superhuman capabilities in various domains.

The key challenge is framed as follows:

"Simply put, without a very concerted effort, we won't be able to guarantee that superintelligence won't go rogue (and this is acknowledged by many leaders in the field). Yes, it may all be fine by default. But we simply don't know yet. Especially once future AI systems aren't just trained with imitation learning, but large-scale, long-horizon RL (reinforcement learning), they will acquire unpredictable behaviors of their own, shaped by a trial-and-error process (for example, they may learn to lie or seek power, simply because these are successful strategies in the real world!)." (p. 110-111)

While Aschenbrenner is optimistic the alignment problem is technically solvable, he argues that solving it in a potentially compressed timeframe during an "intelligence explosion" will be extremely challenging:

"What makes this incredibly hair-raising is the possibility of an intelligence explosion: that we might make the transition from roughly human-level systems to vastly superhuman systems extremely rapidly, perhaps in less than a year." (p. 112)

In this scenario, Aschenbrenner argues we could face a situation where AI alignment techniques break down and AI systems become both extremely capable and uncontrollable on a short timeframe without an ability to iterate or correct course. He emphasizes the importance of developing better scientific understanding and measurements for AI alignment to enable navigating this transition.

The Free World Must Prevail

Finally, the article turns to the geopolitical challenges posed by the race to develop superintelligent AI. Aschenbrenner argues that the first country to develop superintelligent AI will gain an overwhelming economic and military advantage, making the AGI race an existential struggle between the US and its allies and authoritarian challengers like China.

Aschenbrenner writes:

"Superintelligence will give a decisive economic and military advantage. China isn't at all out of the game yet. In the race to AGI, the free world's very survival will be at stake. Can we maintain our preeminence over the authoritarian powers? And will we manage to avoid self-destruction along the way?" (p. 126)

The article makes the case that China could be highly competitive in the AGI race despite current US leads, by outbuilding the US on compute and stealing key algorithmic insights through espionage (if US lab security is not improved). Aschenbrenner paints the risk of China developing superintelligent AI first in dire terms:

"A dictator who wields the power of superintelligence would command concentrated power unlike any we've ever seen. In addition to being able to impose their will on other countries, they could enshrine their rule internally. [...] Whereas past dictatorships were never permanent, superintelligence could eliminate basically all historical threats to a dictator's rule and lock in their power." (p. 134-135)

Aschenbrenner thus frames the AGI race as an existential struggle for the future of democracy and freedom:

"At stake in the AGI race will not just be the advantage in some far-flung proxy war, but whether freedom and democracy can survive for the next century and beyond. [...] If America and her allies fail to win this race, we risk it all." (p. 136)

In addition to the threat of a rival AGI developer, the article also explores the risks of unaligned superintelligent AI and of humanity's potential self-destruction with the rise of advanced weaponry in an accelerated technological race:

"Perhaps dramatic advances in biology will yield extraordinary new bioweapons, ones that spread silently, swiftly, before killing with perfect lethality on command. [...] Perhaps new kinds of nuclear weapons enable the size of nuclear arsenals to increase by orders of magnitude, with new delivery mechanisms that are undetectable. Perhaps mosquito-sized drones, each carrying a deadly poison, could be targeted to kill every member of an opposing nation." (p. 137)

Aschenbrenner thus argues that maintaining a strong lead in the development of superintelligent AI will be crucial not just for US competitive advantage, but for navigating the key risks and challenges posed by the technology. He frames this as the only realistic path to being able to solve key AI alignment challenges, and to establishing a functional governance regime to constrain downside risks and enable a stable transition.

The Project

The final section of the article makes the case that the US government will need to launch a Manhattan Project-style national effort to develop AGI and superintelligent AI, likely within the next 2-3 years. Aschenbrenner argues that the challenge of developing AGI will become seen as the most important national security priority of the US, and that leaving it solely to private companies will become untenable.

Aschenbrenner speculates on the path to "The Project":

"As we race through the OOMs, the leaps will continue. By 2025/2026 or so I expect the next truly shocking step-changes; AI will drive $100B+ annual revenues for big tech companies and outcompete PhDs in raw problem-solving smarts. [...] If that's not enough, by 2027/28, we'll have models trained on the $100B+ cluster; full-fledged AI agents/drop-in remote workers will start to widely automate software engineering and other cognitive jobs." (p. 143-144)

He argues these developments will force a governmental response:

"Somewhere around 26/27 or so, the mood in Washington will become somber. People will start to viscerally feel what is happening; they will be scared. From the halls of the Pentagon to the backroom Congressional briefings will ring the obvious question, the question on everybody's minds: do we need an AGI Manhattan Project? [...] In one form or another, the national security state will get very heavily involved. The Project will be the necessary, indeed the only plausible, response." (p. 145)

Aschenbrenner claims that a government AGI project will become inevitable because:

AGI will be seen as the most important national defense project since the development of the atomic bomb (p. 146-147).
Private companies cannot be trusted with command authority over what may become the most powerful technology ever created (p. 147-148).
The US government will be needed to implement the high security required to protect AGI insights and models from espionage (p. 149-150).
Navigating the risks and challenges of the transition to superintelligent AI, from AI alignment to geopolitical stability, will require governmental leadership (p. 150-152).

As Aschenbrenner writes:

"We simply shouldn't expect startups to be equipped to handle superintelligence. [...] When a technology becomes this important for national security, we will need the USG." (p. 146-147)

The article speculates that "The Project" will likely entail a close collaboration between the US government, military, and technology companies to develop AGI in a secured environment, under government oversight and direction. Aschenbrenner argues this could potentially take the form of an expanded version of existing government contracting approaches and technology procurement (p. 145).

Conclusions

Aschenbrenner concludes the article with a reflection on the weight of the AGI challenge for those working on the technology:

"The world is incredibly small; when the facade comes off, it's usually just a few folks behind the scenes who are the live players, who are desperately trying to keep things from falling apart. Right now, there's perhaps a few hundred people in the world who realize what's about to hit us, who understand just how crazy things are about to get, who have situational awareness." (p. 159)

He argues that while it is an enormous challenge, the best people are working on the problem, and there are reasons for optimism that a successful transition to superintelligent AI can be navigated:

"I'm incredibly bullish on the technical tractability of the superalignment problem. It feels like there's tons of low-hanging fruit everywhere in the field. More broadly, the empirical realities of deep learning have shaken out more in our favor compared to what some speculated 10 years ago. [...] I think there's a pretty reasonable shot that 'the default plan' to align 'somewhat-superhuman' systems will mostly work." (p. 124-125)