Introduction

引言

English

Leopold Aschenbrenner, June 2024

You can see the future first in San Francisco.

Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.

The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace many college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be unleashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.

Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the willful blindness of "it's just predicting the next word". They see only hype and business-as-usual; at most they entertain another internet-scale technological change.

Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.

Let me tell you what we see.


Table of Contents

Each essay is meant to stand on its own, though I'd strongly encourage reading the series as a whole. For a pdf version of the full essay series, click here.

Introduction [this page] History is live in San Francisco.

I. From GPT-4 to AGI: Counting the OOMs AGI by 2027 is strikingly plausible. GPT-2 to GPT-4 took us from ~preschooler to ~smart high-schooler abilities in 4 years. Tracing trendlines in compute (~0.5 orders of magnitude or OOMs/year), algorithmic efficiencies (~0.5 OOMs/year), and "unhobbling" gains (from chatbot to agent), we should expect another preschooler-to-high-schooler-sized qualitative jump by 2027.

II. From AGI to Superintelligence: the Intelligence Explosion AI progress won't stop at human-level. Hundreds of millions of AGIs could automate AI research, compressing a decade of algorithmic progress (5+ OOMs) into ≤1 year. We would rapidly go from human-level to vastly superhuman AI systems. The power—and the peril—of superintelligence would be dramatic.

III. The Challenges

**IIIa. Racing to the Trillion-Dollar Cluster **The most extraordinary techno-capital acceleration has been set in motion. As AI revenue grows rapidly, many trillions of dollars will go into GPU, datacenter, and power buildout before the end of the decade. The industrial mobilization, including growing US electricity production by 10s of percent, will be intense.

**IIIb. Lock Down the Labs: Security for AGI **The nation's leading AI labs treat security as an afterthought. Currently, they're basically handing the key secrets for AGI to the CCP on a silver platter. Securing the AGI secrets and weights against the state-actor threat will be an immense effort, and we're not on track.

IIIc. Superalignment Reliably controlling AI systems much smarter than we are is an unsolved technical problem. And while it is a solvable problem, things could easily go off the rails during a rapid intelligence explosion. Managing this will be extremely tense; failure could easily be catastrophic.

**IIId. ****The Free World Must Prevail** Superintelligence will give a decisive economic and military advantage. China isn't at all out of the game yet. In the race to AGI, the free world's very survival will be at stake. Can we maintain our preeminence over the authoritarian powers? And will we manage to avoid self-destruction along the way?

**IV. The Project **As the race to AGI intensifies, the national security state will get involved. The USG will wake from its slumber, and by 27/28 we'll get some form of government AGI project. No startup can handle superintelligence. Somewhere in a SCIF, the endgame will be on.

V. Parting Thoughts What if we're right?

Next post in series: *** I. From GPT-4 to AGI: Counting the OOMs***


*While I used to work at OpenAI, all of this is based on publicly-available information, my own ideas, general field-knowledge, or SF-gossip. *

Thank you to Collin Burns, Avital Balwit, Carl Shulman, Jan Leike, Ilya Sutskever, Holden Karnofsky, Sholto Douglas, James Bradbury, Dwarkesh Patel, and many others for formative discussions. Thank you to many friends for feedback on earlier drafts. Thank you to Joe Ronan for help with graphics, and Nick Whitaker for publishing help.

Dedicated to Ilya Sutskever.

中文

利奥波德·阿申布伦纳(Leopold Aschenbrenner),2024 年 6 月

在旧金山,你可以最先看到未来。

在过去一年里,全城热议的话题已经从 100 亿美元($10B)的算力集群,转向 1000 亿美元($100B)的集群,再到万亿美元($1T)级的集群。每隔六个月,董事会的计划上就会多一个零。在幕后,各方正在激烈争抢本十年剩余时间里仍可获得的每一份电力合同、每一个还能采购到的变压器。美国大企业正蓄势待发,准备投入数万亿美元,发起一场久违的美国工业实力大动员。到本十年末,美国的发电量将增长百分之几十;从宾夕法尼亚的页岩气田到内华达的太阳能农场,数以亿计的 GPU 将嗡鸣运转。

AGI(通用人工智能)竞赛已经开始。我们正在制造能够思考与推理的机器。到 2025/26 年,这些机器将超越许多大学毕业生。到本十年末,它们将比你我都更聪明;届时我们将拥有名副其实的超级智能(superintelligence)。在这一过程中,半个世纪未见的国家安全力量将被释放,用不了多久,"项目"(The Project)就会启动。如果我们走运,将与 CCP 展开全面竞赛;如果不走运,那就是全面战争。

如今人人都在谈论 AI,但几乎没人对即将到来的冲击有丝毫预感。Nvidia 的分析师仍认为 2024 年可能已接近顶峰。主流评论员固守着"不过是预测下一个词而已"这种刻意的视而不见。他们只看到炒作与一切照旧;至多愿意认真考虑又一场互联网规模的技术变革。

用不了多久,世界就会醒来。但眼下,或许只有几百人——大多在旧金山和各大 AI 实验室——拥有情境意识(situational awareness)。凭着某种奇特命运的安排,我发现自己身处他们之中。几年前,这些人还被嘲笑为疯子——但他们相信趋势线,这让他们得以准确预测过去几年 AI 的进展。这些人对今后几年的判断是否同样正确,仍有待观察。但他们是极聪明的人——是我见过的最聪明的人——而且正是他们在建造这项技术。也许他们会成为历史上一个奇怪的脚注,也许他们会像西拉德(Szilard)、奥本海默(Oppenheimer)和泰勒(Teller)那样载入史册。如果他们对未来的预判哪怕接近正确,我们即将迎来一场狂野之旅。

让我来告诉你我们看到了什么。

目录

每篇文章都旨在独立成篇,不过我强烈建议把整个系列连起来阅读。如需整套系列文章的 PDF 版本,点击此处

引言 [本页] 历史正在旧金山鲜活地上演。

一、从 GPT-4 到 AGI:历数 OOM 到 2027 年实现 AGI 是惊人地合理的。从 GPT-2 到 GPT-4,我们在 4 年内从约"学龄前儿童"水平进步到约"聪明高中生"水平。沿着算力(每年约 0.5 个数量级,即 OOM(order of magnitude,数量级))、算法效率(每年约 0.5 个 OOM)以及"解除束缚"(unhobbling)带来的增益(从聊天机器人到智能体)这些趋势线推算,我们应当预期到 2027 年会出现又一次从学龄前儿童到高中生规模的质性飞跃。

二、从 AGI 到超级智能:智能爆炸 AI 的进步不会止步于人类水平。数以亿计的 AGI 可以自动化 AI 研究,把十年(5 个以上 OOM)的算法进步压缩到 ≤1 年内。我们将迅速从人类水平走向远超人类的 AI 系统。超级智能的力量——以及危险——都将是戏剧性的。

三、各项挑战

IIIa. 竞逐万亿美元集群 一场非同寻常的技术-资本加速已经启动。随着 AI 收入快速增长,本十年结束前将有数万亿美元投入 GPU、数据中心和电力建设。这场工业动员——包括让美国发电量增长百分之几十——将极为激烈。

IIIb. 封锁实验室:面向 AGI 的安全 这个国家顶尖的 AI 实验室把安全当作事后的补丁。目前,它们基本上是在把 AGI 的关键机密银盘奉送给 CCP。要抵御国家行为体的威胁、守住 AGI 的机密与权重(weights),将是一项艰巨的工作,而我们并未走在正轨上。

IIIc. 超级对齐(Superalignment) 可靠地控制远比我们聪明的 AI 系统,仍是一个未解决的技术问题。尽管它是个可以解决的问题,但在一次快速的智能爆炸中,局面很容易失控。管理这一过程将极其紧张;失败则可能轻易酿成灾难。

IIId. 自由世界必须获胜 超级智能将带来决定性的经济与军事优势。中国还远未出局。在通往 AGI 的竞赛中,自由世界的生死存亡系于此。我们能否保持对威权势力的领先地位?我们能否避免在途中自我毁灭?

四、项目(The Project) 随着 AGI 竞赛愈演愈烈,国家安全机器将会介入。美国政府(USG)将从沉睡中醒来,到 27/28 年,我们将看到某种形式的政府 AGI 项目。没有哪家初创公司能驾驭超级智能。在某个 SCIF(敏感隔离信息设施)里,终局即将上演。

五、临别感言 如果我们是对的,会怎样?

系列下一篇:*** 一、从 GPT-4 到 AGI:历数 OOM***

虽然我曾就职于 OpenAI,但以上一切都基于公开可得的信息、我自己的想法、领域通识,或旧金山(SF)的流言。

感谢 Collin Burns、Avital Balwit、Carl Shulman、Jan Leike、Ilya Sutskever、Holden Karnofsky、Sholto Douglas、James Bradbury、Dwarkesh Patel 以及其他许多人富有启发性的讨论。感谢许多朋友对早期草稿的反馈。感谢 Joe Ronan 在图表方面提供的帮助,以及 Nick Whitaker 在出版方面提供的帮助。

谨以此文献给 Ilya Sutskever。