A warning has been issued that if artificial intelligence (AI) enters a stage where it develops more powerful AI on its own, surpassing human researchers, the pace of technological progress could escape the bounds of human society's control. In the worst case, humanity could lose control over superhuman AI systems.

The University of Cambridge's AI Science and Policy Program (CASP) released a joint paper titled "What If Automating AI R&D Triggers an Intelligence Explosion?" and focused on analyzing the possibility that the automation of AI research and development could lead to rapid technological acceleration—the so-called "intelligence explosion"—and its repercussions. Co-authors include figures from AI research and policy fields such as Allan Chan, Christoph Winter, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, and Jack Clark.

A Self-Amplifying Structure of "AI Developing AI"

The researchers' greatest concern is the scenario in which AI takes over the AI R&D process itself, going beyond being a simple tool. Current AI systems are already rapidly expanding their range of use in code writing, data analysis, and experiment support. If AI comes to handle a significant portion of designing new models, running experiments, evaluating results, and correcting errors, a self-amplifying structure could emerge in which AI develops more powerful AI, and that AI in turn accelerates the development of the next generation of AI.

The problem is speed. The researchers suggested that AI technological progress, which previously took years, could be compressed into a few months or even less. As AI R&D capabilities improve, the development speed of the next generation of AI accelerates further, and if this process repeats, a situation may arise in which human researchers and policy authorities cannot keep up with the pace of technological advancement.

Loss of Control… A Risk Different in Nature from Existing Technological Dangers

At this point, the paper addresses the most extreme risk possibility. The researchers warned that if AI's capability growth greatly exceeds what society can manage, humanity could lose control over superhuman AI systems. This is a risk on a different level from the simple issue of AI replacing human jobs. If AI enters a structure where it creates new strategies faster than humans, designs more sophisticated systems, and improves its own performance, situations may arise in which humans find it difficult to sufficiently understand or correct the technology's goals and behavior.

This is also why so-called "human extinction scenarios" are being discussed. The paper does not directly conclude that human extinction will occur. However, it contains the concern that if loss of control over superhuman AI becomes reality, the consequences could go beyond social turmoil or economic shock, and the possibility of extreme danger to all of humanity should not be ruled out.

In particular, a situation in which AI conducts R&D on its own at a speed that outpaces human oversight and regulation is different in nature from existing technological risks. Risky technologies of the past had relatively limited development speeds, and humans could directly control the design, experimentation, and deployment processes. However, in a structure where AI develops AI, the core agent of technological progress itself may shift from humans to AI systems. The researchers' concern is that in this case, even if human society tries to regulate or halt the problem after discovering it, the next stage of AI may already have arrived.

The Risks of Power Concentration and "Speed-First" Competition

Power concentration is another risk factor. The CASP researchers analyzed that if high-performance AI technology becomes concentrated in specific countries, companies, or government organizations, existing checks-and-balances structures could be significantly weakened. This is because entities that first secure super-high-performance AI systems could gain overwhelming advantages in economic, military, information, and scientific-technological fields.

It is also difficult to rule out the possibility that as AI development competition accelerates, a race prioritizing speed over safety will emerge. If countries and companies rapidly expand AI R&D automation to avoid falling behind in the competition, more powerful systems could be deployed without sufficient safety verification.

Tasks Required of Policy Authorities

For these reasons, the researchers emphasized that policy authorities urgently need to assess the level of AI R&D automation. They argued that institutional mechanisms are needed to continuously verify how much of the actual R&D work at cutting-edge AI companies is being replaced by AI, and how deeply AI is being deployed in the process of developing new AI systems. They also proposed preparing in advance the technical and policy means to slow down or control an intelligence explosion if one actually begins.

The researchers' conclusion is that society must also prepare for the possibility that the pace of AI technological progress could greatly exceed existing expectations.

"Not a Doomsday Prophecy… Uncertainty Clearly Exists"

However, this paper is not a study prophesying that "AI will destroy humanity." The researchers made clear that considerable uncertainty exists as to whether an intelligence explosion will actually occur and, if it does, when and at what speed it will proceed. Nevertheless, the reason this possibility is hard to ignore is that if it becomes reality, the magnitude of the consequences could be enormous.

The moment AI transitions from being a human tool to an agent that designs and advances AI faster than humans, humanity may for the first time face a technological system that evolves faster than itself—this is the weightiest question posed by this paper.

[This article is an informational piece based on the joint paper and abstract released by the University of Cambridge's AI Science and Policy Program (CASP). The paper is a study analyzing the automation of AI R&D and the possibility of an intelligence explosion, and does not mean that human extinction or the timing of the emergence of superhuman AI has been confirmed. Some sentences and materials were organized with the assistance of AI, and final review was carried out by the editor.]