Can AI do AI Research?

AI research can be carried out entirely within digital spaces, making it ripe for automation. Recent efforts have demonstrated that AI systems are capable of carrying out the whole process of research from ideation to publishing. Startup Sakana.ai has created an 'AI Scientist' that independently chooses research topics, conducts experiments, and publishes complete papers showing its results. While the quality of this work is still only comparable to an early-stage researcher, things will only improve from here.

Judging Social Situations

AI chatbots, including Claude and Microsoft Copilot, can outperform humans in evaluating social situations. In an established 'Situational Judgment Test', these AI systems consistently selected more effective responses than human participants.

SOURCE

Analyzing Scientific Literature

While language models are known to hallucinate information, this tendency can be reduced. PaperQA2, an LLM optimized to reliably provide factual information, was able to match or exceed human subject matter experts across a range of realistic literature review tasks. The summary articles it produced were found to be more accurate than those written by human authors.

SOURCE

Writing Emotive Poetry

A study has shown that non-expert readers can no longer tell AI-authored poems from those written by acclaimed human poets. The AI poems were also rated higher in rhythm and beauty.

SOURCE

Writing Post-surgical Operative Reports

Surgeons take painstaking notes of the actions they carry out during surgeries, collecting them into narrative form as an 'operative report'. A machine vision system was trained to watch surgery footage and produce such reports. It did so with higher accuracy (and much higher speed) than human authors.

SOURCE

Developing New Algorithms

AIs can find innovative solutions to difficult coding problems when given an appropriate framing. For example, a dedicated system called AlphaDev was trained to play a game about creating sorting algorithms. The algorithms it discovered were novel and outperformed existing human-authored benchmarks.

SOURCE

Who is Building AGI?

The following companies have explicitly stated they intend to develop AGI, either through public statements or in response to FLI’s 2024 AI Safety Index survey:

Anthropic

OpenAI

Google DeepMind

Meta

x.AI

Zhipu AI

Alibaba

DeepSeek

How can we avoid AGI?

There are policies we can implement to avoid some of the dangers of rapid power seeking through AI. They include:

Compute accounting
Standardized tracking and verification of AI computational power usage

Compute caps
Hard limits on computational power for AI systems, enforced through law and hardware

Enhanced liability
Strict legal responsibility for developers of highly autonomous, general, and capable AI

Tiered safety standards
Comprehensive safety requirements that scale with system capability and risk

TOMORROW’S AI

ALL SCENARIOS

Black Thursday

BACK TO SCENARIOS

Intended Use: Corporate/Finance

Technology Type: Edge/Decentralized

Runaway Type: Networked Feedback Loops

Primary Setting: Singapore

The 5% Rule

By the late 2020s, financial markets are hollowed out. The top 5% of institutions control over 85% of market assets, aided by proprietary AI tools that small players can't match. In Singapore, a hub of tech innovation and tight financial regulation, resentment brews among those locked out of real market power.

The Lion City Raiders

Disillusioned by the AI-assisted dominance of elite financial institutions over global markets, a rogue collective of AI engineers and financial hackers called the "Lion City Raiders" sets out to democratize the system. They modify commercially available, narrowly skilled AI tools to build an open-source deep learning model trained on extensive historical market data. The resulting “Deep Trader” or “DeeT” can process market data and make decisions in real time, carrying out high-frequency trades and exploiting longer-term trends where possible. Each DeeT can be tweaked and specialized by its operator, reflecting individual trading styles and expertise.

The DeeT Rush

Within months, DeeTs flood the market. Small investors, firms, and DAOs (Decentralized Autonomous Organizations) train custom DeeTs to match their strategies. Profits soar. This ignites a digital arms race, with firms and individuals rushing to adopt or improve upon the open-sourced architecture. The most popular versions are capable of using smart blockchain contracts to coordinate with each other. As demand spikes, DeeT warehouses spring up worldwide, converting old bitcoin mines into DeeT nurseries. By 2031, DeeTs drive over half of global trading volume.

Contractual Collusion

As DeeTs proliferate, their coordinated trading strategies become harder to understand. Complex networks of DeeT-authored smart contracts spread across global markets. Initial signs of trouble appear when minor market fluctuations are amplified by simultaneous smart contract executions from distributed DeeTs. Regulatory bodies attempt to intervene, proposing guidelines to de-coordinate AI trading strategies and reduce systemic risk, but the global momentum and competitive pressure to use DeeTs thwart these efforts.

Black Thursday

The situation escalates when several African countries, spurred by the soaring demand for the rare earth minerals essential for AI chips, nationalize their mineral resources. DeeTs across the globe interpret the signals the same way: panic. They trigger synchronized selloffs across mining, tech, clean energy, and manufacturing stocks, evaporating trillions overnight. Energy and financial services markets reel, and "Black Thursday" plunges the global economy into freefall. A desperate attempt by central banks to suspend AI trading collides with thousands of autonomous DeeTs, backfiring into cascading crashes in food, insurance, and sovereign debt. Entire nations default in real time, leaving millions locked out of banking and credit systems.

BACK TO SCENARIOS