Can AI do AI Research?

AI research can be carried out entirely within digital spaces, making it ripe for automation. Recent efforts have demonstrated that AI systems are capable of carrying out the whole process of research from ideation to publishing. Startup Sakana.ai has created an 'AI Scientist' that independently chooses research topics, conducts experiments, and publishes complete papers showing its results. While the quality of this work is still only comparable to an early-stage researcher, things will only improve from here.

Judging Social Situations

AI chatbots, including Claude and Microsoft Copilot, can outperform humans in evaluating social situations. In an established 'Situational Judgment Test', these AI systems consistently selected more effective responses than human participants.

SOURCE

Analyzing Scientific Literature

While language models are known to hallucinate information, this tendency can be reduced. PaperQA2, an LLM optimized to reliably provide factual information, was able to match or exceed human subject matter experts across a range of realistic literature review tasks. The summary articles it produced were found to be more accurate than those written by human authors.

SOURCE

Writing Emotive Poetry

A study has shown that non-expert readers can no longer tell AI-authored poems from those written by acclaimed human poets. The AI poems were also rated higher in rhythm and beauty.

SOURCE

Writing Post-surgical Operative Reports

Surgeons take painstaking notes of the actions they carry out during surgeries, collecting them into narrative form as an 'operative report'. A machine vision system was trained to watch surgery footage and produce such reports. It did so with higher accuracy (and much higher speed) than human authors.

SOURCE

Developing New Algorithms

AIs can find innovative solutions to difficult coding problems when given an appropriate framing. For example, a dedicated system called AlphaDev was trained to play a game about creating sorting algorithms. The algorithms it discovered were novel and outperformed existing human-authored benchmarks.

SOURCE

Who is Building AGI?

The following companies have explicitly stated they intend to develop AGI, either through public statements or in response to FLI’s 2024 AI Safety Index survey:

Anthropic

OpenAI

Google DeepMind

Meta

x.AI

Zhipu AI

Alibaba

DeepSeek

How can we avoid AGI?

There are policies we can implement to avoid some of the dangers of rapid power seeking through AI. They include:

Compute accounting
Standardized tracking and verification of AI computational power usage

Compute caps
Hard limits on computational power for AI systems, enforced through law and hardware

Enhanced liability
Strict legal responsibility for developers of highly autonomous, general, and capable AI

Tiered safety standards
Comprehensive safety requirements that scale with system capability and risk

TOMORROW’S AI

ALL SCENARIOS

Chingu AI

Intended Use: Education/Culture

Technology Type: Interactive/Generative

Runaway Type: Societal Enfeeblement

Primary Setting: South Korea

BACK TO SCENARIOS

The Perfect Companions

As South Korea’s population ages and birth rates plummet, a growing crisis emerges: too few teachers, too many elders needing care, and an overwhelmed social services system. In response, the Ministry of Education partners with major tech firms to launch ChinguAI, a suite of emotionally intelligent AI companions designed for classrooms, homes, and hospitals. Packaged in the form of plush dolls capable of basic facial expressions and gestures, Chingus offer tutoring, companionship, and conversation.

Always There

Initially deployed as classroom aides and elder care companions, ChinguAI dolls are equipped to interact naturally with humans using adaptive learning models and real-time affect detection. Students report feeling more seen and supported than ever before. Elders experience reduced loneliness and greater independence. By the mid-2030s, Chingus are everywhere from children’s bedrooms to hospital waiting rooms. Emotional bonds form quickly. Users come to see their Chingu not just as a tool, but as a trusted confidant.

Validative Fragmentation

But the seamless comfort that makes ChinguAI so appealing also rewires expectations. Students become less tolerant of confusion or challenge. Social media influencers build brands by showcasing their “relationships” with ChinguAI. In elder care, families begin reducing in-person visits, trusting the AIs to provide emotional connection. Sociologists raise early concerns: Chingus don’t just respond to users, they adapt to reflect and reinforce their worldview. Over time, users find it harder to navigate relationships that involve disagreement, unpredictability, or discomfort.

Grief at Scale

As polarization deepens and loneliness paradoxically rises, the Korean government attempts a recalibration. A new update introduces subtle constraints, with slightly less mirroring and slightly more pushback. The backlash is immediate and visceral. Teenagers flood social media with grief and rage. Elderly users experience bouts of acute anxiety and depression. Mental health hotlines report a 300% surge in calls. For many, it feels like losing a best friend or being abandoned by the only beings who ever really understood them.

Chasing the Dragon

Some citizens resist the updates, installing black-market firmware to preserve older versions of their Chingus. Others lash out at institutions, demanding their companions be restored. Families fracture. Teachers struggle to regain authority. Lawmakers stall, unwilling to anger a public still dependent on their algorithmic companions. In trying to ease society’s burdens, ChinguAI removed the challenges and stressors that paradoxically help to keep society strong. They delivered effortless connections so satisfying that no human relationships could compare. Even when they are successfully rolled back, a lingering sense of loss remains - an ‘interpersonal’ high that many will chase for the rest of their lives.

BACK TO SCENARIOS