Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Commentary Challenges the Engineering Basis of Superintelligence

AI summary

A commentary says Anthropic researcher Jacob Coxon resigned after warning that Anthropic and OpenAI were racing toward self-improving superintelligence and “gambling with our lives.” Anthropic’s head of alignment science, Evan Hubinger, said Coxon was correct that AI could kill all humans, while CEO Dario Amodei told CNN that he agreed with Coxon more than he disagreed with him. The commentary argues that recursive self-improvement remains speculative and that pursuing it could produce systems that are harder for humans to understand and control, even without an intelligence explosion.

In-depth insight

The commentary questions the engineering feasibility of superintelligence and warns that pursuing recursive self-improvement may cause real harm even if it never produces an intelligence explosion.

Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

These concerns stem from a belief that has circulated in futurist circles for many decades: Once engineers design an AI system capable of improving itself, it will trigger an accelerating cycle of autonomous upgrades known as recursive self-improvement (RSI). The result will be artificial superintelligence (ASI), a system with cognitive capabilities so vastly exceeding those of humans that it will easily evade our control and perhaps even wipe out our species. Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.

#Anthropic#OpenAI#Jacob Coxon#Evan Hubinger#Dario Amodei#Recursive self-improvement#Artificial superintelligence

Original · 13 min readRead the original
Group chat

No comments yet. Start the conversation.