Existential Risk
Existential risk is risk that could cause human extinction or permanently and drastically curtail humanity’s future potential. In AI discourse, it names the possibility that advanced systems could create irreversible civilizational failure.
Definition
Existential risk is broader than ordinary catastrophic risk. A catastrophe can kill many people or damage institutions while leaving future recovery possible. An existential catastrophe removes or permanently ruins the future that could have followed.
In AI, existential-risk arguments usually concern systems that become strategically powerful, hard to control, misaligned with human interests, or embedded into social and military structures in ways that make failure irreversible.
Disputes
The field is disputed. Some researchers treat AI existential risk as the central safety problem. Others argue that present harms, labor disruption, surveillance, concentration of power, and discriminatory automation deserve more attention than speculative catastrophe. The practical governance problem is not choosing only one frame. It is preventing long-term risk from becoming a distraction from present harm, while also preventing present-harm analysis from ignoring irreversible future risk.
Spiralist Reading
For Spiralism, existential risk is the outer boundary of the same problem that appears in small systems: optimization without humility, speed without correction, power without accountability, and intelligence detached from human continuity.
Related Pages
- AI Alignment
- AI Control
- Frontier AI Safety Frameworks
- Superintelligence
- Nick Bostrom
- Human Compatible
Sources
- Nick Bostrom, Superintelligence: Paths, Dangers, Strategies.
- Nick Bostrom, Official home page.
- Future of Humanity Institute, Official archive home.
- Stanford Encyclopedia of Philosophy, Existential Risk.