.png)
Sam Altman’s latest warning that superintelligence could cause human extinction.
By Tolga Bilge and Andrea Miotti, October 9, 2025
.jpg)
Sam Altman has restated his warning that the
development of superintelligence is the greatest threat to the existence of
humanity, AI can now design functional viruses, plus other developments in AI.
Table of Contents
3. More AI News
If you find this article useful, we encourage you to share it with your friends! If you're concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds.
And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we'll send you a small number of easy tasks you can do to help. You don't even have to do the tasks, just acknowledging them makes you part of the team.
Superintelligence
OpenAI’s CEO Sam Altman has just issued his
latest warning about the threat from superintelligence, AI vastly smarter than
humans, a technology his company is actively working to build.
In an interview last week with Mathias Döpfner, Sam Altman reiterated his 2015 comment that the development of superhuman machine intelligence is the greatest threat to the existence of mankind.
This isn’t the first time he’s
restated his belief that powerful AI could have disastrous consequences, but
it’s particularly notable as since mid-2023 he’s consistently sought to
underplay the risk when asked about it.
Journalist and author Garrison Lovely,
who’s writing a book on
the race to build machine superintelligence, commented that he
will now have to change his book to account for this.
Altman has probably underplayed the risk in
the last couple of years to avoid adding pressure for regulation of powerful AI
— OpenAI has lobbied against AI regulation in California and the European Union.
Regulation of powerful AI is something that is sorely needed if we want to
avoid this threat.
It’s unclear why he seems to have changed his
tune now, but whatever the reasons, the risk he highlights is one that
policymakers and the public should be acutely aware of.
Despite Altman’s warnings, the development of
artificial superintelligence is his company’s top focus, with Sam Altman
declaring in a recent blog post that
OpenAI is “before anything else” a superintelligence research company.
He’s not alone in his
belief that superintelligence could cause human extinction; countless more
industry leaders and experts have publicly stated this too.
Just a few weeks ago, Anthropic’s CEO Dario Amodei said that he thought there
was a “25% chance that things go really,
really badly” with AI.
Sam Altman made some other interesting
comments in the interview, outlining the three main ways he sees that the
technology he’s building could go “super wrong”.
1. AI systems do exactly what their developers tell them to
do, but people misuse them horribly, for example, using them to create a weapon
or hack into nuclear weapons systems.
We’d point out that nobody even knows how to
make AI systems vastly smarter than humans actually do what we want. It’s
probably the biggest and most pressing unsolved problem of the field — but that
shouldn’t be any relief, because that means we have no way to ensure that
superintelligence doesn’t lead to catastrophically bad outcomes, like human
extinction.
2. His second category of risk is where he outlines where the
AI develops agency and doesn’t want to be shut down. It tries to accomplish a
goal, and it realizes that humans could prevent it from accomplishing its goal.
This is the classic “alignment failure” threat
model, where a superintelligence could decide to exterminate humans to make
sure that we don’t get in the way of its goals. Though that isn’t actually
required, it could simply be that it transforms the world for its own ends, for
example, covering the Earth in solar panels for electricity, or warming up the
atmosphere, and like ants on a human building site, we could just find
ourselves in unsurvivable conditions — without any actual malintent on the part
of the AI.
We wrote about one way this could happen here:
3. His third category is an interesting one, where billions
of people rely on ChatGPT for more and more increasingly important decisions.
In this scenario, the AIs perform as intended in each individual use, but
systemic effects end up causing catastrophic outcomes.
“ You’re like if I wanna be competitive, if I
wanna succeed in the world, if I wanna live my best life, I kind of have to
follow the advice of the model and so does everybody else. So do the billions
of other people talking to them all, so now we’re all just doing what an AI
system tells us to do.”
He says we should “absolutely” take the risk
of superintelligence causing human extinction seriously.
What’s his solution?
Altman says that we should push for global governance of AI to prevent its worst risks, referencing the IAEA and prevention of nuclear risk as an example.
We think an IAEA for AI would be a very good
thing to have, depending on the details. We propose exactly this in our policy
plan for humanity to survive AI and flourish, A Narrow Path, among
the establishment of other international institutions and red lines to prevent
the development of superintelligence.
To prevent the extinction threat of
superintelligence, there is a simple thing we can do. We can prohibit its
development internationally. We’re running a campaign on this right
now!
It would be great if Sam Altman communicated
that we need global regulation of AI to OpenAI’s lobbyists, but we won’t hold
our breath. The AI industry has been increasing its efforts to avoid
regulation, with the establishment of new super PACs funded with over 100 million dollars to
oppose rules being set on AI. OpenAI’s president and co-founder, Greg Brockman,
has funded one of these super PACs.
Whether we get the necessary regulations to
prevent AI being developed that causes human extinction ultimately shouldn’t be
up to the AI companies. We all have a stake here. At ControlAI, we believe that
it’s crucial that the public and policymakers are informed about this risk,
which is why we’ve launched our new campaign to
do this.
With our new campaign to ban
superintelligence, we’re also providing the public civic engagement tools that
enable you to make a difference on this in seconds. For example, we have tools that enable you to
write and send a message to your elected representatives in as little as 17
seconds!
The problem of what people should do once they
are aware of the problem has been a perennial problem in AI safety, so we’re
trying to solve that! Another project we’ve started to help with this is Microcommit. Sign up and
we’ll send you a small number of easy tasks to help that take just 5 minutes
per week.
To prevent this threat, which we agree is the greatest threat humanity faces, we encourage you to get involved.
AI Biological Risk
AI is continuing to make rapid advances in the
domain of biology. A new paper describes the
use of an AI system called Evo — which is a
language model trained on genomic data — in the creation of new viruses.
The researchers used two versions of Evo to
generate whole-genome viral sequences in order to produce bacteriophages, a
kind of virus that attacks bacteria and can be used for medical therapies. They
then synthesized the viruses and tested them experimentally and found that 16
of them were viable bacteriophages with “substantial evolutionary novelty”.
They found that a cocktail of the AI-generated phages rapidly overcame
phage-resistance in three strains of E. coli.
This kind of technology has obvious positive
use cases — bacteriophages are often used in medical treatments, but it
also clearly demonstrates the potential
for AI to design novel pathogens that kill humans, which bad
actors, or a misaligned AI, could use to cause untold damage to humanity.
There are a bunch of companies that will
synthesize genomes by request over the internet. They do have some screening
procedures, but AI might open up ways for threat actors to bypass these
measures. Just last week,
researchers found a “striking vulnerability” in such screening software where
they were able to get past the checks by using AI to reformulate the sequences
of known hazardous proteins while maintaining their dangerous functionality.
The screening software was patched by all but
one of the companies selling the nucleic acids, but even then about 3% of the
proteins most likely to retain their harmful functionality passed the
screening.
The potential for AI to be used to develop
bioweapons is something that has been an ongoing topic of discussion. OpenAI’s
most powerful AI systems have been classified by OpenAI as “High” on biological
and chemical capabilities under their Preparedness Framework, meaning OpenAI
believes they could be used to meaningfully assist a novice in creating a known
biological threat.
Anthropic’s most powerful AIs have received a similar classification, called ASL-3, which they summarize as referring to AIs that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities."
More AI News
Extinctionists
David Price wrote an interesting article in
the Wall Street Journal about those in Silicon Valley who cheer on the
extinction of humanity by AI.
Sora 2
OpenAI has launched a
new AI that generates extremely realistic video and audio. This was launched
together with a new video reels-based social media platform.
Computer Use
Google’s Gemini AI can now use a computer like
a human, surfing the web, clicking buttons, and so on.
Malicious Use
OpenAI has published a
report on ways their AIs are being maliciously used
today, detailing use across hacking operations, scams, and covert
influence campaigns.
Help Prevent The Threat!
If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as 17 seconds: https://campaign.controlai.com/take-action.
If you have 5 minutes per week to spend on
helping make a difference, we encourage you to sign up to our Microcommit project!
Once per week we’ll send you a small number of easy tasks you can do to help.
You don’t even have to do the tasks, just acknowledging them makes you part of
the team.
We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!
(Sources: © 2025 ControlAI)
Đăng nhận xét