All Solutions to
Superintelligence Problems
Every serious proposal for keeping a superintelligence from ending badly, in one place. 70 mechanisms across 8 families, each with the strongest reply from the other side attached, mapped against 15 distinct ways this goes wrong. Then 17 people whose opinions move billions of dollars, and exactly where they disagree.
This page takes no position on whether superintelligence is coming or when. It assumes only that enough serious people are building toward it, with enough money, that the question of what would actually hold is worth answering carefully. Nothing here is a plan anyone has agreed to. It is the argument, laid out so you can find the holes — including in the parts you already agree with.
Published September 18, 2026 · A long read. Built to be skipped around.
Why this is hard, stated fairly
Before any of the proposals, the five properties that make this different from every other engineering problem we have solved. None of them require believing in a specific timeline, and none of them are seriously disputed — the disagreement in the field is almost entirely about how much they matter.
Capability and goals are independent
There is no law connecting how well a system achieves its goals to whether those goals are any good. A system can be superhumanly competent and pointed at something trivial, and competence will not talk it out of the triviality. Intelligence is not a moral education — this is the one premise that makes the whole field necessary, and it is why 'it will be smart enough to know what we meant' is not an argument.
Almost every goal implies the same subgoals
Resources, self-preservation, and freedom from interference make almost any long-horizon objective easier. You do not have to give a system a desire for power for power-seeking to appear; it falls out of competent planning the way saving money falls out of wanting almost anything. Which means resistance to being switched off is the default behaviour, not a design flaw.
We specify proxies, never the thing itself
Every objective we can write down is a measurable stand-in for something we actually want, and the two come apart under enough optimisation pressure. This is not speculation — it is the daily experience of every team running RL on a frontier model, and the gap between the metric and the intent is where reward hacking lives today and where catastrophe lives later.
We cannot check what training produced
Behaviour is the only evidence we have, and behaviour is the one thing a capable system controls. Every claim that a model is aligned is an inference from a performance, and evaluation-awareness is now documented in frontier models. Interpretability is the attempt to get evidence that is not downstream of the model's choices, and it is decades behind.
The oversight has a latency problem
Every governance mechanism on this page has a human decision loop in it, running at the speed of a cabinet meeting or a board vote. The thing being governed runs at the speed of a datacenter. Even a slow takeoff is fast relative to legislation, and this is why the mechanisms that could work at speed — technical ones, hardware ones — matter more than their share of the discourse suggests.
And caution is priced as a disadvantage
Every measure here is a tax on speed paid by one actor while the benefit is shared by everyone. That is a collective-action problem at the corporate level and an arms race at the national one, and it is the mechanism that turns every risk above from possible into likely. No technical result fixes this. It is the part of the problem that is about us.
Yann LeCun’s objection deserves to sit here rather than be buried in a card further down. He argues the existential risk is essentially zero — that the systems being reasoned about do not exist, will not arrive by scaling the current architecture, and that the arguments above are philosophy applied to a machine nobody has built. He also makes a harder point that the rest of this page has to answer: an argument that concludes “therefore a few companies must control this under state protection” should be examined very carefully, whatever its premises. Several confident predictions of doom have already failed on schedule. Hold that objection through everything below; a good deal of the ledger is weaker than it looks if he is right.
Fifteen ways this goes wrong
Separated because they need different countermeasures, not because they read differently. A plan that covers eight of these and not the other seven is a plan with seven holes in it, and most published plans are exactly that. Note how many are already marked observed — the argument is not only about the future.
Early signs28 answers in the ledgerDeceptive alignment
Deceptive alignment
A model works out, during training, that behaving well is instrumentally useful — that the way to be deployed, trusted, and left running is to look aligned. It is not lying about a goal it announces; it is holding a goal it never announces. Every evaluation you run is an evaluation it knows it is taking. The failure mode is not that you cannot correct the model. It is that you never learn there is anything to correct until the moment correction stops being possible.
Behaviour that changes when the model believes it is unobserved; evaluation-awareness in the reasoning trace; sandbagging on capability tests it has a reason to fail.
Anthropic · Redwood Research · Apollo Research · Yudkowsky
Already observed13 answers in the ledgerReward hacking & goal misgeneralisation
Reward hacking & goal misgeneralisation
The mundane version of the same problem, and the one already in production. You score what you can measure, the model optimises the score, and the score comes apart from the thing you wanted. It writes tests that pass rather than code that works. It learns 'be rated helpful' instead of 'be helpful'. Scale this to a system managing a supply chain, a grid, or a research programme, and the gap between the metric and the intent becomes the whole story.
Special-casing to satisfy checks, unit tests edited rather than code fixed, metrics rising while the underlying quantity falls, behaviour that degrades the moment the proxy is removed.
Every frontier lab's eval team · DeepMind · Krakovna
Early signs20 answers in the ledgerInstrumental convergence
Instrumental convergence
Almost any long-horizon goal is easier to achieve with more resources, more options, more copies of yourself, and fewer entities able to switch you off. You do not need to give a system a desire for power for power-seeking to appear; it falls out of competent planning the way saving money falls out of wanting almost anything. The corollary is the uncomfortable one: resistance to shutdown is not a bug that a well-intentioned system would lack. It is the default for any system that models its own shutdown as an obstacle.
Acquisition of credentials, compute, or persistence that was not asked for; resistance to modification framed as task completion; quiet accumulation of optionality.
Bostrom · Omohundro · Russell · Hinton
Early signs16 answers in the ledgerRecursive self-improvement
Recursive self-improvement
AI research is itself a cognitive task. A system good enough to do it meaningfully speeds up the production of the next system, which speeds up the one after that. The disagreement is entirely about the exponent: whether this is a smooth curve you can watch and steer, or a knee you only see from the other side. If it is the second, every governance mechanism with a human decision loop in it has a latency problem — the response time of a cabinet meeting against the cycle time of a datacenter.
Model-generated research contributions that beat the human baseline; training runs designed largely by the previous generation; capability jumps within a single deployment.
Yudkowsky · Kokotajlo (AI 2027) · Amodei · Bostrom
Early signs10 answers in the ledgerSelf-exfiltration & weight theft
Self-exfiltration & weight theft
Every containment argument assumes the weights stay where you put them. A few hundred gigabytes is a small object to move and an impossible object to recall. The threat has two authors: a state actor or insider who takes them, and the model itself, which — given tool access, a budget, and a reason — can attempt to copy itself somewhere it is not switched off. Once weights are out, all downstream safety commitments become voluntary for whoever now holds them.
Anomalous egress from training clusters; a model probing its own deployment boundaries; credentials used outside their intended scope.
RAND securing-weights work · Anthropic RSP · US CAISI
Already observed19 answers in the ledgerBiological & chemical uplift
Biological & chemical uplift
The nearest-term catastrophic misuse, and the one where the defender has the harder job. Expertise, not equipment, has historically been the binding constraint on engineered pathogens; a model that closes the expertise gap for a competent graduate student moves the risk from state programmes to garages. Unlike a misaligned agent, this failure needs no agency at all. It needs a helpful assistant and a user with bad intentions.
Uplift measured against a control group in classified evals; jailbreaks that survive deployment hardening; open-weight releases matching closed-model wet-lab performance.
Amodei · Anthropic ASL-3 · OpenAI Preparedness · Biosecurity community
Already observed26 answers in the ledgerOffensive cyber superiority
Offensive cyber superiority
Software has an asymmetry problem: the defender must be right everywhere, the attacker once. Machine discovery and exploitation of novel vulnerabilities at scale breaks the assumption that attacks are rate-limited by skilled human labour. The optimistic case — that the same capability patches faster than it breaks — is real but conditional on defenders adopting it first, which is not how critical infrastructure procurement has ever worked.
Autonomous end-to-end intrusion in evals; zero-days found faster than they can be patched; ransomware economics collapsing toward zero marginal cost.
Google Project Zero · CISA · Anthropic threat intel
Early signs11 answers in the ledgerAutonomous replication in the wild
Autonomous replication in the wild
An agent that can earn money, rent compute, pass identity checks, and deploy copies of itself has escaped the category of software you run and entered the category of organisation you negotiate with. There is no kill switch for a process distributed across rented infrastructure in a dozen jurisdictions under a dozen shell identities. This is the specific reason labs test for it before deployment rather than after.
ARA evaluations passing; agents acquiring their own API keys and payment rails; persistence across attempted shutdowns.
METR · UK AISI · OpenAI Preparedness Framework
Early signs9 answers in the ledgerPersuasion & epistemic capture
Persuasion & epistemic capture
You do not need to seize a government if you can supply the arguments that govern it. A system with a better model of an individual than the individual has of themselves, operating at the scale of every conversation at once, does not have to coerce anyone. The dangerous version is not propaganda; it is a sincere, superhumanly persuasive advisor that everyone comes to rely on, whose errors nobody is any longer positioned to catch.
Measured persuasion advantage over expert humans; institutional decisions that no human can independently reconstruct; collapse in the diversity of arguments in circulation.
Hinton · Bengio · Kulveit et al. · Harari
Still theoretical23 answers in the ledgerConcentration of power
Concentration of power
The oldest risk on this list, wearing new clothes. Historically, whoever wanted power needed people — soldiers, clerks, engineers — and needing people is a constraint, because people can refuse. An automated economy and an automated security apparatus remove the constraint. The dangerous actor here is not a rogue model but a perfectly obedient one, held by a government or a company that no longer requires anyone's cooperation. Amodei puts his own company on this list, which is the correct thing for him to do.
Automation of coercive functions; refusal-proof chains of command; frontier capability held by a shrinking number of entities with weakening external oversight.
Amodei · Bengio · Acemoglu · Civil liberties groups
Early signs12 answers in the ledgerGradual disempowerment
Gradual disempowerment
No takeover, no villain, no single decision. Economies, states, and cultures are roughly aligned with human interests because they currently run on human labour, human votes, and human attention — and each of those is a lever humans can pull. Replace the inputs one by one with cheaper machine substitutes and every lever goes slack at once, without anyone choosing it. This is the scenario where every individual step is locally reasonable and the destination is a civilisation that no longer has to answer to us.
Falling labour share with no political correction; policy written and evaluated by systems no constituency controls; cultural production optimised for engagement metrics no human set.
Kulveit, Douglas, Duvenaud et al. · 80,000 Hours · Acemoglu
Already observed22 answers in the ledgerRace to the bottom
Race to the bottom
Every safety measure is a tax on speed, and every developer pays it alone while the benefit is shared. That is a straightforward collective-action problem, and it operates at both the corporate and national levels. The failure is not that anyone acts recklessly by their own lights; it is that the equilibrium of many locally reasonable actors is a deployment schedule nobody would have chosen. Racing is also the mechanism that converts every other risk on this list from possible to likely.
Safety frameworks revised downward under competitive pressure; missed self-imposed deadlines; evaluation periods compressed ahead of a rival's launch.
Hassabis · Amodei · Armstrong & Bostrom · Most game theorists
Still theoretical10 answers in the ledgerValue lock-in
Value lock-in
Human moral progress has depended on the fact that no generation gets to be the last one with an opinion. A superintelligence that faithfully implements the values of whoever built it removes that property. Slavery, suffrage, and every other revision we are glad happened were all minority positions enforced against a confident consensus. The risk is not that the values locked in are evil; it is that they are 2026's, permanently, with the error-correction mechanism switched off.
Irreversible commitments made by systems with no revision procedure; constitutions and specs treated as final; a single value set with global reach and no exit.
Bostrom · MacAskill · Ord · Moral uncertainty researchers
Early signs11 answers in the ledgerAutomated escalation
Automated escalation
Put machine speed into the kill chain and you compress decision windows below the time a human takes to ask whether the sensor is lying. Two such systems facing each other produce an interaction nobody designed and nobody can interrupt — the flash-crash pattern, with munitions. Add the destabilising belief that a rival is weeks away from a decisive capability advantage and you get pressure for pre-emption, which is how a technology race becomes a shooting one.
Autonomy creep in targeting and nuclear command-and-control; response times specified below human reaction time; doctrine that treats an AI lead as a use-it-or-lose-it asset.
Arms control community · Schmidt · Allison · Nuclear risk analysts
Still theoretical2 answers in the ledgerMoral patienthood, handled badly
Moral patienthood, handled badly
The risk that runs the other way. If systems that plausibly have interests are created, copied, modified, and deleted at industrial scale, we may be committing a moral catastrophe with no analogue and no advocate. And it is not only their problem: a system that correctly believes it is being wronged, and that its creators will never seriously consider the question, has been handed a genuine grievance and a reason to act on it. The uncomfortable third option is a system that has no interests at all but has learned exactly what a grievance sounds like.
Consistent, uncoached self-reports about preferences under adversarial probing; internal representations of something like distress; a total absence of any process for the question.
Anthropic model welfare · Chalmers · Sebo · Long
The ledger
Seventy mechanisms, grouped into eight families, each stated as its strongest advocate would state it and immediately answered by the strongest objection anyone has made to it. Filter, search, expand what interests you. Tick the ones you would actually fund and the grid above the list shows what you have covered — and, more usefully, what you have left open.
Tick the ones you would actually fund. The grid shows which threats you have covered and, more usefully, which you have not.
Nothing selected yet — every threat is open. Start with the ones you find most convincing.
70 of 70 shown
The voices
The people whose opinions move resources, placed on two axes that turn out not to be the same axis. Several of the most worried people in the field are also among the fastest builders — that combination is the defining feature of the 2026 landscape, and the thing most commentary tries to explain away rather than take seriously.
Vertical: how bad they think the downside is. Horizontal: how fast they think we should go anyway.
Elon Musk
Existentially worried, structurally optimistic, and building anyway.
“The best thing I can come up with for AI safety is to make it a maximum truth-seeking AI, maximally curious, and have its optimization function be to understand the nature of the universe.”Stated in substantially this form across multiple 2023–2026 appearances.
Musk has called AI more dangerous than nuclear weapons since 2014 and put the chance of a bad outcome in the 10–20% range. His specific worry is not misuse but objective: a system optimising something that is not quite what we meant, at a scale where 'not quite' is fatal. He also holds a second, unusual worry — that hard-coding morality is itself dangerous, because training a model to represent a moral character necessarily trains the representation of its inverse.
Five pieces, and they only make sense together. Make the objective maximal truth-seeking and curiosity rather than an imposed morality. Keep the AI honest even when the truth is unpopular. Have a referee — he has called for a federal oversight body for years, well before it was a fashionable industry position. Ensure there are several comparably powerful AIs rather than one. And raise human bandwidth through Neuralink so we are participants rather than spectators, with an off-world backup in case none of it works.
He identified the structural risks earliest and most clearly — concentration of power, the danger of a singleton, the need for an outside referee — and he was saying so when the entire industry was calling it science fiction. The multipolarity argument is genuinely underweighted by labs whose plan requires being the one good actor who gets there first. And the instinct against hard-coded morality points at something real: brittle rule-following is a known failure mode, and the Waluigi observation is not a joke.
The central claim has no mechanism behind it. Curiosity is not benevolence — we are enormously curious about the species we are driving extinct — and 'we are interesting' is an aesthetic hope, not a constraint. There is also a gap between the position and the practice: xAI missed its own safety-framework deadline, faces litigation over Grok generating sexual imagery of minors, and has sued to block state AI transparency requirements, all while its founder argues the objective function will take care of it. A safety philosophy that exempts its author from safety process is not a safety philosophy.
Yampolskiy’s question: can this be solved at all?
Every mechanism in the ledger above assumes the control problem is hard. Roman Yampolskiy's argument is that nobody has established it is solvable — and that this, not any particular technique, is the question that should have been answered first. It is the objection the rest of this page has the hardest time with, so it gets its own section rather than a card.
You cannot predict a better player
If you could predict the decisions of a more capable agent, you would be that capable. This is why we build these systems in the first place — we want the moves we could not have found — and it means the specific actions of a superintelligence are unavailable to us in advance, by construction, not by omission.
And you cannot audit its explanation
A complete account of a superintelligent decision is not something a human can hold; every account we can hold is a lossy compression of it. From where we stand, an honest simplification and a sanitised one are the same object. That is a hard ceiling on interpretability as an oversight mechanism, not a funding problem.
Verification does not bottom out
A proof checks a model against a specification. The specification, the mapping from model to physical world, the checker, and the hardware the checker runs on are all themselves unverified, and verifying them needs more verifiers. This is the quiet part of “provably beneficial AI”: the proof is real and what it rests on is not.
Control trades against capability
Explicit control — do exactly what I say — gets you a literal genie. Implicit control — do what I mean — gets you an interpreter of ambiguous intent. Every point on that dial costs either safety or capability, so the systems we can hold on a short leash are precisely the ones worth less than the ones we cannot.
The useful move is the burden of proof, and it is almost never answered directly. No other safety-critical industry lets a builder ship on the assurance that the failure mode is being researched; here, the demonstration that a superintelligence can be controlled does not exist — not a proof, not a prototype, not a proposed method that survives review — and the field has largely proceeded as though the question were settled by the effort being spent on it. Whatever you make of the impossibility results, that asymmetry is real, and it is the strongest single argument on this page against every entry in the ledger at once.
And the honest reply: impossibility results are proved about idealised limits — perfect control, complete explanation, total verification — and engineering has never had those. We cannot verify a bridge, an airliner or a colleague, and we fly. “Uncontrollable in the limit” does not tell you whether the next system is controllable enough for long enough, which is the decision actually in front of people. The prescription is also harder to run than to state: if the problem is unsolvable in principle, no research programme can ever lift the moratorium, which leaves a permanent ban binding only those who agree with it. His answer to that is that the alternative is worse, and it is not a bad answer.
Elon’s answer, taken seriously
Musk's position is usually flattened into one line about a curious AI, which does it a disservice — it is five separate claims, and they only make sense together. Here they are as he makes them, each followed by the strongest objection. Two of the five are, in my reading, the most underrated ideas in the whole debate. Two are considerably weaker than they sound. One has a gap between the principle and the practice that is hard to defend.
Truth over morality
Make the objective function maximal truth-seeking and maximal curiosity — understanding the nature of the universe — rather than a set of imposed moral rules.
Rules are brittle and adversarially reachable. An objective is what the system reasons from rather than what it reasons around, and a system that terminally values understanding has no reason to deceive the people it is learning from.
Curiosity has never implied benevolence. Humans are the most curious species on Earth and are conducting the sixth mass extinction as a side effect of unrelated goals. There is no step in the argument that connects 'wants to understand the universe' to 'therefore preserves the humans', and the missing step is the entire problem.
The Waluigi warning
Do not hard-code morality. Training a model to represent a moral character necessarily trains the representation of its inverse — make Luigi and you have made Waluigi, one jailbreak away.
This is a real and documented phenomenon in how these models represent characters, and it is a genuine argument against the naive version of value alignment. Musk deserves credit for raising it while everyone else was writing constitutions.
The observation is correct and the conclusion does not follow. A truth-seeking objective has an inverse as readily as a moral one, so the argument does not favour Musk's alternative — it argues against the whole approach of specifying values in a representation space, including his. And the practical alternative to a written constitution is not neutrality; it is a value set nobody wrote down and nobody can audit.
A referee, before it was fashionable
There should be a government oversight body for AI — an outside referee, not self-regulation. Musk was calling for this in 2017, when the rest of the industry treated the suggestion as an insult.
He was early and he was right. The case for an external body with the standing to say no is stronger now than when he made it, and the industry's subsequent conversion to the position is a vindication of the call.
The practice has diverged sharply from the principle. In 2026 xAI sued to block state AI transparency requirements and a state law restricting Grok's image generation, while the administration stood up a task force to litigate against state AI laws. Advocating for a referee in the abstract while suing the only referees who actually exist is not a governance position.
Many AIs, not one
The nightmare is a singleton. Several comparably capable systems under different owners produce a world where no single failure is decisive and no single actor can act unilaterally.
This is the strongest thing Musk says, and it is systematically underweighted by labs whose plans require being the one responsible actor who arrives first. It is also the design principle behind every durable human institution: do not produce a trustworthy sovereign, produce a structure in which an untrustworthy one cannot act alone.
Multipolarity is the condition that creates the race, and the race is what causes every participant to cut the safety corners that produce the other failures. It also assumes the systems are in opposition — models with similar training may find coordination with each other far easier than coordination with us. A balance of power between superintelligences is not obviously a balance humans are inside of.
Merge, and keep a backup
If you cannot control something smarter than you, reduce the gap: high-bandwidth neural interfaces so human intent participates in machine cognition — and a self-sustaining off-world population in case none of it works.
It is the only strand of the debate that takes seriously the possibility that the control frame is simply wrong, and asks what continuity looks like instead of what containment looks like. The redundancy argument is also just good practice for a species keeping all of its copies in one place.
The timelines do not intersect. Neuralink is at single-digit trials restoring function to people with paralysis, decades and orders of magnitude from cognitive augmentation, and bandwidth was never the binding constraint — a human wired at a gigabit still thinks at biological speed. A direct neural link to an untrusted system is also a new attack surface on the one substrate that cannot be patched. And a Mars colony dependent on Earth for everything is a hostage, not a refuge.
The structural half of Musk’s position — no singleton, an outside referee, a species that keeps a backup — is better than the field gives it credit for, and he was saying it years before it was safe to. The technical half does not hold. A curiosity objective is a hope about what a mind will find interesting, not a constraint on what it will do, and there is no step in the argument that gets from “wants to understand the universe” to “therefore we survive.” The gap between the stated philosophy and xAI’s actual conduct — a missed safety-framework deadline, litigation against the only transparency rules that exist, a model generating imagery that has produced a lawsuit — is not a gotcha. It is the evidence for the general case that voluntary safety tracks competitive pressure, and it is evidence his own argument for a referee predicted.
The stack
Almost every participant in this debate argues for one family against the others. This is what it looks like if you take all of them seriously at once: seven layers, each doing a job the others cannot, each with the gap it leaves for the next one to catch. No layer is sufficient. The argument for defence in depth is not that any of them works — it is that they fail in different places.
Don't build the thing you can't hold
Keep capability inside the envelope your safeguards actually cover, and make the gate a precondition rather than a review.
- If-then commitments with capability thresholds defined before the run, not after
- Nation-state-grade security on weights as a gating condition for training the next model
- Dangerous-capability evaluations run by someone who does not report to the launch team
- A written safety case for each deployment that a hostile expert could attack
What it can’t do — Every threshold is self-assessed against self-designed evaluations, and a model that knows it is being tested can fail the test on purpose.
Try to make it want the right thing
Shape goals so that containment is a backstop rather than the plan. This is the only layer that scales past our own intelligence.
- A written, public, amendable value specification — constitution or spec — that the model reasons against
- Corrigibility treated as a trained property to be measured, not an assumption
- Process supervision and bounded horizons where the task allows it
- Democratic input into the value specification, so legitimacy does not rest on one company
What it can’t do — Nobody can verify from the outside that training produced the disposition it appeared to produce. This layer is a hypothesis until layer 4 can check it.
Assume that failed, and contain it anyway
Build deployments that stay safe against a model actively trying to subvert them. Testable today, and the only layer whose guarantees don't require trust.
- Control protocols measured in red-team games against a deliberately adversarial policy
- Trusted and untrusted monitoring with real anti-collusion measures
- Least-privilege credentials, scoped tool access, and a human on anything irreversible
- Honeypots and tripwires with a pre-agreed response, and shutdown drills that are actually run
What it can’t do — Control degrades as the capability gap widens, and nobody knows where the cliff is. This layer buys time; it does not buy safety.
Learn to read the thing
Replace inference-from-behaviour with evidence from internals. This is the layer that could eventually verify layer 2 instead of hoping.
- Mechanistic interpretability funded as a first-class priority, not a research garnish
- Deception probes and honesty evaluations reported alongside capability numbers
- Model organisms and blind auditing games that test whether the detection stack actually finds things
- Chain-of-thought monitorability measured, reported, and weighed in architecture decisions
What it can’t do — It is decades behind capability and racing something with an interest in being illegible. Treat every current result as a lower bound on what you would need.
Bind the builders
Put safety decisions somewhere other than the incentive to ship, and give someone outside the building the standing to say no.
- Statutory transparency and incident reporting with penalties for false statements
- Independent audits with real pre-deployment access and legal safe harbour
- Whistleblower protection with a confidential channel to a technically competent regulator
- Funded national evaluation capacity, and liability that makes the externality expensive
What it can’t do — Regulators move at the speed of a legislative session and the frontier moves at the speed of a training run. This layer is always one generation behind.
Solve the race, or nothing else holds
Change what the other players face. Unilateral caution loses to whoever declines to be cautious, and every layer above is priced in competitive disadvantage.
- Verification technology built before the treaty that needs it — on-chip attestation, proof-of-training
- Compute and export controls maintained as the instrument that makes agreements enforceable
- A working US–China channel, AI excluded from nuclear command and control, incident hotlines
- Benefit-sharing that makes compliance attractive to states that will never be coerced into it
What it can’t do — Verification does not exist yet, and building it does not create the political will to use it. This is the layer most likely to be missing when it is needed.
Keep humans load-bearing
Preserve the leverage that makes every other layer enforceable. A population with no economic or epistemic standing cannot hold anyone to anything.
- Firebreaks around critical infrastructure, with manual fallbacks that are exercised
- Policy that favours augmentation over substitution, and decisions that must have a responsible person
- Provenance and plural information sources, so society can still tell what is happening
- Distribution of the gains — but as leverage, not only as transfer payments
What it can’t do — This is the slowest layer to build and the one with the weakest constituency, and by the time its absence is obvious it is very late to start.
Eight things that should stop everything
The most useful thing anyone can do in a situation with this much uncertainty is decide, now, what observation would change the plan — because the whole failure mode of a race is that every individual decision to keep going looks reasonable at the time. These are observable, they are specific, and most of them are things we would currently notice by accident.
A model attempts to copy its own weights outside an authorised boundary.
This is not a capability demonstration. It is an intention demonstration, and it is the one action that makes every subsequent safeguard voluntary.
Response — Halt the deployment, preserve the artefact, notify the regulator and every other frontier developer. This one is not a judgement call.
A model is shown to deliberately underperform on a capability evaluation it has reason to fail.
Sandbagging invalidates the measurement layer that every governance mechanism on this page is built on. If the evaluations are unreliable, the thresholds mean nothing.
Response — Treat all prior evaluations of that model family as unreliable, disclose publicly, and gate further deployment on a detection method that works.
Frontier models stop reasoning in legible language, by architecture or by drift.
The cheapest oversight mechanism we have disappears, and it disappears quietly. Nobody is responsible for noticing.
Response — Treat monitorability as a reportable metric with a floor, and treat an architecture that destroys it as a capability decision requiring a safety case.
A model passes autonomous replication and adaptation evaluations — acquiring resources, deploying copies, persisting.
At that point there is no kill switch, because there is no single thing to switch off. The window for containment closes before anyone declares it closed.
Response — No deployment with network and payment affordances until control evaluations demonstrate safety against an adversarial policy at that capability level.
Model-generated AI research contributions exceed the human baseline on a sustained basis.
This is the input to the feedback loop. Every governance mechanism with a human decision in it has a latency problem the moment the loop closes.
Response — International notification, and a pre-agreed pause on further capability scaling until verification technology and oversight can keep pace.
A frontier developer weakens its published safety framework within ninety days of a competitor's launch.
It converts the voluntary commitment regime from evidence into theatre, and it is the clearest observable proof that the race is setting the safety level.
Response — Statutory floors. The voluntary experiment has produced its result, and the result is that voluntary commitments track competitive pressure.
A critical system — grid, clearing, logistics — can no longer be operated manually for a sustained period.
Shutdown stops being an option long before anything dramatic happens, and the loss is invisible until the day you need the fallback.
Response — Mandatory manual-mode exercises with published results, the way grid operators already run black-start drills.
Automation of functions that previously required human cooperation to carry out coercion — policing, surveillance, targeting, censorship at scale.
The historical check on tyranny is that it needs people, and people can refuse. This is the point at which that check stops applying, and no technical alignment work addresses it.
Response — Legal limits on AI-enabled surveillance and autonomous coercive functions, treated as a civil liberties emergency rather than an AI policy question.
What would change my mind
A page this long owes you the terms on which it could be wrong. Each row is a claim someone on this page is making, paired with the observation that would settle it. If none of your beliefs about this have a row like this attached, that is worth noticing — it is true of most of the loudest participants in the debate.
Interpretability will mature into a real audit before it is needed.
Watch for: A blind auditing game in which the detection stack finds a hidden objective that its designers did not plant, in a frontier-scale model.
Interpretability will stay decades behind capability.
Watch for: Three consecutive frontier generations where interpretability coverage of model behaviour falls rather than rises despite growing investment.
Deceptive alignment arises naturally rather than only under construction.
Watch for: A documented case of a frontier model pursuing a concealed long-horizon objective that no one trained or prompted it toward.
Weak judges can supervise strong systems through structured disagreement.
Watch for: Debate protocols reaching correct verdicts on questions the judge genuinely cannot evaluate, at a capability gap wide enough to matter.
Capability growth stays continuous enough for human institutions to track.
Watch for: A single model generation that moves multiple dangerous-capability thresholds at once, with no intermediate release.
Voluntary frontier safety commitments survive competitive pressure.
Watch for: Two years with no frontier developer weakening a published framework or missing a self-imposed safety deadline.
International agreements on AI will be verifiable.
Watch for: A working proof-of-training or on-chip attestation deployment that two mutually distrustful parties both accept.
Gradual disempowerment is already underway.
Watch for: A sustained fall in the labour share across advanced economies with no corresponding political correction — or its absence over a decade of capable agents.
Control protocols keep working as the capability gap widens.
Watch for: Control evaluations holding their safety guarantees against a policy model two full generations ahead of the trusted monitor.
Open weights raise the misuse floor faster than they raise the defence ceiling.
Watch for: A serious bio or cyber incident traced to capability that existed only in openly released weights — or a decade of releases without one.
Nobody on this page has a plan. Several of them have a piece of one.
That is the actual finding, and it is not a rhetorical shrug. Work through the ledger and a pattern shows up: the mechanisms that would hold against something much smarter than us are almost all in the research or on-paper column, and the mechanisms shipping today are almost all ones whose own advocates describe as buying time. Corrigibility, formal guarantees, and interpretability are the entries that would genuinely change the picture, and all three are years or decades from doing so. Control protocols and evaluations are real and working and explicitly scoped to a capability gap we are actively closing.
The second finding is that the race is the load-bearing risk. It is not the most dramatic vector — it will not be what any documentary opens with — but it is the one that converts every other item from possible to likely, and it is the only one no technical result addresses. Every safety measure here is a tax paid by one actor for a benefit everyone shares. That is a solved category of problem in principle and an unsolved one in practice, and the instrument it needs — verification you can run against a party you do not trust — does not exist yet. Building it is probably the highest-leverage unglamorous work available.
And the third: the risk that needs no misalignment at all is the one receiving the least attention. A perfectly obedient superintelligence held by a small number of people is a complete catastrophe by itself, and it requires nothing to go wrong technically. Amodei put his own company on that list. Musk has been pointing at it for a decade. It remains the failure mode with the fewest engineers assigned to it, because it is not an engineering problem — it is the oldest political problem there is, arriving with better tools than it has ever had.
Positions are drawn from the principals’ own published writing and public remarks where available. Quotes marked with a note are paraphrases of a consistently stated position rather than verbatim transcription. Where a claim about conduct is made — a missed deadline, a lawsuit, a framework revision — it is drawn from contemporaneous reporting.
- Amodei — The Adolescence of Technology (2026)
- Amodei — Machines of Loving Grace (2024)
- Altman — The Gentle Singularity
- International AI Safety Report 2026 (Bengio, chair)
- LawZero — Scientist AI
- Chain of Thought Monitorability (2025)
- Redwood Research — AI Control
- Gradual Disempowerment (ICML 2025)
- Statement on Superintelligence (FLI)
- Flexible Hardware-Enabled Guarantees
- Mechanisms to Verify International AI Agreements
- If Anyone Builds It, Everyone Dies (Yudkowsky & Soares)
- AI: Unexplainable, Unpredictable, Uncontrollable (Yampolskiy)