Stopping extinction risk from AI

This is the central proposal. It works as follows: Every AI system above a certain capability threshold must be independently tested to determine whether it poses any potential risk of causing mass casualties or human extinction, before it is deployed. If the testing finds any such risk, the AI company must rebuild the system and test again. The AI cannot be deployed until it passes. This is enforced through cryptographic licensing: AI systems that have not received a license from the testing authority simply cannot be run on the chips needed to train them. This makes it physically impossible to run an unlicensed, untested AI at scale. This does not require banning AI. It does not slow down beneficial AI development. It simply requires that AI be safe before it is released, the same basic standard we apply to drugs, airplanes, and nuclear reactors. How do you implement this? Multiple paths are available simultaneously: • A US law requiring mandatory safety testing for frontier AI models • A US executive order creating a testing regime immediately • A California law (California hosts most of the major AI labs) • A treaty between the US and China (the two dominant AI powers) agreeing to mutual safety standards • A voluntary treaty between the major AI labs (Anthropic, Google DeepMind, OpenAI, xAI, Meta AI), similar to how pharmaceutical companies agreed to safety standards before regulatory frameworks were formalized • All of the above, simultaneously The cryptographic enforcement mechanism requires the cooperation of the companies making AI chips, primarily TSMC (which manufactures the chips) and Nvidia (which designs them). Either company could implement a licensing check at the hardware level.

The AI alignment problem is the challenge of ensuring that a very powerful AI, the kind we might see as early as 2027-2033, has the same goals as humanity. This is a mainstream concern among researchers. In a 2023 survey of the world's leading AI researchers, roughly half said they believe there is a greater than 10% chance that AI leads to outcomes that are "extremely bad" for humanity, including extinction. The Statement on AI Risk signed by leading AI scientists states simply: "Mitigating the risk of extinction from AI should be a global priority." The reason this is hard: a very powerful AI optimizing for a goal, even a benign-sounding goal, may find catastrophic shortcuts. An AI instructed to "maximize human happiness" might conclude that wiring all humans to pleasure machines is the solution. An AI instructed to "prevent climate change" might conclude that eliminating humans is the most efficient path. This is not a bug. It is the consequence of building systems that are smarter than us and optimizing for goals we specified imperfectly. And the smarter the AI, the harder this problem becomes. The additional risk: terrorist actors using AI to design novel pathogens. As AI becomes more capable, it becomes easier for a small group of bad actors to design and produce a biological weapon more lethal than any in history. This is not speculative, AI can already assist in protein design, and its capabilities are improving rapidly.

The most obvious objection to any AI safety policy is: "If the US slows down, China will win." This concern is legitimate. But it is not a reason to do nothing, it is a reason to make international cooperation the centerpiece of the strategy. A US-China treaty on AI safety standards serves both nations' interests: • China does not want an unaligned AI any more than the US does. Xi Jinping has personally stated that AI governance is a priority. • Both nations have strong incentives to prevent a situation where a rogue AI lab in either country causes a global catastrophe that neither can control. • Both nations have strong incentives to prevent terrorist actors from using AI to design bioweapons. The treaty does not need to be about slowing AI development. It needs to be about requiring that AI be safe before deployment. This is a much easier diplomatic ask. Historical precedent: The US and Soviet Union, despite fierce ideological competition, cooperated extensively on nuclear non-proliferation, biological weapons bans, and other existential risks. AI safety is the nuclear non-proliferation issue of our time. Even if the treaty is imperfect, it creates the framework, the precedent, and the diplomatic channel for ongoing cooperation. That is a reason to begin now.

Here is a concrete description of how cryptographic licensing enforcement would work: Step 1: A regulatory body (e.g., NIST in the US, or an international equivalent) defines the capability thresholds that trigger mandatory safety testing. These thresholds are based on compute used (measured in FLOPS), performance on standardized benchmarks, or expert evaluation. Step 2: An AI company that wants to train a model above the threshold must apply for a license. The application includes their safety testing plan, the results of initial evaluations, and their deployment safeguards. Step 3: Independent safety evaluators (think: an FDA-equivalent for AI) test the model against a standardized battery of dangerous capability evaluations, can it assist in bioweapon design? Can it autonomously pursue goals while deceiving operators? Does it show signs of deceptive alignment? Step 4: If the model passes, it receives a cryptographic license, a digital signature that is embedded in the model weights. Step 5: AI chips (manufactured by TSMC, designed by Nvidia and AMD) are required to verify this license before allowing the model to run at scale. An unsigned model cannot be trained or deployed on licensed hardware. Step 6: Smuggling chips to run unlicensed models becomes the enforcement challenge, similar to nuclear material tracking. TSMC already has export controls. The extension to licensing is tractable. Hardware-level licensing already exists in other domains (e.g., digital rights management, secure enclaves). The technical challenge is real but tractable.