United States policy on catastrophic AI risk

AI Policy · Legislation & Regulation · United States — Federal · Browse by topic

United States policy on catastrophic AI risk encompasses measures intended to prevent advanced artificial intelligence from enabling mass harm or escaping effective human control. The approaches range from developer transparency and independent audits to government testing, restrictions on dangerous uses, liability, negotiated slowdowns, and proposed prohibitions on superintelligence. These approaches assign different roles to developers, regulators, courts, and national-security agencies.

This overview covers selected federal and state measures and policy proposals, with core policy coverage through September 11, 2026 and additional constitutional, insurance, and comparative context reviewed September 12, 2026. Individual developments and proposals are dated below. It distinguishes enacted statutes, future effective dates, executive directions, introduced legislation, announced proposals, and voluntary or scholarly frameworks. Its scope is catastrophic-risk prevention rather than the full range of AI-related consumer, employment, copyright, and civil-rights law.

Meaning and scope of catastrophic harm

In California's Transparency in Frontier Artificial Intelligence Act (SB 53), catastrophic risk means a foreseeable, material risk of a frontier model materially contributing to more than 50 deaths or serious injuries, or more than $1 billion in property damage or loss, in a single incident involving specified conduct. That conduct includes assistance with chemical, biological, radiological, or nuclear weapons; certain autonomous criminal or cyber conduct; and evasion of developer or user control. The definition contains exclusions, including lawful federal-government activity.

The statutory threshold is not synonymous with human extinction. It covers a class of severe incidents; the Sanders–Casar proposal separately addresses superintelligent systems and loss of human control. Neither a legislative definition nor an advocate's prediction establishes how likely a catastrophe is. The California policy report identifies substantial uncertainty and disagreement about advanced-AI risks, and recommends drawing on experiments, simulations, historical comparisons, and adversarial testing as well as observed harms.

For prevention, the distinction between misuse and loss of control matters. Misuse policy focuses on people exploiting a model's capabilities; control policy also considers systems circumventing oversight or acting beyond authorized purposes. NIST's draft misuse guidance addresses the former, while California's statute expressly includes evasion of control and certain deceptive behavior outside evaluations.

Debate over the framing

The decision to organize policy around catastrophe is itself contested. In a 2023 opinion essay, Alex Hanna and Emily M. Bender argue that extinction-focused messaging can divert attention and resources from documented discrimination, surveillance, and exploitation of workers. They also question the influence of technology companies over the public agenda and the evidentiary basis of sweeping predictions. Their criticism concerns which harms and affected communities receive priority, as well as claims about future capabilities.

The California frontier-policy report approaches uncertainty differently, recommending empirical investigation and adaptable safeguards. These positions expose a policy choice about the allocation of oversight and research resources; neither a catastrophe-focused statute nor this article's narrower scope substitutes for protections against discrimination and other present harms.

Comparison of approaches

Approach	Principal intervention	Illustrative U.S. measure	Status of cited measure
Transparency and safety plans	Disclose risks and enforce compliance with developer frameworks	California SB 53; New York RAISE Act	California: enacted, operative requirements. New York: enacted; amended principal requirements effective January 1, 2027.
Independent verification	External examination of compliance and safety practices	Illinois Artificial Intelligence Safety Measures Act (SB 315)	Enacted July 6, 2026; effective January 1, 2027.
Federal testing and oversight	Mandatory risk assessment or supervised testing	FRONTIER Act (H.R. 9925); Secure AI Development Act	FRONTIER: introduced bill, July 23, 2026. Secure AI Development: published legislative proposal, July 2026.
Voluntary national-security review	Early government access and defensive cooperation	Executive Order 14409	Executive order issued June 2, 2026, directing a voluntary framework.
Control and intervention	Limit agent access, preserve human authority, and enable shutdown	Stop Rogue AI Act; AI Guardrails Act (S. 4113); AI Kill Switch Act (H.R. 9917)	Introduced bills: September 9, March 17, and July 23, 2026, respectively.
Liability and financial incentives	Internalize costs through liability and insurance	Weil; Schwarcz and Wolff	Scholarly proposals and critiques; existing insurance markets discussed separately below.
Pacing, pauses, and prohibition	Slow or prevent specified capability development	Mutually Agreed Pacing Framework; Ban Artificial Superintelligence Act	Industry agreement proposed September 4, 2026. Legislation announced September 3, 2026 as forthcoming.

State laws: transparency, audits, and preventive duties

California: SB 1047 and SB 53

Scott Wiener's 2024 Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, SB 1047, proposed safety protocols, a shutdown capability for systems within its scope, independent compliance audits, and restrictions on use or release where a covered model posed an unreasonable risk of causing or materially enabling critical harm. Its initial coverage combined training-compute and cost thresholds. These were proposed preventive obligations, not merely duties to disclose a risk after an incident.

Governor Gavin Newsom vetoed SB 1047 on September 29, 2024. His objections included the bill's focus on the largest models rather than sufficiently accounting for deployment context or risks from smaller systems. The veto illustrates disagreement over the scope and design of preventive regulation; SB 1047's proposed obligations never became law through that bill.

California subsequently enacted SB 53 on September 29, 2025. The Attorney General describes requirements for large frontier developers to publish and comply with frameworks for catastrophic-risk management, accompanied by transparency and reporting obligations. Covered employees receive protections for specified disclosures about violations or dangers to public health or safety. The resulting mechanism makes compliance with a developer's framework enforceable; it does not reproduce SB 1047's general unreasonable-risk restriction on release.

New York: amended RAISE Act

The New York RAISE Act was revised by S8828, signed March 27, 2026. The amendment establishes a transparency regime with published frameworks, model reports, and state oversight, and moves the principal requirements to January 1, 2027. Qualifying critical incidents generally must be reported within 72 hours of the specified determination or knowledge trigger, with a 24-hour route for imminent threats of death or serious injury. The law also requires large developers to file disclosure statements and contribute to oversight costs. Attorney General penalties can reach $1 million for a first violation and $3 million for subsequent violations; the article does not create a private right of action.

Illinois: external audits

The Illinois Artificial Intelligence Safety Measures Act (SB 315) adds independent third-party audits to a framework of public safety disclosures, incident reports, compliance processes, and whistleblower protections. The governor's July 6, 2026 signing announcement identifies January 1, 2027 as its effective date. Independent examination is a distinct safeguard because it introduces scrutiny beyond the developer's own public account of its practices.

Federal approaches

Executive action and security cooperation

President Joe Biden's Executive Order 14110 of October 30, 2023 directed Commerce to use Defense Production Act authorities to obtain information from developers of certain dual-use foundation models, including safety-test results and information on protecting model weights. It also directed work on biological, cyber, and critical-infrastructure risks. This was an executive reporting and agency-action approach rather than a comprehensive congressional licensing statute. President Donald Trump revoked that order on January 20, 2025; its requirements should not be presented as the current governing executive order.

America's AI Action Plan, released in July 2025, combines faster innovation with national-security evaluations and targeted defenses. Its recommendations include government evaluation of frontier models for cyber and weapons-related risks, security research, and protection against vulnerabilities in adversaries' systems. Implementation depends on the agencies and legal authorities responsible for each recommendation.

Executive Order 14409 of June 2, 2026 directs agencies to design a voluntary framework permitting government access to covered frontier models for up to 30 days before their planned release to other trusted partners. It also directs classified cyber benchmarking and collaboration on early access for defensive uses. Section 3 expressly does not authorize mandatory government licensing, preclearance, or permitting for model development or release. Separately, the order prioritizes enforcement of existing criminal laws against AI-assisted unlawful computer access.

Licensing, mandatory testing, and federal supervision

The September 2023 Blumenthal–Hawley framework proposed an independent oversight body and licensing for sophisticated general-purpose models or models used in high-risk settings. Licensing would depend on risk management, testing, data governance, and incident-reporting programs. It also proposed audits and legal accountability. It remains a historical framework for legislative design.

The FRONTIER Act, H.R. 9925, was introduced July 23, 2026. Its text combines developer frameworks and reporting with annual independent compliance audits for large frontier developers, an independent-verification structure, and emergency federal intervention provisions. It therefore goes beyond the proposition that companies should voluntarily publish safety principles. The introduced text also addresses the relationship between federal requirements and state law.

Senator Mark Warner's July 2026 Secure AI Development Act proposal takes another route: mandatory secure testing before public deployment, coordinated vulnerability handling, stronger information sharing, and a voluntary incident-reporting system modeled on aviation. The published draft would create an Artificial Intelligence Risk Board within NIST. Mandatory testing and voluntary incident reporting are separate parts of that proposal and should not be conflated.

Shutdown and intervention powers

Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act, H.R. 9917, on July 23, 2026. The bill would require covered entities to maintain shutdown and access-control capabilities and empower the Department of Homeland Security, acting through CISA, to order proportionate intervention following a defined covered incident. It includes compliance verification and judicial review. Red-teaming and other structured testing are excluded from the covered-incident definition, so an alarming demonstration in a test environment would not by itself establish that this particular intervention trigger was met.

Federalism and the September 2026 negotiations

The White House's March 2026 legislative recommendations call for federal preemption of burdensome state AI laws, oppose creating a new federal AI rulemaking body, and favor existing sector regulators and industry standards. They also propose preserving specified state powers, including generally applicable consumer protections and a state's own AI procurement. Implementing this preemption agenda would require congressional action.

The FRONTIER Act's introduced § 9 would preempt new substantive state obligations in specified areas of catastrophic-risk transparency, independent verification, and incident reporting, subject to exceptions. Consequently, a national framework's protective effect depends on both the federal duties it creates and the state remedies or requirements it displaces.

September 11 negotiating snapshot. Reuters reported that Senate negotiators were considering a duty of care for the most capable models, federal power to block unsafe models with judicial challenge available, and preemption of some state AI-risk laws. The terms remained unsettled; this dated report should not be read as the current negotiating text. Subsequent developments belong with the linked news coverage and any published bill text. See September 11 coverage.

Constitutional challenges and legal durability

Constitutional limits on state regulation are distinct from Congress's choice to preempt it. Compelled public disclosures can raise First Amendment questions. In xAI v. Bonta, the district court denied a preliminary injunction against California AB 2013 on March 4, 2026, including the company's compelled-speech challenge. That procedural ruling concerned training-data transparency; it did not adjudicate the constitutionality of SB 53, RAISE, or Illinois SB 315.

Bahrad A. Sokhansanj and Mackenzie Arnold identify implications for frontier transparency laws, emphasizing the legal distinctions among public disclosures, confidential regulatory reporting, and different forms of commercial speech. The durability of a particular duty therefore depends on its text and justification, rather than a general assumption that all AI reporting is either protected from challenge or unconstitutional.

Interstate burdens present a separate potential issue under the dormant Commerce Clause, which limits discriminatory or unduly burdensome state regulation even without federal legislation. However, National Pork Producers Council v. Ross (2023) rejected a broad rule invalidating state laws simply because they influence commerce outside the state. Applying that doctrine to frontier-AI requirements would require analysis of the particular law and record.

Litigation coverage note (September 12, 2026): The sources verified for this overview establish the adjacent AB 2013 litigation, but do not establish a filed challenge directly against SB 53, RAISE, or Illinois SB 315. This is a limit of the reviewed record, not a comprehensive certification that no such case exists.

Technical standards and control of dangerous uses

Voluntary standards and corporate commitments

The NIST AI Risk Management Framework is voluntary guidance for organizations developing, deploying, or using AI. It provides a general risk-management structure rather than a government determination that a particular model is safe. NIST's January 2025 draft misuse guidance offers a more targeted sequence: identify risks, plan responses, protect access, measure and mitigate risks before deployment, monitor misuse, and disclose practices. It cautions that safeguards can be brittle and that following the practices cannot guarantee prevention of misuse.

Corporate policies can link capability assessments to stronger safeguards, but their content and strength can change. Anthropic's February 2026 explanation of Responsible Scaling Policy version 3.0 separates company plans from industry-wide recommendations and describes its Frontier Safety Roadmap goals as nonbinding public targets. It also introduces recurring risk reports and external review in specified circumstances. This illustrates why a corporate roadmap, an operational commitment, and a legally required compliance framework need separate treatment.

AI agents and military applications

The Stop Rogue AI Act, whose introduction Gottheimer's office announced on September 9, 2026, would direct NIST to develop standards for discovering, identifying, monitoring, and controlling agents on organizational networks. The announcement emphasizes inventories, verifiable provenance, detection of unauthorized behavior, and the ability to deny or revoke access. This targets the permissions and environment in which an agent acts, rather than only the capabilities of its underlying model.

The AI Guardrails Act of 2026, S. 4113, instead addresses Department of Defense use. Its introduced text would prohibit AI execution of nuclear launch or detonation, restrict certain domestic surveillance, and require appropriate human judgment and supervision for autonomous lethal force. It includes a conditional waiver mechanism for the autonomous-weapons restriction. A military-use proposal of this kind is distinct from a general civilian frontier-model safety law.

Biosecurity and infrastructure defenses

Another approach intervenes at the practical steps needed to turn harmful information into physical harm. The July 2025 AI Action Plan recommends requiring federally funded scientific institutions to use nucleic-acid synthesis providers and tools with sequence screening and customer verification, backed by enforcement. It also recommends information sharing to identify malicious customers. These are downstream biosecurity interventions alongside model evaluations.

For cyber risks, Executive Order 14409 directs a government–industry clearinghouse to coordinate vulnerability discovery, remediation, and patches, and measures to expand defensive tools for critical infrastructure. This illustrates a prevention strategy focused on making potential targets harder to compromise as AI capabilities increase.

Expert proposals for new institutions

Cass Sunstein's September 11, 2026 essay sketches a dedicated AI Regulatory Commission: a multimember body with investigative powers, periodic company disclosures, civil penalties, rulemaking, and authority to act against imminent or catastrophic threats. Licensing and industry funding remain open design questions. This preliminary institutional proposal complements the earlier licensing debate but has no legislative status. Sunstein favors insulation from political pressure while recognizing constitutional obstacles to agency independence.

In a September 11, 2026 New York Times opinion essay, Stephen Witt advocates a temporary slowdown while permanent oversight is established. He reports Yoshua Bengio's support for compulsory incident investigation modeled on the National Transportation Safety Board and Daniel Kokotajlo's proposal for an international public database of training runs and their locations. Witt argues for powers to obtain evidence and compel testimony, public scrutiny of development, and enforceable shutdown capabilities. These are proposals discussed in an opinion essay; the AI Kill Switch Act's specific legislative mechanism is described under shutdown and intervention powers above. Witt also acknowledges economic disruption and foregone benefits as costs of restrictions.

Liability, insurance, and public investment

Liability incentives and their limits

Liability proposals seek to make developers and deployers bear costs otherwise imposed on people outside the commercial transaction. Gabriel Weil argues that strict liability can create continuing incentives for precautions, including precautions that a regulator has not anticipated. His case differs from a negligence standard focused on whether a defendant took reasonable care and from a licensing system centered on permission before deployment. He presents this as a reform to the allocation of legal responsibility.

Insurance could strengthen those incentives if coverage and premiums reward effective safety measures. Daniel Schwarcz and Josephine Wolff question whether insurers can do that reliably for frontier AI: relevant loss data may be sparse, risks change quickly, and the effectiveness of safeguards is difficult to assess. Their critique concerns the ability to price and reduce risks, not merely the formal availability of a damages claim.

Weil also identifies limits: catastrophic losses can exceed realistically collectible compensation; some harms may not produce useful earlier warning incidents; foreign enforcement can be difficult; and government conduct is less responsive to ordinary liability incentives. He argues that liability cannot by itself solve underinvestment in broadly useful safety research, for which public subsidies or prizes may be appropriate. Compensation, deterrence, and prevention before an irreversible disaster are therefore different policy objectives.

Insurance regulation and market practice

Insurance regulation already addresses insurers' own use of AI. The NAIC adopted a model bulletin in December 2023 setting expectations for written governance programs, risk management, lawful consumer outcomes, and examination records. It is guidance for state regulators rather than a model statute. New York's July 2024 Circular Letter No. 7 sets supervisory expectations for AI and external data in underwriting and pricing, including discrimination testing, actuarial validity, senior oversight, and vendor diligence. These measures govern insurance decisions; they do not require insurers to cover frontier-model catastrophes.

Commercial coverage provides another, narrower connection. Aon's 2026 guidance maps AI exposures across cyber, technology errors and omissions (E&O), professional indemnity, and directors and officers (D&O) insurance, among other lines. Different policies address different losses and claims; D&O, for example, may respond to allegations about management representations or oversight. Marsh emphasizes that the facts of a loss, policy language, and exclusions determine the response, while limited loss data complicate pricing.

AI-specific products also exist: Munich Re's aiSure offers protection associated with AI performance commitments and errors, describing a focus on ordinary business losses rather than catastrophic loss. Such products demonstrate a market for bounded AI exposures, but their availability does not establish reliable pricing or sufficient capacity for correlated frontier-model catastrophes. That gap is central to the Schwarcz–Wolff critique.

Pacing agreements, pauses, and bans

On September 4, 2026, Scott Wiener and other state lawmakers called for a Mutually Agreed Pacing Framework: an agreement among frontier laboratories, independently verified, to pace capability development while safety work catches up. The announcement expressly presented this as additional to state, federal, and international action. Laboratory participation and an enforceable agreement would require further negotiation.

The Ban Artificial Superintelligence Act announced by Bernie Sanders and Greg Casar on September 3, 2026 proposes a more restrictive model: a permanent superintelligence prohibition, a temporary pause on advanced-AI development pending a new regulator and safety rules, and international efforts to prevent superintelligence development. The sponsors described the legislation as forthcoming. The distinction is substantive: a conditional slowdown leaves a route to further development, while a permanent prohibition closes that route for the defined class of systems.

International coordination also appears in scholarly proposals. Simon Goldstein and Peter Salib propose a joint U.S.–China frontier laboratory to reduce incentives for a destabilizing race. They offer this as a scholarly institutional design. It highlights the separate question of how a domestic safety regime would work when capable development can occur abroad.

International comparison: the European Union

For the companion overview, see European Union policy on catastrophic AI risk.

The EU AI Act combines several regulatory layers. Its prohibited-practice rules address specified unacceptable uses, while its high-risk system regime focuses on applications such as employment and access to essential services. Those categories cover harms much broader than catastrophe. The closer comparator for U.S. frontier-model policy is the Act's separate regime for general-purpose AI models with systemic risk: providers face additional duties to assess and mitigate those risks, with supervision by the European AI Office. General-purpose model obligations began applying in August 2025, with AI Office enforcement powers applying from August 2026, subject to the Act's transitional arrangements. A voluntary code of practice supports compliance with underlying legal obligations. This structure combines application-specific protections with model-level systemic-risk oversight, rather than treating either category as a substitute for the other.

Central design questions

The approaches differ on what evidence should trigger intervention and who must act on it. The California policy report recommends adaptable thresholds and independent evidence, while warning that thresholds are imperfect. Its analysis supports examining deployment context and revising metrics as capabilities change, rather than treating a numerical model-size threshold as a complete measure of danger.

They also differ on whether a failed assessment produces a legal duty to stop, a duty to mitigate, a reporting obligation, or a voluntary decision. SB 1047's vetoed release restriction, SB 53's framework-compliance duties, and Executive Order 14409's voluntary federal access illustrate materially different consequences. An assessment requirement alone does not establish which consequence follows.

Finally, accountability requires attention to institutional capacity and incentives. The Blumenthal–Hawley proposal addresses conflicts of interest in an oversight body; Illinois emphasizes independent audits; and the liability debate asks whether courts and insurers can identify and deter risks effectively. These mechanisms can be combined, but none should be described as a demonstrated guarantee against catastrophic harm.

Related articles

• Frontier-model safety index
• United States Federal Authorities
• California AI Law
• New York RAISE Act
• Illinois SB 315 Artificial Intelligence Safety Measures Act
• FRONTIER Act (119th Congress)
• Ban Artificial Superintelligence Act
• Stop Rogue AI Act
• Scott Wiener
• XAI v Bonta
• California AB 2013
• EU AI Act
• Executive Order 14365 — Ensuring a National Policy Framework for Artificial Intelligence
• NIST Generative AI Profile