European Union policy on catastrophic AI risk

European Union policy on catastrophic AI risk combines preventive duties for advanced AI model providers, public supervision, technical evaluation, cybersecurity, and international cooperation. Its central legal mechanism is the EU AI Act regime for general-purpose AI models with systemic risk. That regime addresses risks from powerful, widely usable models alongside the Act’s separate protections for particular AI applications.

This thematic overview covers selected EU approaches and their tradeoffs, with sources reviewed September 12, 2026. It is a companion to United States policy on catastrophic AI risk; the EU AI Act article supplies the broader instrument-level reference. Binding law, voluntary compliance tools, political commitments, and announced investment plans are distinguished below.

Meaning and scope of catastrophic harm

The AI Act defines systemic risk by reference to a GPAI model’s high-impact capabilities, its reach or foreseeable adverse effects, and the possibility of effects spreading at scale through the value chain. Protected interests include health, safety, public security, fundamental rights and society as a whole. Article 3(65) does not set a single death-toll or monetary-loss threshold. Its scope therefore extends beyond catastrophe narrowly understood, and does not require a prediction of human extinction.

Recital 110 identifies major accidents, disruption of critical sectors, public-health consequences, and threats to democratic and economic security. It expressly discusses chemical, biological, radiological and nuclear risks; offensive cyber capabilities; interference with infrastructure; self-replication; and unintended loss of human control. It also includes discrimination, disinformation and privacy harms. The recital explains the legislative rationale; the operative obligations are in the articles.

Two distinctions shape this approach. Misuse involves people exploiting a model’s capabilities, while loss of control concerns failures to keep the model aligned with human direction. Separately, a high-risk AI system under Article 6 is classified by regulated product or use context; a GPAI model with systemic risk is classified under Article 51. A dangerous application need not use a model above the GPAI threshold, and a powerful general-purpose model can support many different applications. These layers are complementary.

Comparison of approaches

Approach	Principal intervention	EU mechanism and legal status
Model-level prevention	Evaluate capabilities and mitigate systemic risks before and after release	AI Act Article 55: binding provider duties, subject to transition rules.
Technical implementation	Turn broad legal duties into documented safety processes	GPAI Code of Practice: voluntary compliance route; underlying law remains binding.
Independent scrutiny and intervention	Obtain information, evaluate models, require mitigation or market restrictions	Commission powers under Articles 88–94; scientific-panel support.
Resilience and evaluation capacity	Strengthen defenses and external testing capacity	July 2026 Action Plan on Cybersecurity and AI: Commission policy programme, distinct from statutory duties.
Liability	Compensate covered damage caused by defective products, including software	Directive (EU) 2024/2853: enacted directive with a December 2026 transposition deadline and prospective product coverage.
International coordination	Develop shared approaches to severe risks and developer conduct	Hiroshima Code and Seoul commitments: voluntary guidance and political cooperation.

Which models and providers are covered?

Capability thresholds and designation

Article 51 establishes a presumption of high-impact capabilities when cumulative training computation exceeds 10²⁵ floating-point operations. The presumption is a regulatory screening mechanism, not a finding that a model will cause catastrophe. Providers can submit substantiated arguments that a qualifying model exceptionally does not present systemic risks because of its specific characteristics. The Commission assesses those arguments.

The Commission can also designate a model based on equivalent capabilities or impact, including following a qualified alert from the scientific panel. Annex XIII includes training computation, parameters, data, modalities, capabilities, autonomy, tool access and user reach. A model below the numerical training threshold is therefore not automatically outside systemic-risk supervision. Article 51 also provides for updating thresholds and indicators as technology changes.

A provider must notify the Commission without delay, and within two weeks after the high-impact condition is met or it becomes known that it will be met. Designated providers can seek reassessment on the statutory timetable using new, objective and detailed reasons. The scheme combines an initial numerical presumption with case-specific judgment and subsequent review.

Market reach, research and national security

The Act applies to providers placing GPAI models on the EU market even when they are established outside the EU. This gives the regime significance for overseas developers serving EU users. It is a market-based jurisdictional rule, rather than a claim that the EU directly regulates every AI laboratory worldwide.

Article 2 preserves Member States’ national-security competences and excludes AI systems insofar as they are supplied or used exclusively for military, defense or national-security purposes. It also excludes specified scientific research and pre-market research, testing and development, with qualifications including real-world testing. Those exclusions limit the AI Act’s reach; they are not findings that excluded activity is safe or exempt from other applicable law.

Open-source models

Open-source release does not provide a blanket exemption. Article 53(2) removes certain documentation duties for models satisfying its open-license and public-information conditions, but preserves copyright and training-summary obligations. That exception does not apply to models with systemic risk. Article 54 similarly retains the authorized-representative requirement for systemic-risk models within its scope.

The policy rationale recognizes both transparency benefits and release-related dangers. Recital 110 identifies distribution strategies and the possibility of removing safeguards as risk factors; recital 104 states that openness alone does not justify exempting systemic-risk models. Openness and safety classification are consequently separate questions under the statute.

Preventive duties and technical implementation

What the law requires

Article 55 adds four central duties to the general GPAI requirements:

• Evaluation: use methods reflecting the state of the art, including documented adversarial testing to identify and reduce systemic risks.
• Risk assessment and mitigation: address possible EU-level systemic risks arising from development, market placement or use, including their sources.
• Incident reporting: track, document and report serious incidents and possible corrective measures without undue delay to the AI Office and, where appropriate, national authorities.
• Cybersecurity: protect both the systemic-risk model and its physical infrastructure adequately.

These duties reach beyond publishing a safety promise. Assessment and mitigation are legal obligations, while incident reporting supplies information when safeguards fail. The same providers also face the general documentation and downstream-information duties under Article 53. Information for system builders is intended to help them understand model capabilities and limitations and meet their own obligations.

The Code of Practice

The Commission published the GPAI Code of Practice on July 10, 2025. Its Transparency and Copyright chapters concern general GPAI obligations; its Safety and Security chapter concerns systemic-risk models. The Commission and AI Board subsequently endorsed it as an adequate voluntary compliance tool. Signing is voluntary; compliance with the applicable AI Act duties is not.

The Safety and Security chapter calls for a written framework, risk identification and analysis, model reports, and risk-acceptance decisions. Its specified risks include CBRN harms, loss of control, cyber offense and harmful manipulation. Measure 4.1 calls for justified acceptance criteria and safety margins reflecting uncertainty. Measure 4.2 commits signatories to proceed only where risks are acceptable; otherwise they must take appropriate measures, potentially including withholding, restricting, withdrawing or recalling a model, mitigation and reassessment. These are commitments within the voluntary compliance route, not a freestanding EU ban on advanced AI.

Under Article 55(2), providers using neither an approved code nor a relevant harmonized standard must demonstrate alternative adequate compliance for Commission assessment. A harmonized standard supplies a presumption of conformity only to the extent it covers the relevant obligations. Neither the availability of a code nor a signature should be confused with a guarantee that a model cannot cause harm.

The Commission’s separate GPAI guidelines explain its interpretation of coverage and provider obligations. They are nonbinding guidance intended to inform enforcement, and complement the Code rather than amend the Act.

Public supervision, intervention and legal safeguards

Commission and scientific expertise

The Commission has exclusive power to supervise and enforce Chapter V’s GPAI obligations, with implementation entrusted to the AI Office. The scientific panel supplies independent expertise, including qualified alerts concerning systemic risks and assistance with evaluations. This separates the central model-supervision function from the Act’s wider system-level enforcement arrangements.

Articles 91 and 92 permit requests for documents and information and, under specified conditions, model evaluations. The Commission can appoint independent experts and request access needed for those evaluations. These powers provide a route to check provider claims instead of relying entirely on public disclosures or voluntary access.

Market restrictions and penalties

Article 93 permits the Commission to require appropriate compliance measures, mitigation where evaluation produces serious and substantiated concern about EU-level systemic risk, and restrictions on market availability, withdrawal or recall. A structured dialogue may precede action; mitigation commitments offered in that process can be made binding. The intervention is directed to the relevant model and statutory grounds, rather than automatically banning an entire category of future intelligence.

Article 101 authorizes GPAI-provider fines up to the higher of €15 million or 3% of the preceding year’s worldwide annual turnover for specified intentional or negligent infringements, including failure to cooperate with information requests, evaluations or corrective measures. The penalty must be proportionate, and providers must have an opportunity to be heard. The Court of Justice has jurisdiction to review the amount. These are statutory ceilings and procedures, not automatic penalties for developing a capable model.

Article 78 protects confidential information, including intellectual property and trade secrets, while Article 94 provides procedural protections for affected operators. The design therefore pairs access for supervisors with limits on disclosure and safeguards against unsupported intervention.

Cybersecurity, biosecurity and public capacity

Model-level safeguards are one layer of the policy. The Commission’s July 7, 2026 Action Plan on Cybersecurity and AI also calls for stronger European pre-release evaluation capacity, qualification criteria for external evaluators, access to advanced defensive capabilities, and preparation of critical sectors for AI-enabled attacks. It identifies biological misuse as another field relevant to external evaluation. These are announced policy actions; the plan alone does not establish that the proposed capacity is already operational.

The plan connects AI oversight with the Cyber Resilience Act, NIS2 and financial-sector digital resilience. The policy distinction is between reducing a model’s dangerous capabilities or availability and making potential targets harder to harm. Investment in evaluation and defensive use complements the provider obligations, while raising implementation questions about expertise, access and resources.

The EU’s innovation strategy also matters. The April 2025 AI Continent Action Plan presents trustworthy AI, competitiveness and European capability development as joint objectives. This supplies context for interpreting systemic-risk regulation: the stated policy includes promoting AI development and uptake, with safeguards. An investment strategy should not be read as proof that its safety or economic objectives have been achieved.

Liability and compensation

The revised Product Liability Directive, Directive (EU) 2024/2853, includes software within its product definition and provides compensation rules for specified damage caused by defective products. Articles 9 and 10 address evidence disclosure and rebuttable presumptions, including situations where technical or scientific complexity creates excessive evidentiary difficulties. These mechanisms can matter to AI claims without making every harmful output proof of defectiveness.

Member States must transpose the directive by December 9, 2026; its product coverage applies to products placed on the market or put into service after that date. This is a separate timetable from the AI Act’s GPAI duties. The directive is a defective-product compensation regime, not a dedicated catastrophic-AI insurance scheme or a substitute for preventive supervision. Covered harm, causation, responsible operators and defenses remain legally relevant.

International cooperation and limits on development

The Commission’s Hiroshima Process Code of Conduct publication of October 2023 presents voluntary guidance for organizations developing advanced AI, covering the lifecycle and allowing different jurisdictions to implement the principles differently. Such guidance can encourage compatible practices without itself creating the EU Act’s enforcement powers.

The May 2024 Seoul Ministerial Statement included the EU among its signatories. Participants committed to work on shared severe-risk thresholds, including capabilities assisting chemical or biological weapons and capabilities evading human oversight. The associated government announcement also distinguished company safety commitments from governmental cooperation. These political and voluntary arrangements support coordination; they are not equivalent to an EU regulation.

Within the AI Act regime, Article 55’s mitigation duty and Article 93’s model-specific intervention powers are the central preventive mechanisms. They should be distinguished from proposals for a general development pause or permanent prohibition on superintelligence. The framework reviewed here uses classification, continuing obligations and proportionate interventions; it does not make crossing the training-compute threshold an automatic prohibition.

Policy debate and central design questions

Evidence, thresholds and uncertainty

The statutory threshold makes initial coverage easier to identify, while designation criteria allow attention to capabilities and reach that computation alone may miss. The possibility of changing thresholds and reassessing designations makes adaptation part of the legal structure. A central implementation question is how supervisors translate evolving evaluation evidence into defensible classification and mitigation decisions.

The Act’s broad conception of systemic risk also prevents catastrophic-risk policy from exhausting the EU’s AI agenda. Fundamental rights, democratic processes and present-day harms appear alongside severe physical and security risks. The issue is how those protected interests receive effective attention within the different regulatory layers, rather than whether the legislation recognizes only one class of harm.

Innovation, openness and oversight

In its July 30, 2025 announcement that it would sign the GPAI Code, Google argued that aspects of the Act and Code could slow European AI development and deployment and raise concerns about trade-secret exposure and competitiveness. These were a regulated company’s policy objections, not a judicial finding that the Code was unlawful or an empirical demonstration of its effects.

The Commission’s endorsement presents the Code as a means of making compliance clearer, while the statute preserves confidentiality and alternative compliance routes. The resulting design question is whether practical implementation can provide meaningful scrutiny with proportionate costs and useful predictability. Formal obligations, agency expertise and the quality of evidence each matter to that assessment.

Comparison with the United States

For comparison with United States policy on catastrophic AI risk, the distinctive EU feature is a binding, EU-wide model-provider regime centered on systemic risk, coupled with a voluntary technical compliance route and Commission supervision. Useful comparisons should ask separately what triggers coverage, who evaluates the evidence, whether mitigation is compulsory, and what follows when risk remains unacceptable. “Voluntary code” describes one compliance mechanism; it does not describe the legal force of the whole regime.

Implementation timeline

Date	Significance for this overview
August 1, 2024	AI Act entered into force; application was phased.
July 10, 2025	Final GPAI Code of Practice published.
August 2, 2025	GPAI obligations began applying, subject to the transition for older models.
July 7, 2026	Commission issued its Cybersecurity and AI Action Plan.
August 2, 2026	Commission GPAI enforcement phase, including Article 101 fines.
December 9, 2026	Product Liability Directive transposition deadline; directive covers products marketed or put into service after this date.
August 2, 2027	Compliance deadline for GPAI models placed on the market before August 2, 2025.

Related articles

• Anu Bradford
• Marietje Schaake
• United Kingdom policy on catastrophic AI risk
• China policy on catastrophic AI risk
• United States policy on catastrophic AI risk
• EU AI Act — full scope, amendments and application dates.
• European Union AI Law — related EU legislation and implementation resources.
• Cyber Resilience Act
• Frontier-model safety index
• Policy

Related organizations

• Institute for Law & AI
• Centre for the Governance of AI
• Future of Life Institute