Systemic Risk Obligations
Additional requirements for GPAI models with systemic risk.
Systemic Risk Obligations (Article 55)
Learning Objectives
By the end of this chapter, you will be able to:
- Explain the four core obligations for systemic risk GPAI models under Article 55
- Design comprehensive adversarial testing programmes meeting regulatory standards
- Implement systemic risk assessment methodologies at Union level
- Establish incident tracking and reporting systems compliant with AI Office requirements
- Develop appropriate cybersecurity measures for GPAI models and infrastructure
The Enhanced Obligation Framework
Article 55 imposes four additional obligations on providers of GPAI models with systemic risk, supplementing (not replacing) the baseline Article 53 requirements. These obligations address the heightened risks posed by the most capable AI models.
Obligation Structure Overview
| Obligation | Article Reference | Primary Focus | Key Deliverable |
|---|---|---|---|
| Model Evaluation | Article 55(1)(a) | Adversarial testing and capability assessment | Evaluation reports with methodology documentation |
| Risk Assessment & Mitigation | Article 55(1)(b) | Union-level systemic risk identification and control | Risk assessment documentation and mitigation plan |
| Incident Tracking & Reporting | Article 55(1)(c) | Serious incident documentation and notification | Incident reports to AI Office |
| Cybersecurity Protection | Article 55(1)(d) | Model and infrastructure security | Cybersecurity assessment and measures |
Obligation 1: Model Evaluation (Article 55(1)(a))
Regulatory Text Analysis
Article 55(1)(a) requires providers to:
"perform model evaluations in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risk"
State-of-the-Art Adversarial Testing
"State of the art" is a dynamic standard that evolves with the field. Current best practices include:
| Testing Category | Techniques | Focus Areas |
|---|---|---|
| Capability Elicitation | Systematic prompting, task decomposition | Dangerous capabilities, dual-use potential |
| Jailbreak Testing | Prompt injection, roleplay attacks, encoding | Safeguard circumvention |
| Red Teaming | Human adversaries, automated attacks | Real-world misuse scenarios |
| Bias & Fairness | Demographic probing, representation analysis | Discrimination risks |
| Factual Accuracy | Knowledge probing, hallucination detection | Misinformation potential |
| Multi-modal Risks | Cross-modality attacks (if applicable) | Emergent behaviours |
Comprehensive Testing Programme Structure
| Phase | Duration | Activities | Outputs |
|---|---|---|---|
| Pre-deployment Testing | 4-8 weeks | Core capability evaluation, safety assessments | Baseline risk profile |
| Extended Red Team | 2-4 weeks | Expert adversarial testing, domain specialist review | Vulnerability catalogue |
| Stress Testing | 1-2 weeks | Edge cases, high-volume scenarios, combined attacks | Failure mode documentation |
| Post-deployment Monitoring | Continuous | Production monitoring, user feedback analysis | Ongoing risk assessment |
Documentation Requirements
| Document | Content | Frequency |
|---|---|---|
| Testing Protocol | Methodology, tools, scenarios, success criteria | Pre-testing |
| Vulnerability Register | Identified weaknesses, severity ratings, status | Continuous |
| Mitigation Record | Actions taken for each vulnerability | Per finding |
| Summary Report | Overall assessment, residual risks | Per evaluation cycle |
Expert Insight
The AI Office has indicated that documentation should demonstrate not just that testing occurred, but that it was thorough, systematic, and appropriate to the model's capabilities. Superficial testing will not satisfy the "state of the art" requirement.
Obligation 2: Systemic Risk Assessment and Mitigation (Article 55(1)(b))
Regulatory Framework
Article 55(1)(b) requires providers to:
"assess and mitigate possible systemic risks at Union level, including their sources, that may stem from the development, the placing on the market, or the use of general-purpose AI models with systemic risk"
Defining "Systemic Risk at Union Level"
Systemic risks are those that could affect the Union as a whole or significant parts of it. Categories include:
| Risk Category | Examples | Union-Level Impact |
|---|---|---|
| Democratic Processes | Large-scale disinformation, electoral manipulation | Undermining democratic institutions across Member States |
| Critical Infrastructure | Cyberattack automation, control system vulnerabilities | Cross-border infrastructure disruption |
| Public Health | Medical misinformation, healthcare system attacks | Health system strain across multiple countries |
| Public Safety | Weapons development assistance, CBRN risks | Security threats affecting multiple Member States |
| Economic Stability | Market manipulation, automated fraud at scale | Financial system disruption |
| Fundamental Rights | Mass surveillance enablement, discrimination at scale | Rights violations affecting large populations |
Risk Assessment Methodology
| Stage | Activities | Outputs |
|---|---|---|
| 1. Risk Identification | Capability mapping, threat modelling, scenario analysis | Risk register with systemic risks identified |
| 2. Likelihood Assessment | Probability estimation, attack feasibility analysis | Likelihood scores (1-5 or qualitative) |
| 3. Impact Assessment | Severity evaluation, Union-level scope analysis | Impact scores and justification |
| 4. Risk Prioritisation | Risk matrix application, ranking | Prioritised risk list |
| 5. Mitigation Planning | Control identification, implementation planning | Mitigation roadmap |
| 6. Residual Risk Evaluation | Post-mitigation assessment | Residual risk documentation |
Mitigation Measure Categories
| Measure Type | Examples | Effectiveness |
|---|---|---|
| Training-time Controls | Data filtering, RLHF, constitutional AI | High—addresses root causes |
| Inference-time Controls | Output filtering, content moderation, rate limiting | Medium—can be circumvented |
| Access Controls | API restrictions, use case verification, tiered access | Medium—reduces but doesn't eliminate risk |
| Monitoring & Response | Usage monitoring, anomaly detection, incident response | Reactive—limits damage |
| External Collaboration | Information sharing, coordinated disclosure | Supportive—enhances ecosystem resilience |
Value Chain Risk Assessment
Article 55(1)(b) explicitly requires consideration of risks throughout development, placing on market, and use:
| Value Chain Stage | Risk Considerations | Assessment Focus |
|---|---|---|
| Development | Training data risks, capability emergence, security of development environment | Internal processes and controls |
| Placing on Market | Distribution channel risks, access control adequacy, documentation sufficiency | Release procedures and safeguards |
| Use | Downstream applications, foreseeable misuse, unintended applications | End-use scenarios and user population |
Compliance Note
Risk assessment must be forward-looking. You must consider "reasonably foreseeable" negative effects—not just known, existing risks. This requires ongoing capability monitoring as models are fine-tuned or applied in new contexts.
Obligation 3: Incident Tracking and Reporting (Article 55(1)(c))
Regulatory Requirement
Article 55(1)(c) requires providers to:
"track, document and report, without undue delay, to the AI Office and, as appropriate, to national competent authorities, relevant information about serious incidents and possible corrective measures to address them"
Defining "Serious Incidents"
While the AI Act defines serious incidents in the context of high-risk AI systems, for GPAI models with systemic risk, serious incidents likely include:
| Incident Category | Examples | Reporting Priority |
|---|---|---|
| Actual Systemic Harm | Documented mass misinformation campaign, demonstrated critical infrastructure attack | Immediate |
| Near-Miss Events | Prevented large-scale attack, circumvented safeguard exploitation | High |
| Significant Capability Failures | Major safety system bypass discovered | High |
| Novel Risk Emergence | Unexpected dangerous capability identified | Medium-High |
| Third-Party Reports | External researcher identifies critical vulnerability | Medium |
Incident Management Framework
| Phase | Timeline | Activities | Documentation |
|---|---|---|---|
| Detection | Immediate | Monitoring alerts, user reports, researcher disclosure | Initial incident log |
| Triage | < 1 hour | Severity assessment, classification, escalation decision | Triage record |
| Containment | < 24 hours | Immediate mitigation measures, access restrictions if needed | Containment actions |
| Investigation | 1-7 days | Root cause analysis, scope assessment, impact evaluation | Investigation report |
| Reporting | "Without undue delay" | AI Office notification, supporting documentation | Formal incident report |
| Remediation | Ongoing | Corrective measures, prevention improvements | Remediation record |
| Closure | Post-remediation | Effectiveness verification, lessons learned | Closure report |
AI Office Reporting Requirements
| Report Element | Content Requirements |
|---|---|
| Incident Description | What occurred, when discovered, how detected |
| Affected Scope | Geographic reach, user impact, downstream systems |
| Systemic Risk Connection | How incident relates to systemic risk potential |
| Immediate Response | Containment and mitigation actions taken |
| Root Cause | Underlying technical or procedural failure |
| Corrective Measures | Planned or implemented remediation |
| Prevention Measures | Changes to prevent recurrence |
Expert Insight
"Without undue delay" is context-dependent. For serious incidents with ongoing harm potential, this means hours, not days. Establish clear internal escalation procedures and pre-approve notification templates to enable rapid response.
Obligation 4: Cybersecurity Protection (Article 55(1)(d))
Regulatory Requirement
Article 55(1)(d) requires providers to:
"ensure an adequate level of cybersecurity protection for the general-purpose AI model with systemic risk and the physical infrastructure of the model"
Scope of Cybersecurity Obligations
"Adequate" cybersecurity encompasses both the model itself and supporting infrastructure:
| Protection Domain | Components | Key Threats |
|---|---|---|
| Model Security | Weights, architecture, fine-tuning procedures | Model theft, adversarial manipulation, extraction attacks |
| Training Infrastructure | Training clusters, datasets, pipelines | Data poisoning, compute compromise |
| Serving Infrastructure | API endpoints, inference servers, load balancers | Service disruption, unauthorised access |
| Development Environment | Code repositories, CI/CD pipelines, testing environments | Supply chain attacks, insider threats |
| Data Storage | Training data, evaluation data, user data | Data breach, regulatory violations |
Cybersecurity Framework Alignment
Consider alignment with established frameworks:
| Framework | Applicability | Key Areas |
|---|---|---|
| ISO 27001 | Information security management | Risk management, access control, incident management |
| NIST CSF | Cybersecurity risk management | Identify, Protect, Detect, Respond, Recover |
| SOC 2 | Service organisation controls | Security, availability, processing integrity, confidentiality |
| CSA STAR | Cloud security | Cloud-specific security controls |
Model-Specific Security Measures
| Threat | Protection Measures | Monitoring |
|---|---|---|
| Model Extraction | Rate limiting, output perturbation, access monitoring | Query pattern analysis |
| Weight Theft | Encryption at rest and transit, access logging, hardware security | Access anomaly detection |
| Adversarial Inputs | Input validation, anomaly detection, robustness testing | Real-time input analysis |
| Prompt Injection | Input sanitisation, context separation, privilege minimisation | Prompt pattern monitoring |
| Fine-tuning Attacks | Access controls, modification logging, integrity verification | Change detection |
Physical Infrastructure Security
| Area | Security Measures |
|---|---|
| Data Centres | Physical access controls, environmental monitoring, redundancy |
| Network | Segmentation, encryption, intrusion detection |
| Compute Resources | Secure boot, hardware security modules, resource isolation |
| Backup Systems | Encrypted backups, geographic distribution, tested recovery |
Implementation Roadmap
Pre-Systemic Risk Status Preparation
| Activity | Timeline | Priority |
|---|---|---|
| Establish adversarial testing capability | 3-6 months before threshold | Critical |
| Develop risk assessment methodology | 2-4 months before threshold | Critical |
| Implement incident management system | 2-3 months before threshold | High |
| Conduct cybersecurity assessment | 2-3 months before threshold | High |
| Create documentation templates | 1-2 months before threshold | Medium |
| Train relevant personnel | 1 month before threshold | Medium |
Ongoing Compliance Activities
| Activity | Frequency | Responsible |
|---|---|---|
| Adversarial testing cycles | Quarterly minimum | Safety/Red team |
| Risk assessment updates | Bi-annual or on significant changes | Risk team |
| Incident response drills | Annual | Security team |
| Cybersecurity audits | Annual | Security/External auditor |
| AI Office reporting | As required | Compliance |
| Documentation updates | Continuous | All teams |
Compliance Checklist
Model Evaluation (Article 55(1)(a))
- Established adversarial testing programme with documented protocols
- Testing covers state-of-the-art techniques appropriate to model capabilities
- Vulnerability register maintained and updated
- Mitigation measures documented for identified risks
- Evaluation methodology documentation available for AI Office review
- External red team engagement (recommended)
Systemic Risk Assessment (Article 55(1)(b))
- Risk assessment methodology established and documented
- All six systemic risk categories assessed
- Value chain risks (development, market, use) evaluated
- Mitigation measures identified for high-priority risks
- Residual risk documented and accepted
- Assessment updated for significant changes
Incident Management (Article 55(1)(c))
- Incident classification criteria defined
- Escalation procedures documented
- AI Office reporting templates prepared
- Response team identified and trained
- Incident logging system operational
- Post-incident review process established
Cybersecurity (Article 55(1)(d))
- Model security measures implemented
- Infrastructure security assessed and hardened
- Access controls and logging in place
- Security monitoring operational
- Incident response capability tested
- Third-party security assessment conducted (recommended)
What You Learned
Key concepts from this chapter
Article 55 imposes **four additional obligations** on systemic risk GPAI providers: model evaluation, risk assessment, incident management, and cybersecurity
**Adversarial testing** must reflect the "state of the art"—superficial testing will not satisfy regulators
**Systemic risk assessment** must address Union-level impacts across democratic, infrastructure, health, safety, economic, and rights domains
**Incident reporting** must be "without undue delay"—establish processes enabling rapid notification
**Cybersecurity** encompasses both the model and supporting infrastructure—comprehensive protection is required
Chapter Complete
GPAI Compliance
4/9
chapters