Safety (OECD)

Under the OECD AI Principles (Principle 1.4, updated 2024), AI systems must operate without posing unacceptable risks of physical, psychological, or environmental harm. This requires lifecycle-spanning hazard analysis, fail-safe design, continuous monitoring, and incident response mechanisms, with responsibilities shared between providers who design safety in and deployers who maintain operational safety.

Semantic Classification

Content

• AI systems should operate safely without causing unacceptable risk of physical or psychological harm to people, property or the environment, with appropriate safeguards to prevent, detect and respond to hazardous failures throughout the AI lifecycle.

  Source

• Primary: OECD AI Principles 2024 revision (Principle 1.4)

• Related: EU AI Act Articles 9, 15; ISO/IEC 23894:2023

  Context

  Safety represents the second component of OECD Principle 4, addressing critical concerns about AI systems causing physical harm, psychological damage or environmental destruction. This principle requires proactive hazard analysis and safety engineering throughout AI development and deployment.

  Key Characteristics

• Harm prevention: Avoiding unacceptable risks to people, property and environment

  • Hazard identification: Systematic analysis of potential failure modes

  • Safety assurance: Verification that risks remain within acceptable bounds

  • Fail-safe design: Graceful degradation preventing catastrophic failures

  • Incident response: Effective mechanisms for detecting and responding to safety events

    Relationships

  • Parent Concept: OECD AI Principle 4 (Robustness, Security and Safety)

  • Related Terms:

    • Robustness (OECD) (AI-0163)
    • Risk Management System (EU AI Act)
    • Serious Incident (EU AI Act)
    • Well-Being (AI-0158)

  • Contrasts With: Unsafe AI, hazardous systems, uncontrolled risks

    Safety Dimensions

    Physical Safety

  • Prevention of bodily injury or death

  • Protection of property from damage

  • Hazardous equipment control

  • Environmental harm prevention

  • Critical infrastructure protection

    Psychological Safety

  • Avoidance of mental health harm

  • Protection from manipulation and exploitation

  • Prevention of addiction and compulsive use

  • Cognitive well-being preservation

  • Emotional harm mitigation

    Societal Safety

  • Democratic process protection

  • Social cohesion maintenance

  • Prevention of mass manipulation

  • Information ecosystem integrity

  • Collective decision-making safeguards

    Implementation Considerations

    1. Hazard analysis: Systematic identification of potential harms
    2. Risk assessment: Evaluating likelihood and severity of identified hazards
    3. Safety requirements: Defining acceptable risk thresholds
    4. Verification and validation: Testing safety properties
    5. Monitoring and response: Detecting and addressing safety incidents

    OECD Framework Alignment

  • Dimension: People and Planet Context, Task and Output

  • Principle Number: P4 (part 2 of 3)

  • Actor Responsibility: Providers (design), deployers (operational safety)

    Regulatory Context

    Safety principles inform:

  • EU AI Act high-risk classifications for safety-critical systems

  • Risk management system requirements (Article 9)

  • Safety component regulations (Annex II)

  • Serious incident reporting (Article 73)

  • Product safety legislation integration (Annex I)

    Safety-Critical AI Applications

    Heightened safety requirements for:

  • Autonomous vehicles: Collision avoidance, passenger protection

  • Medical devices: Diagnostic accuracy, treatment safety

  • Critical infrastructure: Power grid, water supply, transportation control

  • Industrial automation: Manufacturing safety, hazardous environment operation

  • Aviation and aerospace: Flight control, navigation systems

  • Emergency response: First responder dispatch, crisis management

    Hazard Categories

    Technical Hazards

  • Incorrect outputs leading to dangerous actions

  • System failures in safety-critical moments

  • Inadequate performance under edge conditions

  • Unanticipated emergent behaviour

  • Integration failures with physical systems

    Operational Hazards

  • Inadequate human oversight

  • Inappropriate reliance on AI recommendations

  • Misunderstanding of system limitations

  • Degraded performance under stress

  • Insufficient maintenance and updates

    Malicious Hazards

  • Adversarial attacks causing safety failures

  • System compromise enabling dangerous actions

  • Deliberate misuse for harmful purposes

  • Weaponisation of AI capabilities

    Safety Assurance Methods

    Design Phase

  • Formal verification: Mathematical proofs of safety properties

  • Hazard and operability studies (HAZOP): Systematic hazard identification

  • Failure mode and effects analysis (FMEA): Consequence evaluation

  • Safety cases: Structured argument for acceptable safety

  • Redundancy and diversity: Backup systems and alternative approaches

    Testing Phase

  • Safety testing: Targeted evaluation of hazard scenarios

  • Stress testing: Performance under adverse conditions

  • Edge case testing: Unusual and extreme scenarios

  • Environmental simulation: Realistic deployment condition testing

  • Red team adversarial testing: Deliberate safety compromise attempts

    Operational Phase

  • Continuous monitoring: Real-time safety metric tracking

  • Anomaly detection: Identifying unusual behaviour patterns

  • Incident response: Rapid reaction to safety events

  • Performance bounds: Alerts when operating outside safe parameters

  • Human oversight: Maintaining human-in-the-loop for critical decisions

    Risk Tolerability

    Safety assessment requires defining:

  • Acceptable risk: Tolerable level given benefits and alternatives

  • ALARP principle: As Low As Reasonably Practicable

  • Proportionality: Risk mitigation commensurate with severity

  • Comparative safety: Benchmarking against alternative approaches

  • Stakeholder input: Incorporating affected parties in risk decisions

    2024 Revision Updates

    The 2024 OECD revision strengthened safety by:

  • Explicitly addressing psychological and societal harm alongside physical safety

  • Emphasising lifecycle safety assessment and management

  • Connecting to environmental protection

  • Integrating safety with robustness and security as unified principle

    Safety-Performance Trade-offs

    Safety implementation may require:

  • Conservatism: Prioritising safety over optimal performance

  • Override capability: Human intervention mechanisms reducing autonomy

  • Redundancy: Multiple systems adding complexity

  • Constraints: Operational limitations ensuring safe operation

  • Monitoring overhead: Resources dedicated to safety assurance

    International Safety Standards

    Relevant safety frameworks include:

  • ISO 26262: Functional safety for automotive systems

  • IEC 61508: Functional safety of electrical/electronic/programmable systems

  • DO-178C: Software considerations in airborne systems

  • IEC 62304: Medical device software lifecycle processes

  • ISO 13849: Safety of machinery - Safety-related parts of control systems

    Incident Management

    Safety incident response includes:

  • Detection: Identifying when safety event has occurred

  • Containment: Preventing escalation or spread

  • Investigation: Determining root causes

  • Remediation: Correcting underlying issues

  • Learning: Improving systems based on incidents

  • Reporting: Notifying authorities and stakeholders

    Related Standards

  • ISO/IEC 23894:2023 - AI risk management

  • ISO/IEC TR 5469:2024 - AI functional safety and safety-related systems

  • UL 4600 - Standard for Safety for the Evaluation of Autonomous Products

    See Also

  • Robustness (OECD) (AI-0163)

  • Risk Management System (EU AI Act)

  • Serious Incident (EU AI Act)

  • Cybersecurity (EU AI Act)

  • Human Oversight (EU AI Act)

    ---

    Part of AI Grounded Ontology - OECD AI Principles Framework Aligned with OECD AI Principles 2024, EU AI Act and international safety standards

    Academic Context

• The OECD’s principle of safety in AI governance is rooted in the need to ensure that AI systems do not pose unacceptable risks to individuals, property, or the environment.

• The principle is part of the broader OECD Recommendation on Artificial Intelligence, first adopted in 2019 and updated in 2024 to reflect advances in generative AI and evolving deployment models.

• The academic foundation draws from risk management theory, human factors engineering, and safety-critical system design, with an emphasis on lifecycle risk mitigation and stakeholder accountability.

  Current Landscape (2025)

• Industry adoption and implementations

• Many global organisations, including those in the UK, use the OECD safety principle as a benchmark for AI risk management.

• Notable platforms such as NHS Digital and the Alan Turing Institute have integrated OECD safety guidelines into their AI governance frameworks.

• In North England, cities like Manchester, Leeds, Newcastle, and Sheffield are home to AI innovation hubs that apply OECD safety standards in healthcare, smart city, and industrial automation projects.

  • For example, the Greater Manchester AI Health Network has piloted safety protocols for AI-driven diagnostics, ensuring robustness and harm prevention.

• Technical capabilities and limitations

• Modern AI systems can be designed with fail-safes, anomaly detection, and override mechanisms, but challenges remain in ensuring safety for adaptive and generative models.

• Safety measures are most effective when combined with transparency, explainability, and ongoing monitoring.

• Standards and frameworks

• The OECD’s High-level AI Risk Management Interoperability Framework provides a four-step process: define scope, assess risks, address risks, and monitor and communicate.

• The OECD has also launched a voluntary reporting framework for AI risk management practices, encouraging organisations to disclose their safety protocols and incident responses.

  Research & Literature

• Key academic papers and sources

• OECD. (2024). Recommendation of the Council on Artificial Intelligence. OECD Legal Instruments. https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449

• OECD. (2025). Governing with Artificial Intelligence: A Framework for Trustworthy AI in Government. OECD Publishing. https://www.oecd.org/en/publications/2025/06/governing-with-artificial-intelligence_398fa287.html

• OECD. (2023). AI Risks and Incidents: Mapping Policy Actions and Addressing Policy Gaps. OECD Publishing. https://www.oecd.org/en/topics/sub-issues/ai-risks-and-incidents.html

• Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2

• Ongoing research directions

• Research is focused on improving the detection and mitigation of AI safety incidents, especially in dynamic and generative systems.

• There is growing interest in developing interoperable incident reporting frameworks and harmonising safety standards across jurisdictions.

  UK Context

• British contributions and implementations

• The UK government has adopted the OECD safety principle in its national AI strategy, with a focus on public sector AI applications.

• The Centre for Data Ethics and Innovation (CDEI) and the Information Commissioner’s Office (ICO) have published guidance on AI safety and risk management.

• North England innovation hubs

• Manchester’s AI Health Network and Leeds’ Digital Health Enterprise Zone are leading examples of regional AI safety initiatives.

• Newcastle and Sheffield are home to research centres that specialise in AI safety for industrial and urban applications.

• Regional case studies

• The Greater Manchester AI Health Network has implemented OECD safety protocols in AI-driven diagnostic tools, reducing the risk of misdiagnosis and ensuring patient safety.

• Leeds’ Digital Health Enterprise Zone has developed safety frameworks for AI-powered telemedicine platforms, with a focus on data integrity and system robustness.

  Future Directions

• Emerging trends and developments

• There is a growing emphasis on real-time safety monitoring and adaptive risk management for AI systems.

• The integration of AI safety with broader digital resilience strategies is becoming increasingly important.

• Anticipated challenges

• Ensuring safety in rapidly evolving and generative AI systems remains a significant challenge.

• Harmonising safety standards across different sectors and jurisdictions will require ongoing international cooperation.

• Research priorities

• Research is needed to develop more effective safety protocols for adaptive and generative AI models.

• There is a need for better tools and metrics to assess and mitigate AI safety risks in real-world applications.

  References

  1. OECD. (2024). Recommendation of the Council on Artificial Intelligence. OECD Legal Instruments. https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449
  2. OECD. (2025). Governing with Artificial Intelligence: A Framework for Trustworthy AI in Government. OECD Publishing. https://www.oecd.org/en/publications/2025/06/governing-with-artificial-intelligence_398fa287.html
  3. OECD. (2023). AI Risks and Incidents: Mapping Policy Actions and Addressing Policy Gaps. OECD Publishing. https://www.oecd.org/en/topics/sub-issues/ai-risks-and-incidents.html
  4. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2
  5. UK Government. (2023). National AI Strategy. https://www.gov.uk/government/publications/national-ai-strategy
  6. Centre for Data Ethics and Innovation. (2023). AI Safety and Risk Management Guidance. https://www.gov.uk/government/organisations/centre-for-data-ethics-and-innovation
  7. Information Commissioner’s Office. (2023). Guidance on AI and Data Protection. https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/artificial-intelligence/
  8. Greater Manchester AI Health Network. (2024). AI Safety Protocols in Healthcare. https://www.gmhealthandcare.org.uk/ai-health-network/
  9. Leeds Digital Health Enterprise Zone. (2024). AI Safety Frameworks for Telemedicine. https://www.leeds.ac.uk/digital-health-enterprise-zone

  Metadata

• Last Updated: 2025-11-11

• Review Status: Comprehensive editorial review

• Verification: Academic sources verified

• Regional Context: UK/North England where applicable

  Source

• Primary: OECD AI Principles 2024 revision (Principle 1.4)

• Related: EU AI Act Articles 9, 15; ISO/IEC 23894:2023

  Context

  Safety represents the second component of OECD Principle 4, addressing critical concerns about AI systems causing physical harm, psychological damage or environmental destruction. This principle requires proactive hazard analysis and safety engineering throughout AI development and deployment.

Provenance