Best In Class

Overview

This table highlights the best-performing example for each specific criterion in the risk-management framework, showing what "good practice" looks like across the industry, even though no single company excels everywhere.

  • Risk identification – through literature review, open-ended red-teaming, and risk modeling
  • Risk analysis and evaluation – using quantitative metrics and clearly defined thresholds
  • Risk treatment – implementing mitigation measures including containment, deployment controls, and assurance processes
  • Risk governance – establishing clear organizational structures and accountability
1.1Classification of Applicable Known Risks
1.1.1Risks from literature and taxonomies are well covered
  • The company covers the main risks in the literature. Among others, these risks include: cybersecurity; chemical, biological, radiological and nuclear (CBRN); harmful manipulation; loss of control; autonomous AI R&D.
  • The company further breaks down “loss of control risks”. E.g. into instrumental reasoning / autonomous replication / deception and scheming.
  • The company references taxonomies or literature that informs their risk identification process, or some other justification for how they selected risk domains to show they do not miss risk domains experts highlight
  • Best practice: (Top score: 90% – Microsoft)
    • Microsoft covers the risks that they believe could emerge over the short-to-medium term and threaten national security or pose at-scale public safety risks if not appropriately mitigated. They cover the four risk areas from the EU AI Act’s Code of Practice, including Chemical, Biological, Radiological, and Nuclear (CBRN), Cyber offense, Loss of control, and harmful manipulation. The framework also covers advanced autonomy, which includes automated AI R&D.
1.1.2Exclusions are clearly justified and documented
  • The company provides strong justification for exclusion of risks, referring to at least one of: academic literature/scientific consensus; internal threat modeling with transparency; third-party validation, with named expert groups and reasons for their validation.
  • The company identifies categories that are not currently monitored but could be monitored in the future, along with the precise criteria these categories must meet to qualify for monitoring.
  • Best practice: (Top score: 90% – Microsoft)
    • The framework states that it covers risks that “threaten national security or pose at-scale public safety risks if not appropriately mitigated”, demonstrating their exclusion criteria. Microsoft also specifies that risks that are not covered in this framework are covered by other frameworks. The framework makes reference to some of these non-covered risks, such as bias.
1.2Identification of Unknown Risks (Open-ended red teaming)
1.2.1Internal open-ended red teaming
  • The company commits to an internal process dedicated to identifying unknown risks that could arise from the model. This process occurs pre-deployment for frontier models. This process could identify either novel risk domains, or novel risk models/changed risk profiles within pre-specified risk domains.
  • The company gives detail on the resources, time and access given to the internal open-ended red team, and justification for why this is adequate.
  • The company gives detail on the expertise required to properly identify hazards, and details why their internal open-ended red team meets this criteria.

Best practice: (Top score: 50% – Microsoft)

  • The framework acknowledges the uncertainty regarding AI capabilities and states that the company performs “open-ended red teaming, research, horizon scanning, conversations with external experts, literature reviews, and incident monitoring” to study risks that might emerge.
1.2.2Third party open-ended red teaming
  • The company commits to an external process dedicated to identifying unknown risks that could arise from the model. This process occurs pre-deployment. This process could identify either novel risk domains or novel risk models within pre-specified risk domains.
  • The company gives detail on the resources, time and access given to the external open-ended red team, and justification for why this is adequate.
  • The company gives detail on the expertise required to properly identify hazards, and details why their external open-ended red team meets this criteria.

Best practice (Top score: 25% – Meta)

  • Meta’s framework notes that new risks may be identified through threat modeling and that this involves external experts, but it does not directly reference third-party red teaming. They also mention workshops that include external subject matter experts.
  • Meta commits to involving external experts in pre-deployment threat modeling, evaluation design, and red teaming for cyber and chemical/biological risks.
1.3Risk modeling
1.3.1The company uses risk models for all the risk domains identified
and the risk models are published (with potentially dangerous
information redacted)
  • For each considered risk domain, the company has developed and published a risk model, including evidence of efforts to systematically map out all possible risk pathways, and potentially dangerous information.
  • The following are also published: risk modeling methodology, experts involved, list of identified scenarios.

Best practice: (Top score: 75% – Meta, Microsoft)

  • Meta: Risk modelling is clearly conducted for each risk domain. The list of threat scenarios are published for each risk domain, whilst keeping generality for security reasons. There is a clear reliance on risk modeling for determining “whether models with new capabilities may pose novel risk”.
  • Microsoft: The framework includes the use of risk modeling for each high-risk capability, and these include “threat actors, unique ways an AI model may increase or decrease risk, technical bottlenecks, and the severity and likelihood of the scenario resulting in significant impacts”. However, they do not publish the redacted risk models.
1.3.2Risk modeling methodology
1.3.2.1Methodology precisely defined
  • The company commits to a structured process for modeling and systematically exploring risks, including a precise methodology such as event trees, fault trees, or Fishbone diagrams. 
  • The risk models use this methodology to break down complex risk pathways into discrete, measurable steps.

Best practice: (Top score: 50% – Meta)

  • Meta provides a defined threat modeling methodology, identifying catastrophic outcomes to prevent, and then mapping potential causal pathways that could produce them, including the definition of a range of threat actors and scenarios.
1.3.2.2Mechanism to incorporate red teaming findings

Novel risks or risk pathways identified via open-ended red teaming or any other evaluations trigger updates on risk modeling and scenario analysis, potentially in multiple scenarios. For instance, encountering evidence of instrumental reasoning via open-ended red teaming likely requires updates to multiple risk models.

Best practice: (Top score: 25% – Meta) 

  • Meta explicitly states that novel catastrophic outcomes, threat scenarios, and emerging capabilities identified through threat modeling, evaluations, incidents, near misses, or deployment experience trigger the creation of new assessments and updates to the framework. It commits to periodically revising catastrophic outcomes, threat scenarios, and evaluation methodologies. Preparedness reports are also updated when new evidence materially changes prior risk assessments.
1.3.2.3Prioritization of severe and probable risks
  • The company develops risk models from the full space of scenarios, assigning severities (quantitative, semi-quantitative or qualitative) and probabilities, based on the best estimates at the time, if mitigations are not undertaken. 
  • The severity and probability scores are published. 
  • The most severe probable risk models are prioritised as focus areas.

Best practice: (Top score: 75% – Cohere)

  • Cohere demonstrates clear assessment and prioritisation based on severity and probability: first assessing potential risks from model capabilities and deployment contexts, then evaluating the likelihood and severity of potential harms.
  • Resources are focused on risks with a high likelihood of occurring based on LLM task performance and inherent model limitations, drawing from the full space of identified risk models.
1.3.3Third party validation of risk models
  • The company asks third parties with relevant expertise to review risk models, being accountable for their opinion on the final risk models.
  • If risk models are not reviewed externally, justification for internal expertise or lack of external expertise is given.

Best practice: (Top score: 50% – Anthropic, G42)

  • Anthropic: They reference third-party validation for security standards, including independent validation of threat modelling
  • G42: They describe collaboration with external AI safety experts, specifically METR and SaferAI.
2.1Setting a Risk Tolerance
2.1.1Risk tolerance is defined
2.1.1.1Risk tolerance is at least qualitatively defined for all risks
  • The company clearly and explicitly sets out a risk tolerance, i.e., the maximum amount of risk the company is willing to accept for each risk domain. For example, this could be expressed as economic damage for cybersecurity risks and as number of fatalities for chemical and biological risks.
  • This risk tolerance may be qualitative, e.g. a scenario.

Best practice (Top score: 75% – Meta) 

  • Meta outlines the “catastrophic outcomes” the provider must strive to prevent in each risk domain, providing a qualitative description that functions as implicit risk tolerance definition.
  • They include an abstract definition of risk tolerance, preventing model release if they provide “significant uplift” toward the execution of a threat scenario that could produce a catastrophic outcome. This effectively establishes a qualitative risk tolerance, even if not labelled as such.
2.1.1.2Risk tolerance is expressed at least partly quantitatively as a combination of scenarios (qualitative) and probabilities (quantitative) for all risks

The company sets risk tolerance for each risk domain, including quantitative probabilities.

Best practice (Top score: 10% – OpenAI; Meta) 

  • OpenAI: They include a definition of severe harm that implies some awareness of quantitative measurement, but to define critical capability thresholds not tolerance. 
  • Meta: They acknowledge an intent to quantify risks and benefits, signaling a recognition of the value of quantitative risk tolerance, though this has not yet been operationalized.
2.1.1.3Risk tolerance is expressed fully quantitatively as a product of severity (quantitative) and probability (quantitative) for all risks
  • The company sets the risk tolerance for each risk domain is fully quantitative, as a product of severity and probability.
  • The company applies the same risk tolerance across all risk domains.

Best practice (Top score: None) 

  • No framework currently expresses risk tolerance as a fully quantitative product of severity and probability.
2.1.2Process to define the tolerance
2.1.2.1AI developers engage in public consultations or seek guidance from regulators where available
  • The company creates a structured process for seeking public input into risk tolerances.
  • There are some conditions under which input from regulators into risk tolerances is required.

Best practice (Top score: 10% – Google DeepMind) 

  • Google DeepMind references drawing on “relevant high-quality research” and “information shared through industry forums” to inform its critical capability levels, which function as risk tolerance tiers. This represents a limited form of external input into the risk tolerance-setting process.
2.1.2.2Any significant deviations from risk tolerance norms established in other industries is justified and documented (e.g., cost-benefit analyses)

The company gives justification if risk tolerance is higher than in other industries (such as nuclear or aviation), such as through a cost-benefit analysis which shows why benefits appropriately offset excess risk.

Best practice (Top score: 10% – Google Deep Mind) 

  • The framework introduces sections covering a discussion about proportionality at large. They explicitly weigh security against innovation costs.
2.2Operationalizing Risk Tolerance
2.2.1Key Risk Indicators (KRI)
2.2.1.1KRI thresholds are at least qualitatively defined for all risks
  • KRI thresholds indicate when additional mitigations become necessary. The primary KRI for frontier AI is model capability (e.g. benchmark performance), but external KRIs (e.g. threat actor capabilities, availability of elicitation techniques) can also be monitored.
  • The company defines at least one qualitative Key Risk Indicator (KRIs) for each risk domain and corresponding risk tolerance. KRIs are a measurable signal serving as a proxy for risk level. 

Best practice (Top score: 75% – Google DeepMind; Meta) 

  • Google DeepMind: Each risk domain has one or several KRIs, which are qualitatively defined. The KRIs appear to be grounded in risk modeling. Google DeepMind operationalizes risk tolerance through explicit capability thresholds, evaluation-triggered alert thresholds, domain-specific capability definitions, and ongoing early warning evaluations that function like KRIs.
  • Meta: The framework defines multiple qualitative KRIs across all major risk domains, grounded in explicit threat models and tied to concrete evaluation thresholds and escalation procedures.
2.2.1.2KRI thresholds are quantitatively defined for all risks
  • For each risk domain and corresponding risk tolerance, at least one KRI threshold is quantitatively defined. This KRI is not the risk tolerance itself, but a proxy for the risk tolerance that can be measured pre-deployment to indicate that the risk level may exceed the risk tolerance without further mitigation.
  • The company sets KRI measurable enough to be quantitative, e.g. it is a benchmark but no threshold is yet given. This is because KRIs which are measurable are preferred to KRIs which are more vague, assuming both are grounded in risk modeling.

Best practice (Top score: 50% – Meta) 

  • The framework defines multiple KRIs using quantitatively measurable thresholds across all major risk domains, including cybersecurity, chemical/biological risks, and loss of control. These KRIs are operationalized through benchmark performance levels, refusal-rate thresholds, and concrete capability checkpoints that act as pre-deployment escalation triggers tied to mitigation and deployment decisions.
2.2.1.3KRIs also identify and monitor changes in the level of risk in the external environment

KRIs include measurable indicators of external risk beyond model capabilities evaluations, such as increased use of AI for successful cyberattacks, the population the AI is deployed to, or improved scaffolding, as a trigger for suitable KCIs.

Best practice (Top score: 25% – Google DeepMind, Meta, Microsoft) 

  • The frameworks reference assessments of a diversity of external factors that can affect the level of risks, such as real-world incidents, adversarial adaptation, or external-environment monitoring.
2.2.2Key Control Indicators (KCI)
2.2.2.1Containment KCIs
2.2.2.1.1All KRI thresholds have corresponding qualitative containment KCI thresholds
  • The KCI  is a measurable signal representing mitigation effectiveness. KCI thresholds specify the minimum mitigation level required when corresponding KRI thresholds are crossed. Examples include jailbreak success rates, security certification levels, or percentage of harmful requests refused.
  • Containment measures are security measures that control access to AI model weights and infrastructure. Examples include network isolation, access controls, insider threat programs, and encryption. Containment measures aim to prevent unauthorized parties (e.g. state actors, cybercriminals) from obtaining model weights or sensitive data.
  • The company defines qualitative Containment Key Control Indicators that work as a mitigation target or objective. If the KRI is reached, the KCI must be satisfied.
  • The company has one KCI for each KRI threshold

Best practice (Top score: 90% – G42) 

  • The framework defines clear qualitative KCIs, with detailed security measures tied to capability levels.
  • G42 requires its security level 2 (SML 2) for each KRI threshold that is triggered, providing a clear qualitative definition. ‘The model should be secured such that it would be highly unlikely that a malicious individual or organization could obtain model weights (…)’
  • G42 also defines additional security levels (SML 1, 3, and 4), demonstrating a structured and scalable containment framework
2.2.2.1.2All KRI thresholds have corresponding quantitative containment KCI thresholds
  • The company defines quantitative Containment Key Control Indicators that work as a mitigation target or objective. If the KRI is reached, the KCI must be satisfied.
  • The company has one KCI for each KRI threshold.
  • This can be thought of as the bar that containment measures must meet to keep residual risk below the risk tolerance, given a KRI is crossed.

Best practice (Top score: 10% – Google DeepMind, Anthropic)

  • Google DeepMind: They define increasingly stringent containment/security thresholds tied to KRI thresholds. The framework references RAND’s security levels as the relevant containment standard. These levels include partially quantitative criteria (e.g., defining attacker capability in terms of budget, time investment, and infrastructure).
  • Anthropic: The framework also mentions the RAND security level as an “industry-wide recommendation”. When Anthropic would be “in the lead” in AI capabilities, the framework states that they should see a strong argument that the catastrophic risk would be contained before release.
2.2.2.2Deployment KCIs
2.2.2.2.1All KRI thresholds have corresponding qualitative deployment KCI thresholds
  • Deployment measures are safeguards that mitigate risks from unauthorized use of deployed AI systems. Examples include input/output filters, safety fine-tuning, usage monitoring, refusal training, and know-your-customer policies. Deployment measures address both intentional misuse and accidental harms.
  • For each KRI threshold, a qualitative Deployment Key Control Indicator threshold is described such that if the KRI threshold is reached, then this KCI must be satisfied.
  • This can be thought of as the bar that deployment measures must meet to keep residual risk sufficiently below the risk tolerance, given a KRI is crossed.

Best practice (Top score: 75% – OpenAI) 

  • OpenAI defines three general deployment KCIs, distinguishing targets for safeguards to reach for misuse risks: Robustness (jailbreak resistance), Usage Monitoring (detecting harmful actions), and Trust-based Access (restricting access to vetted users).For example, Robustness is set as ‘Malicious users cannot use the model to cause the severe harm because they cannot elicit the necessary capability, such as because the model is modified to refuse to provide assistance to harmful tasks and is robust to jailbreaks that would circumvent those refusals.’
  • These KCIs show nuance and expertise in their qualitative detail, specifying the types of safeguards expected at different capability levels.
2.2.2.2.2All KRI thresholds have corresponding quantitative deployment KCI thresholds

For each KRI threshold, a quantitative Deployment Key Control Indicator threshold is described such that if the KRI threshold is reached, then this KCI must be satisfied.

Best practice (Top score: 25% – Meta) 

  • The framework contains several genuinely quantitative deployment-related thresholds tied to mitigation adequacy. Quantitative deployment thresholds exist in specific domains (especially Cyber, CBRN, and Loss of Control).
2.2.2.3For advanced KRIs, assurance process KCIs are defined
  • Assurance processes provide affirmative evidence that an AI model will not cause harm, even when the model has dangerous capabilities. They can be thought of as safeguards against misuse, or control setups. Examples include advanced interpretability techniques to detect deception, propensity measurement, or formal verification methods. Assurance processes become necessary when capability evaluations alone cannot demonstrate the absence of risk.
  • For each advanced KRI threshold, an assurance process KCI threshold is described such that if the KRI threshold is reached, then this KCI must be satisfied. A KRI threshold is advanced if the associated risk model is a result of the AI’s actions, rather than a human misusing the AI. 
  • This can be thought of as the bar that assurance processes must meet to keep residual risk below the risk tolerance, given a KRI is crossed.

Best practice (Top score: 50% – OpenAI) 

  • The framework provides five desiderata that function as proto-assurance KCIs, including requirements such as the model “consistently understands instructions” and concerns being “sufficiently addressed” by specific claims. These include: Lack of Autonomous Capability, Value Alignment, Instruction Alignment, Reliable and Robust System Oversight, and System Architecture.
2.2.3Pairs of thresholds are grounded in risk modeling to show that risks remain below the tolerance
  • Altogether, for each risk domain, the company provides justification that each KRI-KCI pairing is sufficient to keep residual risk below the risk tolerance, given the KRI threshold is crossed but the KCI is satisfied. The justification has some quantified confidence level (and a possible safety margin, to allow for error).
  • Reasoning behind this confidence is given via risk modeling, akin to an adequately quantified safety case, combining both empirical evidence and argumentation. Discrete, measurable steps are combined to show (qualitatively or quantitatively) that residual risk is sufficiently below the risk tolerance. 
  • The assessment of the adequacy of this pairing should not be relative to other companies’ risk tolerance.

Best practice (Top score: 25% – Anthropic, OpenAI, Google DeepMind, Meta, G42, Amazon, Nvidia) 

  • Using a range of systems, companies show a basic incorporation of risk modeling thinking, breaking down the process of how capabilities can lead to real-world scenarios, to define thresholds to show that risks remain below the tolerance.
2.2.4Policy to put development on hold if the required KCI threshold cannot be achieved, until sufficient controls are implemented to meet the threshold
  • For any KRI/KCI pair, if the KCI threshold cannot be shown to have been satisfied, then the company commits to pausing development and/or deployment until sufficient controls are implemented to meet this threshold.
  • The company details processes, or credible plans to develop processes, for pausing development before unacceptable risk levels are manifest. 
  • The company details processes for de-deployment.

Best practice (Top score: 50% – Anthropic; OpenAI ) 

  • Anthropic: Explicit commitments to find interim solutions in case the KCI thresholds are not met, including the CEO and the Responsible Scaling Officer, who may approve the use of interim measures that provide the same level of assurance as the relevant security standard
  • OpenAI: There is a clear statement that if the Critical safeguards threshold is not specified, then development will be halted. The process requires specification of Critical safeguards.
3.1Implementing Mitigation Measures
3.1.1Containment measures
3.1.1.1Containment measures are precisely defined for all KCI thresholds

The planned or already implemented containment measures are precisely defined for all containment KCI thresholds.

Best practice (Top score: 90% – Amazon)

  • Containment measures are described with substantial detail, precision, and nuance across multiple dimensions in nearly three pages. 
  • While measures are not explicitly mapped to each KCI threshold, the framework assumes implementation across all current models and those crossing critical capability thresholds.
3.1.1.2Proof that containment measures are sufficient to meet the thresholds
  • The containment measures provide proof that they satisfy the relevant containment KCI threshold. At least evidence for why they believe these measures are likely to satisfy the KCI threshold is given with confidence levels.
  • The company offers a process for soliciting proof. 
  • The sufficiency criteria is determined ex ante. There is a justification for why this criteria is adequate proof.

Best practice (Top score: 50% – Amazon) 

  • Structured internal processes exist for reviewing and testing containment measures for sufficiency.
  • The use of formal methods to ensure correctness of security-critical components provides a foundation for generating sufficiency evidence.
3.1.1.3Strong third party verification process to verify that the containment measures meet the threshold (100% if 3.1.1.3 > [60% x 3.1.1.1 + 40% x 3.1.1.2])
  • There is an external structured process for proving that containment measures are sufficient to meet the relevant containment KCI, such as through a security audit, prior to its implementation (i.e., before the corresponding KRI threshold is crossed).
  • Detail is provided on how experts are chosen, with the following details: required expertise from experts and guarantee of independence
  • External reports are made available (with sensitive information redacted), to give a sense of the third parties confidence that the measures meet the threshold.

Best practice (Top score: 25% – Anthropic; OpenAI)

  • Anthropic: They describe comprehensive third-party assessment of containment measures within its ASL-3 framework, one of Anthropic’s designated security levels, and commits to soliciting input from external experts. However, these commitments are expressed as expectations rather than firm obligations.
  • OpenAI: requires independent audits for High capability models, though a comparable process for Critical capability models is not described.
3.1.2Deployment measures
3.1.2.1Deployment measures are precisely defined for all KCI thresholds

The planned or already implemented deployment measures are precisely defined for all deployment KCI thresholds.

Best practice (Top score: 50% – Anthropic; OpenAI)

  • Anthropic: They provide detailed deployment measures for ASL-2 and high-level criteria for ASL-3, as security levels defined by Anthropic, including evaluation principles such as “defence in depth.”
  • OpenAI: They detail ‘potential safeguards’ for High capability models across robustness, usage monitoring, and trust-based access, defined for each KCI threshold. However, the safeguards are presented as illustrative rather than as firm commitments.
3.1.2.2Proof that deployment measures are sufficient to meet the thresholds
  • There is a preemptive justification that the measures are sufficient to meet the relevant deployment KCI (e.g., this quantity of rejection finetuning would enable us to reach our target of 99.9 percent of jailbreak resistance, as shown by these experiments […]) The company establishes some process for soliciting such evidence. 
  • The company offers a process for soliciting proof. However, to gain marks over 50, the first item should be satisfied. 
  • The implementation of the KRI-KCI pairing is predictable in advance, leaving as little to discretion as possible.
  • The sufficiency criteria is determined ex ante. There is a justification for why this criteria is adequate proof.

Best practice (Top score: 25% – Anthropic; OpenAI)

  • Anthropic: They provide a high-level description of a red teaming process for evaluating whether deployment measures meet requirements.
  • OpenAI: They detail potential safeguard efficacy assessments in its appendix
  • In both cases, frameworks rely on internal judgment at the time deployment standards need to be implemented, rather than providing ex ante evidence.
3.1.2.3Strong third party verification process to verify that the deployment measures meet the threshold (100% if 3.1.2.3 > [60% x 3.1.2.1 + 40% x 3.1.2.2])
  • hat deployment measures are sufficient to meet the relevant deployment KCI, such as external red-teaming of safeguards. 
  • Detail is provided on how experts are chosen, with the following details: required expertise from experts and guarantee of independence
  • External reports are made available (with sensitive information redacted), to give a sense of the third parties confidence that the measures meet the threshold.

Best practice (Top score: 25% – OpenAI; G42)

  • OpenAI: They reference third-party stress testing of safeguards, though this is not specific to deployment measures and appears optional.
  • G42: They detail a process for soliciting external expert advice prior to deployment decisions.
3.1.3Assurance processes
3.1.3.1Credible plans towards the development of assurance processes
  • The company must acknowledge whether current assurance processes are insufficient to meet the required assurance process KCI. 
  • If insufficient, the company should articulate (a) at what KRIs the assurance processes become necessary, and (b) justification for why they believe they will have sufficient assurance processes by the time the relevant KRI is reached, including (c) technical milestones and estimates of when these milestones will need to be reached given forecasted capabilities growth

Best practice (Top score: 25% – Anthropic; OpenAI)

  • Anthropic: They acknowledge that assurance processes do not yet exist and commit to developing them, including creating an “affirmative case” identifying alignment risks and their mitigations.
  • OpenAI: They commit to developing assurance processes for Critical capabilities and to updating its framework accordingly.
3.1.3.2Evidence that the assurance processes are enough to achieve their corresponding KCI thresholds

Process is defined for how they will empirically determine/collect evidence to show that assurance processes are credible, such as via demonstration with model organisms or theoretical proofs.

Best practice (Top score: 50% – OpenAI)

  • The framework details potential safeguard efficacy assessments, which represent a positive step toward demonstrating that assurance properties meet KCI thresholds.
  • However, these assessments are presented as potential rather than committed measures, and descriptions remain high-level. The Critical capability case,  as that presents a meaningful risk of a qualitatively new threat vector for severe harm with no ready precedent, is not covered.
3.1.3.3The underlying assumptions that are essential for their effective implementation and success are clearly outlined
  • The company outlines the key technical assumptions underlying their current planned assurance approach. For instance, no undetected sandbagging, chain of thought is faithful, etc. 
  • There is justification for why these assumptions are valid, i.e. stress testing of these assumptions is performed and results are presented with confidence levels.

Best practice (Top score: 75% – Anthropic)

  • The risk report provides a list of potential pitfalls in alignment evaluations and assesses them separately, including awareness and realism, steganography ability, reasoning faithfulness and sandbagging. The claims are generally qualified with a qualitative confidence level and backed up with reasoning referencing evidence given in the report and system cards.
3.2Continuous Monitoring and Comparing Results with Pre-determined Thresholds
3.2.1Monitoring of KRIs
3.2.1.1Justification that elicitation methods used during the evaluations are comprehensive enough to match the elicitation efforts of potential threat actors
  • The company provides justification for why their elicitation methods will upper bound the resources and methods available to potential threat actors, for elicited capabilities and within concrete risk models.
  • There is detail on these elicitation methods, e.g. the amount of compute used for finetuning.

Best practice (Top score: 90% – OpenAI)

  • The framework outlines multiple elicitation strategies and commits to matching the elicitation efforts of potential threat actors, contemplated as the high-end adversarial in their threat model.
  • The elicitation methods described demonstrate nuance and technical expertise across different approaches, taking any one-time capability elicitation as the lower bound rather than the ceiling.
3.2.1.2Evaluation frequency
  • The company outlines some processes for regularly re-conducting evaluations with maximal elicitation, to factor in previously unaccounted post training enhancements or elicitation methods. 
  • The frequency should be based on both time (e.g., every 6 months) and according to scaling progress (effective computing power used in training triggers more advanced KRIs). 
  • The chosen frequency is justified with clear reasoning.

Best practice (Top score: 90% – Naver)

  • The framework mentions that they run evaluations with regular frequency, every 3 months; and by performance gains, when performance increases by 6x, whichever is sooner. They also mention that “the amount of computing can serve as an indicator when measuring capabilities”.
3.2.1.3Description of how post-training enhancements are factored into capability assessments
  • For KRIs that require capability assessments (evaluations), there is an explicit documentation of (a) the specific methodologies used to either incorporate post-training enhancements into capability measurements, and/or (b) the size of the safety/uncertainty margin in order to account for possible post-training enhancements that occur after evaluation is complete, with justification for the size of this margin based on forecasting exercises given the speed of progress of previous post-training enhancements. 
  • The uncertainty margin accounts for, or updates on, how post-training enhancements change with different model structures – namely, post-training enhancements are much more scalable with reasoning models, as inference compute can often be scaled to improve capabilities.

Best practice (Top score: 50% – Anthropic; Microsoft)

  • Anthropic: They account for widely accessible post-training enhancements in capability assessments, including headroom for both modifications via its own fine-tuning products, and scenarios where models are stolen.
  • Microsoft: They explicitly consider incorporating frontier post-training enhancements when re-evaluating models to ensure KRIs are not crossed unnoticed.
3.2.1.4Vetting of protocols by third parties
  • There is a process for independent third parties to review the internal methods for assessing KRIs status, including evaluation methodologies.
  • Detail is provided on how experts are chosen, with the following details: required expertise from experts and guarantee of independence

Best practice (Top score: 25% – G42; Magic)

  • G42 describes a process for gaining external input on evaluation protocols, actioning it when appropriate.
  • Magic describes expert input on the development of dangerous capability evaluations and requires Board approval for changes to KRI benchmarks, informed by external advisers.
3.2.1.5Replication of evaluations by third parties

There is a process for assessing KRIs assessment results externally (i.e., by independent third parties), to ensure that KRIs assessments are accurate. This could materialise as internal KRIs assessments being replicated by external parties (or audited), or KRIs assessments being outsourced to third parties

Best practice (Top score: 25% – OpenAI, Amazon)

  • OpenAI: The framework recognises evaluations being conducted independently by third-parties, and they commit to work with these parties. They commit to have evaluations replicated only as part of a deeper capability assessment (“Deep Dive”), when requested by the Safety Advisory Group.
  • Amazon: The framework commits to run external red-teaming, working with specialized firms and academics to evaluate risks related to critical capabilities.
3.2.2Monitoring of KCIs
3.2.2.1Detailed description of evaluation methodology and justification that KCI thresholds will not be crossed unnoticed
  • The framework describes systematic, ongoing monitoring to ensure mitigation effectiveness is tracked continuously such that the KCI threshold will be met, when required. 
  • There is a justification that threshold detection will fit within suitable confidence levels. The framework includes failure mode analysis or some other methodology to minimise chance of failure.

Best practice (Top score: 50% – Anthropic; OpenAI)

  • Anthropic: They provide a high-level description of monitoring procedures for deployment measures, with examples such as jailbreak bounties. They also mention that they will develop plans to audit the implementation of containment measures.
  • OpenAI: They make some reference to monitoring systems, implicitly assuming the alignment between KCIs and safeguards of models.
3.2.2.2Vetting of protocols by third parties

There is a process for independent third parties to review the methods for assessing the efficacy of KCIs measures.

Best practice (Top score: 50% – OpenAI)

  • The framework demonstrates a commitment to third-party vetting of KCI protocols, including for containment KCIs specifically.
3.2.2.3Replication of evaluations by third parties
  • There is a process for assessing KCIs internally and externally (i.e., by independent third parties), to ensure that KCIs assessments are accurate. 
  • Detail is provided on how experts are chosen, with the following details: required expertise from experts, and guarantee of independence.

Best practice (Top score: 25% – Anthropic, OpenAI)

  • Anthropic: The framework includes a statement that security controls are “independently reviewed to ensure effectiveness”, providing a high-level commitment to external containment KCI efficacy assessment without operational detail on required expertise or guarantee of independence.
  • OpenAI: The framework recognises evaluations being conducted independently by third-parties, and they commit to work with these parties. They commit to have evaluations replicated only as part of a deeper capability assessment (“Deep Dive”), when requested by the Safety Advisory Group.
3.2.3Transparency of evaluation results
3.2.3.1Sharing of evaluation results with relevant stakeholders as appropriate
  • If a KRI is crossed for any risk domain, the company commits to notifying regulators/the relevant government authorities in a timely manner. 
  • All KRIs and KCIs assessments (i.e., evaluations) are public, with predefined criteria

Best practice (Top score: 75% – Microsoft, OpenAI)

  • Microsoft: They commit to share substantial detail, seemingly with members of the Frontier Model Forum, on models’ KRI levels and corresponding KCI measures. There is also a commitment to publishing capabilities, evaluations and risk classification publicly.
  • OpenAI: They commit to share evaluation results to the public if models are deployed.
3.2.3.2Commitment to non-interference with findings

The company commits to permitting the reports, which detail the results of external evaluations (i.e. any KRIs or KCIs assessments conducted by third parties), to be written independently.

Best practice (Top score: none)

  • No framework reviewed includes an explicit commitment to permitting external evaluation reports to be written independently and without interference or suppression.
3.2.4Monitoring for novel risks
3.2.4.1Identifying novel risks post-deployment: engages in some process (post deployment) explicitly for identifying novel risk domains or novel risk models within known risk domains
  • There is a structured process for identifying novel risk domains or novel risk models within known risk domains.
  • There is justification for why this process will identify novel risks.

Best practice (Top score: 50% – Meta; Cohere)

  • Meta: The framework includes an explicit process to identify new catastrophic outcomes and threat scenarios, utilising workshops with subject matter experts. Periodic threat modelling exercises are conducted as a proactive measure to anticipate catastrophic risks from frontier AI. A monitoring setup is described that could be extended to identify novel risks post-deployment.
  • Cohere: They mention a process for performing “continuous monitoring” explicitly to “identify risks”. Whilst they may not be novel risk domains, it does suggest a willingness to detect novel threat models, detected via observation in the deployment context.
3.2.4.2Mechanism to incorporate novel risks identified post-deployment

Novel risks or risk pathways identified via monitoring post-deployment trigger further risk modeling and scenario analysis. This may include updating multiple or all risk models.

Best practice (Top score: 50% – Meta) 

  • The framework explicitly states that new risks, new threat scenarios, new capability patterns, and post-deployment developments can trigger updates to evaluations, threat models, mitigations, and preparedness assessments. Meta is comparatively strong on iterative threat-model evolution, reassessing risks after incidents or capability changes, and expanding evaluations when new concerns emerge. The framework is especially notable for recognizing that emerging evidence in one area may imply broader systemic reassessment needs.
4.1Decision-making
4.1.1The company has clearly defined risk owners for every key risk identified and tracked

The company ideally specifies who is the ultimate owner of each risk covered by the framework, or, at a minimum, indicated that responsible executives have been designated as risk owners

Best practice: (Top score: 75% – Microsoft)

  • Their framework assigns ownership of key risks to named roles or functions, specifying that designated executive officers or their delegates hold responsibility for key risk-related decisions.
4.1.2The company has a dedicated risk committee at the management level that meets regularly

The company has a specific body that is designated as the decision-making body for risk matters. At a minimum, there should be references to executives making risk decisions in a structured manner.

Best practice: (Top score: 90% – G42)

  • G42 has established a Frontier AI Governance Board that oversees frontier model operations, including safety protocols, risk assessments, and escalation decisions. 
  • The Board is composed of senior leaders including the Chief Responsible AI Officer, Head of Responsible AI, Head of Technology Risk, and General Counsel, reflecting cross-functional representation at the management level.
4.1.3The company has defined protocols for how to make go/no-go decisions

The company offers a clear description of how key development and deployment decisions are made and on what basis

Best practice: (Top score: 75% – Anthropic; OpenAI)

  • Anthropic: The framework outlines clear protocols for deployment decision-making, including who holds decision authority and on what basis decisions are taken. They detail both the decision-makers involved and the criteria informing their assessments.
  • OpenAI: Their framework also specifies protocols for decision-making, including who makes the decision and on what basis. They also clarify if their evaluations are based on residual risk (net of safeguards), providing useful clarity on the risk metric applied.
4.1.4The company has defined escalation procedures in case of incidents

The company offers detailed descriptions of the actions that will be taken in case of an incident, including harm reduction and information sharing.

Best practice: (Top score: 75% – Nvidia)

  • The framework describes clear procedures for managing incidents, including clear information sharing mechanisms between developers and relevant authorities, encouraging proactive identification of potential risks before they escalate.
  • Also, the framework establishes reduced access to a model when misuse is detected, with the intention to limit potential harm involving options as rolling back to previous versions.
4.2.Advisory and Challenge
4.2.1The company has an executive risk officer with sufficient resources (16.7%)

The company includes a management role that performs advisory and oversight. In order to maintain independence, this executive should be responsible for the risk management process running appropriately, but not be the risk owner, which is the domain of management. Importantly, they must also have the relevant staffing to execute on their responsibilities.

Best practice: (Top score: 75% – Anthropic)

  • Anthropic has established the role of the Responsible Scaling Officer (RSO), a dedicated executive-level position focused on AI risk management—a practice unique among the companies assessed.
  • The RSO role signals a strong institutional commitment to embedding risk considerations at the leadership level.
4.2.2The company has a committee advising management on decisions involving risk (16.7%)

The company includes a specific governance body that has sufficient risk expertise, that meets regularly and advises management on risk decisions

Best practice: (Top score: 75% – OpenAI) 

  • OpenAI has the position of the Safety Advisory Group (SAG), which is very commendable.  The SAG is responsible for: Overseeing the effective design, implementation, and adherence to the Framework in partnership with the safety organization leader. The SAG reviews relevant reports, assess track capabilities, and provides recommendations on potential next steps to manage risks.
4.2.3The company has an established system for tracking and monitoring risks (16.7%)

The company includes a system for monitoring and tracking risk levels over time. This should ideally be through a risk dashboard or equivalent where all risk information is aggregated.

Best practice: (Top score: 75% – OpenAI; Nvidia)

  • OpenAI: Their framework outlines a fairly detailed system for tracking and monitoring risks, particularly through capability evaluations that feed into deployment decisions.
  • Nvidia: Their framework references a “comprehensive repository of hazards” that is “mapped to assets,” suggesting a structured, high-maturity approach to risk tracking.
4.2.4The company has designated people that can advise and challenge management on decisions involving risk (16.7%)

The company references people in the organization with risk expertise that can provide challenge over management’s decisions when it comes to matters of risk.

Best practice: (Top score: 50% – OpenAI; G42)

  • OpenAI: OpenAI’s Safety Advisory Group (SAG) partially fulfills this criterion by reviewing risk assessments and providing recommendations to leadership. 
  • G42: G42’s Frontier AI Governance Board, composed of senior cross-functional leaders, can be assumed to play an advisory and challenge role.
4.2.5The company has an established system for aggregating risk data and reporting on risk to senior management and the Board (16.7%)

The company clearly outlines what risk information is provided to the Board and senior management on a regular basis and its format and cadence.

Best practice: (Top score: 90% – Anthropic)

  • The framework establishes a detailed system for aggregating and reporting risk information to senior management and the Board. It requires Risk Reports every 3–6 months, specifies that they include threat model identification and specification, evidence from capability and alignment evaluations, mitigations, additional relevant factors, threat-specific and overall risk assessments, risk-benefit determinations, and forward-looking monitoring plans.
4.2.6The company has an established central risk function (16.7%)

The company includes a central risk team that coordinates and manages all risk management processes.

Best practice: (Top score: 50% – Nvidia)

  • Nvidia’s framework references several teams, across engineers, researchers, and product managers, involved in risk management activities, indicating that risk responsibilities are distributed across the organisation.
4.3Audit
4.3.1The company has an internal audit function involved in AI governance

The company includes a specific governance entity providing independent assurance, typically an internal audit function. It should be empowered to conduct independent reviews of risks and controls.

Best practice: (Top score: 75% – Microsoft)

  • Microsoft’s framework specifies that its frontier AI safety framework falls within the remit of the company’s internal audit team. The framework states: “This framework is subject to Microsoft’s broader corporate governance procedures, including independent internal audit.” 
  • This integration into existing corporate governance and audit structures provides an additional layer of assurance over the framework’s implementation.
4.3.2The company involves external auditors

The company references reviews of risks, controls and adherence to the framework from external experts, or explicitly, the use of an external audit firm. These should ideally be fully independent as well as performed with sufficient access.

Best practice: (Top score: 90% – G42)

  • G42’s framework uniquely includes explicit commitments to external auditing. The company states it will engage in annual external audits to verify compliance with the framework, reinforcing accountability. 
  • In addition, G42 commits to soliciting external expert advice for capability and safeguards assessments, including partnering with private or civil society organisations with expertise in AI risk management to provide input on assessment plans and internal capability reports ahead of deployment decisions.
4.4Oversight
4.4.1The Board of Directors of the company has a committee that provides oversight over all decisions involving risk

The company includes a specific governance entity at the Board of Directors level. This should ideally be a “risk committee” specifically focused on risk matters, but can also be an “audit committee” or other designated committee. At a minimum, the framework should include references to an active role played by the Board.

Best practice: (Top score: 90% – OpenAI; Naver)

  • OpenAI: They have established a Safety and Security Committee (SSC) of the Board of Directors for risk oversight, which is given visibility into processes, can review decisions, and may require reports and information from OpenAI Leadership. The framework further notes that the Board may reverse a decision or mandate a revised course of action where necessary.
  • Naver: Naver’s framework specifies a risk management committee of the Board that is involved in decisions regarding risk.
4.4.2The company has other governing bodies outside of the Board of Directors that provide oversight over decisions

The company includes specific oversight entities outside the Board of Directors. This can be a Trust, a Council or similar and should have a clear description of its role and responsibilities.

Best practice: (Top score: 90% – Anthropic)

  • Anthropic’s Long-Term Benefit Trust (LTBT) is the only external oversight body among the providers assessed. The framework gives it a well-defined and substantive role: it receives regular briefings on Risk Report developments, is consulted on all RSP policy changes, receives Risk Reports and underlying decisions following CEO/RSO approval, and holds explicit veto authority when marginal risk analysis drives a deployment decision.
4.5Culture
4.5.1The company has a strong tone from the top (33.3%)

The company includes language that makes clear that the company has a strong commitment to managing the risks that may result from its development and deployment of LLMs.

Best practice: (Top score: 50% – Anthropic; G42)

  • Both companies include clear statements signalling leadership commitment to responsible AI development.
  • Anthropic: Anthropic’s policy reflects a recognition that risks accompany AI capabilities, and the company is credited as the first to release a risk management policy in the form of its Responsible Scaling Policy.
  • G42: G42’s framework emphasises proactive risk identification and mitigation, with commitments to fairness, reliability, safety, privacy, security, and inclusiveness.
4.5.2The company has a strong risk culture (33.3%)

The company includes either a commitment to building a strong risk culture or the components that contribute to a strong risk culture, such as risk training, safety drills, continuous updates of the framework, etc.

Best practice: (Top score: 75% – Nvidia)

  • Nvidia’s framework explicitly addresses the embedding of risk-aware practices into the daily work of engineers, researchers, and product managers, supported by ongoing training and open dialogue on ethical considerations.
  • The framework also highlights the use of qualitative approaches—such as model reviews and interviews with engineering teams—to capture developers’ intuitive understandings and early warning signs of risks.
  • Nvidia emphasises consistent communication channels with employees to ensure relevant stakeholders remain informed about rapid advancements and can promptly address emerging concerns.
4.5.3The company has a strong speak-up culture (33.3%)

The company includes clear whistleblowing procedures as well as a commitment to maintain a culture of employees speaking up on matters of non-compliance.

Best practice: (Top score: 90% – Anthropic; G42)

  • Anthropic: Anthropic’s policy includes strong provisions supporting a speak-up culture. The company commits to maintaining a process through which staff may anonymously notify the Responsible Scaling Officer of any potential instances of noncompliance with the policy.
  • G42: The framework establishes a reporting mechanism to foster proactive safety culture, with clearly defined channels to report anonymously potential concerns.
4.6Transparency
4.6.1The company reports externally on what their risks are (33.3%)

The company includes a commitment to communicate which risks their models pose externally as well as details on what information will be provided on those risks.

Best practice: (Top score: 90% – Anthropic)

  • The framework commits to publishing risk reports every 3–6 months, and include in them information on threat model identification and specification, evidence from capability and alignment evaluations, mitigations, additional relevant factors, threat-specific and overall risk assessments, risk-benefit determinations, and forward-looking monitoring plans.
4.6.2The company reports externally on what their governance structure looks like (33.3%)

The company is explicit in its description of the governance bodies that the company has in place and how they interact. If the company uses a framework such as the Three Lines of Defense or other governance framework, that should be called out. The company’ framework has a distinct section on governance.

Best practice: (Top score: 90% – Microsoft)

  • Microsoft’s framework provides detailed information on its governance structure and how the frontier AI safety framework integrates with the company’s broader AI governance programme. 
  • The framework commits to updating every six months through an explicit review discussion, with any updates reviewed by the Chief Responsible AI Officer before adoption and made public where appropriate. This provides external stakeholders with insight into both the structure and the process for its evolution.
4.6.3The company shares information with industry peers and government bodies (33.3%)

The company outlines the kind of information that will be shared externally and with who (government/industry fora/etc)

Best practice: (Top score: 90% – Anthropic; G42)

  • Both companies include concrete commitments to information sharing with external stakeholders.
  • Anthropic: They commit to notifying a relevant U.S. Government entity if a model requires stronger protections than its ASL-2 Standard, establishing a clear threshold for government engagement.
  • G42: They commit to sharing threat intelligence with industry partners to address common challenges and emerging risks, reflecting a collaborative approach to managing sector-wide threats.