Google DeepMind

1.3.2.1 Methodology precisely defined (70%) 10%

There is an indication of an awareness of risk modeling methodologies, but there are no concrete details about implementation.

Quotes:

“CCLs are determined by identifying and analyzing the main foreseeable paths through which a model could result in severe harm: we then define the CCLs as the minimal set of capabilities a model must possess to do so.” (p.4)

“We identify potential risks that could stem from our models and analyze their characteristics to determine which of the identified risks could be significant or severe risks.” (p.5)

1.3.2.2 Mechanism to incorporate red teaming findings (15%) 0%

No mention of risks identified during open-ended red teaming or evaluations triggering further risk modeling.

Quotes:

No relevant quotes found.

1.3.2.3 Prioritization of severe and probable risks (15%) 10%

The framework centers around critical capability levels (CCLs) which “may pose heightened risk of severe harm”, which are “determined by identifying and analyzing the main foreseeable paths through which a model could result in severe harm”. However, no concrete risk models are given, and beyond descriptions of severity and probability do not go beyond “heightened risk” and “severe harm”, which are imprecise terms that can be interpreted in various ways. No examples are given of specific risk models considered and excluded.

Quotes:

“The Framework is built primarily around capability thresholds called ‘Critical Capability Levels (CCLs).’ These are capability levels at which, absent mitigation measures, frontier AI models or systems may pose heightened risk of severe harm. CCLs are determined by identifying and analyzing the main foreseeable paths through which a model could result in severe harm: we then define the CCLs as the minimal set of capabilities a model must possess to do so.” (p.4)

“As explained above, we have identified risk domains where, based on early research, we have determined severe risks may be most likely to arise from future models” (p.5)

“The Frontier Safety Framework focuses on possible severe risks stemming from high-impact capabilities of frontier AI models.” (p.4)

“Critical Capability Levels (CCLs): are the main capability thresholds around which we have built the Framework process. They represent the capability levels at which, absent mitigation measures, frontier AI models or systems may pose heightened risk of severe harm.
Tracked Capability Levels (TCLs): are capability thresholds which capture a lower level of risks than our CCLs. They represent the capability levels at which, absent mitigation measures, frontier AI models or systems may pose heightened risk of significant but not severe levels of harm.” (p.18)

2.1.1.1 Risk tolerance is at least qualitatively defined for all risks (33%) 10%

They indicate that they will not tolerate certain risks of “significant or severe harm” which is not further defined. Each capability threshold functions as an implicit risk tolerance, e.g. “Cyber uplift level 1: Provides sufficient uplift with high impact cyber attacks for additional expected harm at severe scale.” However, these are vague. The framework references acceptable levels of residual risk, but does not specify what acceptable means.

Quotes:

“Cyber uplift level 1: Provides sufficient uplift with high impact cyber attacks for additional expected harm at severe scale.” (p.10)

“In particular, we will deem deployment mitigations adequate if the evidence suggests that for the T/CCLs the model has reached, the increase in likelihood of harm from the proposed external deployment has been reduced to an acceptable level.” (p.10)

“These are capability levels at which, absent mitigation measures, frontier AI models or systems may pose heightened risk of severe harm” (p.4)

“This process is designed to ensure that residual risk remains at acceptable levels”. (p 10)

2.1.1.2 Risk tolerance is expressed at least partly quantitatively as a combination of scenarios (qualitative) and probabilities (quantitative) for all risks (33%) 0%

The risk tolerance, implicit or otherwise, is not expressed fully or partly quantitatively.
No indication of expressing the risk tolerance beyond “severe harm”, which is not further defined. To improve, the risk tolerance should be expressed fully quantitatively or as a combination of scenarios with probabilities.

Quotes:
No relevant quotes found.

2.1.1.3 Risk tolerance is expressed fully quantitatively as a product of severity (quantitative) and probability (quantitative) for all risks (33%) 0%

No indication of expressing the risk tolerance beyond “severe harm”, which is not further defined. There is no quantitative definition of severity nor probabilities given.

Quotes:

No relevant quotes found.

2.1.2.1 AI developers engage in public consultations or seek guidance from regulators where available (50%) 10%

No evidence of asking the public what risk levels they find acceptable. No evidence of seeking regulator input specifically on what constitutes acceptable risk levels. However, there is a process which draws on “relevant high-quality research” and “information shared through industry forums” which informs T/CCLs (which function as risk tolerances/unacceptable risk tiers). They reference engaging governments regarding practices overall.

Quotes:

“Where appropriate, we may engage relevant external actors, including governments, to inform our responsible development and deployment practices.” (p.6)

2.1.2.2 Any significant deviations from risk tolerance norms established in other industries is justified and documented (e.g., cost-benefit analyses) (50%) 10%

There is no evidence of comparing against norms. The closest analogus is a discussion related to proportionality writ large. Also, in several places, they explicitly weigh security against innovation costs

Quotes:
“The concept of proportionality is central to our determination of whether a particular mitigation has sufficiently reduced the risk to acceptable levels. The mitigation and the effects of such mitigation should also be assessed holistically and be proportionate with expected impact of a model’s risk, thus balancing safety with innovation”. (p.8)

“Models able to greatly assist cyber attack might be of interest to well-resourced state actors. However, the potential for automated
cyber-defense and social adaptation as a response to exfiltration means that higher levels of security, and the resulting costs to
innovation, are likely not warranted.” (p.12)

2.2.1.1 KRI thresholds are at least qualitatively defined for all risks (45%) 75%

Each risk domain has one or several KRIs, which are qualitatively defined. The KRI appears to be grounded in risk modelling. Google DeepMind operationalizes risk tolerance through explicit capability thresholds (TCLs/CCLs), evaluation-triggered alert thresholds, domain-specific capability definitions, and ongoing early warning evaluations, that function like KRIs.

Quotes:
“The Framework is built primarily around capability thresholds called “Critical Capability Levels (CCLs). These are capability levels at which, absent mitigation measures, frontier AI models or systems may pose heightened risk of severe harm. CCLs are determined by identifying and analyzing the main foreseeable paths through which a model could result in severe harm: we then define the CCLs as the minimal set of capabilities a model must possess to do so. CCLs are one important component of our risk acceptance determination.” (p.4)

“This update to the Framework introduces “Tracked Capability Levels (TCLs).” TCLs are meant to capture significant risks that may manifest at a lower capability threshold than our CCLs and we apply our mitigation and risk acceptance determination process proportionately.”(p.4)

“Central to our critical capability assessments are “early warning evaluations,” which we use to test the specific threats and risk scenarios identified through our threat modeling, determine a model’s capability, and assess the proximity of the model to a T/CCL” (p.5)

“CBRN uplift level 1: Provides low to medium resourced actors uplift in reference scenarios resulting in additional expected harm at severe scale.” (p.11)

2.2.1.2 KRI thresholds are quantitatively defined for all risks (45%) 10%

The framework does not provide fully quantitative KRI thresholds in the sense of numerical benchmark cutoffs or explicit scores but many of the KRIs are framed around measurable capability evaluations and benchmarkable model behaviors. The evaluations and alert thresholds point to this. A statement such as “resulting in AI progress substantially accelerating from historical rates” is not quantified, but has the potential to be.

Quotes:

“Central to our critical capability assessments are “early warning evaluations,” which we use to test the specific threats and risk scenarios identified through our threat modeling, determine a model’s capability, and assess the proximity of the model to a T/CCL.” (p.5)

“These assessments draw on model characteristics, such as its provenance, size, modality, and performance on general capability benchmarks. (p.5)

“ML R&D acceleration level 1: Has been used to accelerate AI development, resulting in AI progress substantially accelerating from historical rates.” (p.15)

2.2.1.3 KRIs also identify and monitor changes in the level of risk in the external environment (10%) 25%

This criterion is partially satisfied. The framework explicitly incorporates several forms of external-environment monitoring beyond static model capability evaluation, even if they are not formulated as pure KRIs. The framework explicitly considers broader threat landscapes, external model availability, post-market monitoring, historical incident data, evolving attacker capabilities, and deployment scope.

Quotes:
“We conduct further analysis, including reviewing model-independent information, external evaluations, and post-market monitoring as appropriate.” (p.6)

“What capabilities and mitigations are available on other publicly available models.” (p.7)

“whether data suggests a high (or low) likelihood of attempted misuse of models at the T/CCL”. (p.10)

2.2.2.1 Containment KCIs (35%) 43%

2.2.2.2 Deployment KCIs (35%) 43%

2.2.2.3 For advanced KRIs, assurance process KCIs are defined (30%) 25%

This criterion is partially satisfied. The framework is comparatively sophisticated in this respect. The clearest advanced KRI is the Stealth and Situational Awareness TCL.

Quotes:

“Stealth and Situational Awareness TCL: The instrumental reasoning abilities of the model enable enough situational awareness (ability to discover and use relevant details of its deployment setting) and stealth (ability to circumvent basic oversight mechanisms) such that, absent additional mitigations, we cannot rule out the model significantly undermining human control. ” (p.14)

3.1.1.1 Containment measures are precisely defined for all KCI thresholds (60%) 25%

Google DeepMind’s framework partially satisfies this criterion, but not strongly. The framework does provide fairly concrete categories of containment/security mitigations, some examples of specific controls, explicit linkage between capability thresholds and recommended security levels, mitigation processes tied to T/CCLs. However containment KCIs themselves are not explicitly defined,
thresholds for what constitutes sufficient containment are mostly qualitative, many measures are illustrative (“may include…”), implementation specifics are intentionally left flexible and evolving, and there is no comprehensive mapping from each threshold to a precisely specified required control set.

Quotes:
“Security mitigations against exfiltration or unauthorized modification risk, such as identity and access management practices and hardening interface-access to unreleased model parameters, are important for models reaching misuse CCLs.” (p.9)

“We use security levels to indicate security goals/principles in line with the corresponding level in the
RAND model weight security framework. We also define and recommend “Security Level 2+,” which uses RAND Security Level 2 (SL2) as a baseline, with additional security measures designed to address risks from insider threats and well-resourced non-state external actors. These additional measures may include, for example: dedicated insider risk teams; background checks and ID verification for personnel with sensitive access; review of model training data for signs of tampering; mandating that the processing of untrusted inputs occurs within sandboxed environments; advanced red-teaming that simulates well-resourced adversaries (including Advanced Persistent Threat (APT) groups); and proactive threat hunting with 24/7 incident response capabilities.” (p.9)

“We recommend a security level for each CCL, which reflects our assessment of the minimum appropriate level of security the field of frontier AI should apply to models reaching each CCL.” (p.11)

“Because we cannot always anticipate what security and deployment mitigations will be appropriate for models beyond the current frontier, the specific mitigations we implement may be determined when a T/CCL is reached.” (p.6)

3.1.1.2 Proof that containment measures are sufficient to meet the thresholds (40%) 10%

Google DeepMind’s framework only weakly satisfies this criterion. The framework does contain a formal residual risk assessment process, references to testing, threat modeling, empirical evaluation, red teaming, and safety cases, governance review before deployment, and iterative mitigation assessment processes. However, it does not provide proof that containment/security measures are sufficient to satisfy predefined containment thresholds, explicit KCI thresholds, quantified sufficiency criteria, ex ante evidentiary standards, predefined proof requirements, confidence levels, or predictable/non-discretionary implementation rules.

Quotes:
“Deployment mitigations: we will use a combination of threat modeling, empirical testing, and other sources of information to assess the effectiveness and limitations of our deployment mitigations.” (p.7)

“This residual risk assessment could take into account factors such as i. The effectiveness of the mitigations. For example, tests run on mitigated models may suggest that the refusal rate and jailbreak robustness together imply that threat actors are unlikely to be able to circumvent safeguards.” (p.10)

“Where the model has reached a CCL, the residual risk assessment will be supplemented with a safety case.” (p.10)

3.1.1.3 Strong third party verification process to verify that the containment measures meet the threshold (100% if 3.1.1.3 > [60% x 3.1.1.1 + 40% x 3.1.1.2]) 0%

Google DeepMind’s framework does not satisfy this criterion. The framework does acknowledge involvement of external experts, possible involvement of governments, external evaluations, external organizations, and disclosure of evaluation results in some circumstances. However, it does not establish a structured third-party verification or audit regime, mandatory external audits before deployment, independent certification of containment/security sufficiency, independence guarantees, publication of external assurance reports or external sign-off requirements for containment thresholds. The framework gestures toward external involvement, but not in a structured assurance-oriented way.

Quotes:
No relevant quotes found.

3.1.2.1 Deployment measures are precisely defined for all KCI thresholds (60%) 50%

Google DeepMind’s framework moderately satisfies this criterion. Deployment mitigations are described in some operational detail, concrete examples of deployment controls are given and mitigation assessment processes are articulated. However, the framework falls short because deployment KCIs are not explicitly defined, deployment thresholds are not operationalized into measurable control requirements, and mitigations remain partly illustrative and discretionary.

Quotes:
“The following mitigation process for external deployments will be applied to models reaching a misuse T/CCL, allowing for iterative and flexible tailoring of mitigations to each risk and use case. 1. Development and assessment of mitigations: safeguards and an accompanying residual risk assessment are developed by iterating on the following: a. Developing and improving a suite of safeguards targeting the capability, which may include measures such as safety post-training, monitoring and analysis, account
moderation, jailbreak detection and patching, user verification, and bug bounties. b. Assessing the robustness of these mitigations against the risk posed through testing (e.g. automated evaluations, red teaming) and threat modeling research.” (p.10)

3.1.2.2 Proof that deployment measures are sufficient to meet the thresholds (40%) 25%

The framework describes a process, assumedly internal, for “evaluate the effectiveness and limitations of mitigations”, but does not detail why they ex ante believe their deployment measures to be sufficient. Instead, it relies on the “appropriate corporate governance body” and their discretion. To improve, this proof should be garnered in advance, to be sure that the measures will be sufficient to meet the KCI threshold once the model crosses the relevant KRI threshold, and indeed have “proactive mitigation plans”.

Quotes:

“We will use various processes to evaluate the effectiveness and limitations of mitigations: […] Deployment mitigations: we will use a combination of threat modeling, empirical testing, and other sources of information to assess the effectiveness and limitations of our deployment mitigations. These will form the basis of a safety case for models reaching CCLs, that will be reviewed before deployment.” (p. 6)

“Prepare and articulate proactive mitigation plans to ensure severe risks are adequately mitigated when such capability levels are attained.” (p. 2)

“This process is designed to ensure that residual risk remains at acceptable levels: evidence of efficacy collected during development and testing, as well as expert-driven estimates of other parameters, will enable us to assess residual risk and to detect substantial changes that invalidate our risk assessment. With iteration on safeguards and safety cases, we believe that we are able to make informed decisions about the level of risk via a CCL before a model is released, and reliably prevent models posing unacceptable levels of risk from being deployed.” (p. 9)

3.1.2.3 Strong third party verification process to verify that the deployment measures meet the threshold (100% if 3.1.2.3 > [60% x 3.1.2.1 + 40% x 3.1.2.2]) 0%

Google DeepMind’s framework very weakly satisfies this criterion. The framework does contain references to external experts, references to external evaluations, references to red teaming, some disclosure language and optional engagement with governments and external organizations. However, it does not establish a structured third-party verification regime for deployment safeguards, mandatory external red teaming prior to deployment, independent verification that deployment mitigations satisfy thresholds, expert qualification criteria, independence guarantees, publication of external assurance reports or any external sign-off or certification processes.

Quotes:
“When a model reaches an alert threshold for a CCL, we will assess the proximity of the model to the CCL and analyze the risk posed, involving internal and external experts as needed.” (p.6)

“Assessing the robustness of these mitigations against the risk posed through testing (e.g. automated evaluations, red teaming) and threat modeling research.” (p.10)

3.1.3.1 Credible plans towards the development of assurance processes (40%) 25%

Google DeepMind’s framework weakly-to-moderately satisfies this criterion, primarily because it does acknowledge that current methods and mitigations are evolving and potentially insufficient, and it outlines a process for iterative improvement. However, it falls well short of providing a concrete assurance-process development roadmap.The framework does reasonably well on acknowledging uncertainty and immaturity, recognizing that mitigations and evaluations will need to evolve, articulating escalation thresholds (TCLs/CCLs/alert thresholds) and describing future-oriented monitoring and assessment processes. But it does not provide explicit assurance-process KCIs, explicit acknowledgement that current assurance processes are insufficient, detailed assurance roadmaps,
technical milestones, timelines, capability-growth-linked preparedness plans, forecasting-based readiness targets, or concrete plans for developing future assurance infrastructure.

Quotes:
“The Framework is based on early and evolving research. We may change our approach over time as we
gain experience and insights on the projected capabilities of future frontier models. We will review the
Framework periodically and we expect it to evolve substantially as our understanding of the risks and
benefits of frontier models improves.” (p.2)

“Because the science of AI risk assessment is still developing, our assessments will often involve some level of subjective analysis.” (p.8)

“Because we cannot always anticipate what security and deployment mitigations will be appropriate for models beyond the current frontier, the specific mitigations we implement may be determined when a T/CCL is reached.” (p.6)

“We may run early warning evaluations more frequently or adjust alert thresholds if the rate of progress suggests our safety buffer is no longer adequate.” (p.5)

3.1.3.2 Evidence that the assurance processes are enough to achieve their corresponding KCI thresholds (40%) 0%

Google DeepMind’s framework does not satisfy this criterion. The framework contains defined processes for empirical evaluation, red teaming, and some notion of evidence collection. However, it does not provide a dedicated assurance-process validation methodology, empirical validation of the assurance processes themselves, model-organism demonstrations, theoretical guarantees, formal assurance benchmarks, evidence that oversight/audit/safety-case processes are themselves reliable, or any meta-evaluation of assurance quality.

Quotes:
No relevant quotes found.

3.1.3.3 The underlying assumptions that are essential for their effective implementation and success are clearly outlined (20%) 10%

Google DeepMind’s framework very weakly satisfies this criterion. The framework does contain some implicit assumptions underlying its assurance approach, acknowledgment of uncertainty and evolving science, discussion of safety buffers, alert thresholds, and evaluation limitations and some recognition that evaluations and mitigations may fail or become inadequate. However, it does not explicitly articulate core technical assumptions underlying assurance validity, assumptions about evaluator reliability, assumptions about chain-of-thought faithfulness, assumptions about absence of sandbagging/deception, assumptions about generalization from evals, assumptions about robustness of monitoring or assumptions about interpretability or detectability. Nor does it provide systematic stress testing of such assumptions, empirical validation of assumption robustness, confidence levels around assumption validity or contingency plans for assumption failure.

Quotes:

“Because the science of AI risk assessment is still developing, our assessments will often involve some level of subjective analysis.” (p.8)

“We may run early warning evaluations more frequently or adjust alert thresholds if the rate of progress suggests our safety buffer is no longer adequate.” (p.5)

“Because we cannot always anticipate what security and deployment mitigations will be appropriate for models beyond the current frontier.” (p.6)

‘In our evaluations, we seek to apply appropriate scaffolding, inference compute, and other augmentations to also assess the capabilities of systems that will likely be produced with the model.” (p.5)

3.2.1.1 Justification that elicitation methods used during the evaluations are comprehensive enough to match the elicitation efforts of potential threat actors (30%) 25%

Google DeepMind’s framework moderately satisfies this criterion. The framework explicitly recognizes the importance of strong elicitation, the possibility that threat actors may elicit more capability than evaluators, the need for scaffolding/augmentation, the need to upper-bound realistic system capabilities rather than merely base-model behavior and the need for conservative evaluation practices. However, it falls short of fully satisfying the criterion because it provides very little operational detail, there are no concrete elicitation protocols, no compute budgets, no finetuning procedures, no explicit capability upper-bound methodology, or quantified justification that evaluations truly upper-bound adversarial elicitation.

Quotes:
“Risk assessment must take into account the fact that other actors may put significantly more effort into eliciting capabilities than we put into assessing risk, thus requiring conservatism in the form of evaluations.” (p.6)

“In our evaluations, we seek to apply appropriate scaffolding, inference compute, and other augmentations to also assess the capabilities of systems that will likely be produced with the model.” (p.5)

3.2.1.2 Evaluation frequency (25%) 25%

Google DeepMind’s framework moderately satisfies this criterion. The framework does fairly well on establishing recurring and lifecycle-wide evaluation processes, tying evaluations to model updates and capability changes, recognizing the need for more frequent evaluations under rapid capability progress and incorporating both periodic review and capability-triggered reassessment logic. However, it falls short because evaluation cadence is not concretely specified, there are no fixed periodic schedules for evaluations themselves, there are no explicit compute-trigger thresholds, “material capability increase” remains qualitative, and there is little operational detail about scaling-based triggers.

Quotes:
“We conduct our risk management process as appropriate, throughout the model development process, on various checkpoints or versions of a model, both before and after deployment.” (p.5)

“For external deployment of subsequent versions of the model, we determine whether a substantial modification has been made and whether a further critical capability assessment is required if (1) the model has meaningful new capabilities or material increases in performance; and (2) we believe such material capability increase could materially undermine the justification for why the risks stemming from the model are acceptable.” (p.5)

“To understand if a subsequent version of the model has meaningful new capabilities or material increases in performance relative to the last checkpoint subject to a critical capability assessment, we conduct material capability change assessments on various checkpoints upon the completion of a post-training run.” (p.5)

“We may run early warning evaluations more frequently or adjust alert thresholds if the rate of progress suggests our safety buffer is no longer adequate.” (p.5)

3.2.1.3 Description of how post-training enhancements are factored into capability assessments (15%) 25%

Google DeepMind’s framework moderately satisfies this criterion conceptually, but operationalizes it weakly. It explicitly discusses post-training capability increases, material capability change assessments, repeated reassessment after post-training runs, scaffolding/inference-compute augmentation, safety buffers, alert thresholds andincreasing evaluation frequency if progress accelerates. However, it still falls short because there is no concrete methodology for incorporating post-training enhancements, no quantified uncertainty margins, no forecasting methodology, no explicit treatment of reasoning-model inference scaling, no operationalized safety-margin calculation process and no measurable thresholds tied to post-training capability uplift.

Quotes:
“To understand if a subsequent version of the model has meaningful new capabilities or material increases in performance relative to the last checkpoint subject to a critical capability assessment, we conduct material capability change assessments on various checkpoints upon the completion of a post-training run.” (p.5)

“In our evaluations, we seek to apply appropriate scaffolding, inference compute, and other augmentations to also assess the capabilities of systems that will likely be produced with the model based on the specific threats and risk scenarios we have identified for the T/CCL.” (p.5)

“Material Capability Increases: are meaningful new capabilities or material increases in model performance that we believe could materially undermine our justifications for a model’s level of risk being acceptable.” (p.19)

3.2.1.4 Vetting of protocols by third parties (15%) 10%

Google DeepMind’s framework weakly satisfies this criterion. The framework does contain references to external experts, references to external evaluations, some governance-review mechanisms andsome engagement with governments and external actors. However, it does not establish a structured process for third-party review of KRI assessment methodologies, independent vetting of evaluation protocols, external review of elicitation methods, or formal methodology review governance.

Quotes:
“When a model reaches an alert threshold for a CCL, we will assess the proximity of the model to the CCL and analyze the risk posed, involving internal and external experts as needed.” (p.6)

“Where appropriate, we may engage relevant external actors, including governments, to inform our responsible development and deployment practices.” (p.6)

3.2.1.5 Replication of evaluations by third parties (15%) 0%

Google DeepMind’s framework does not satisfy this criterion.

Quotes:
No relevant quotes found.

3.2.2.1 Detailed description of evaluation methodology and justification that KCI thresholds will not be crossed unnoticed (40%) 25%

Google DeepMind’s framework moderately satisfies this criterion conceptually, but only weakly satisfies it operationally. The framework clearly establishes ongoing monitoring processes, residual risk reassessment, post-market monitoring, iterative mitigation review, empirical testing, reassessment after capability changes and adjustment of evaluation frequency and alert thresholds. However, it still falls short because KCIs themselves are not explicitly defined, no quantitative efficacy thresholds are specified, no confidence levels are provided, failure mode analysis is only implicit, there is no rigorous statistical detection methodology and no formal false-negative/error-rate treatment is given.

Quotes:
“We conduct our risk management process as appropriate, throughout the model development process, on various checkpoints or versions of a model, both before and after deployment.” (p.5)

“Assessing the robustness of these mitigations against the risk posed through testing (e.g. automated evaluations, red teaming) and threat modeling research.” (p.10)

“We conduct further analysis, including reviewing model-independent information, external evaluations, and post-market monitoring as appropriate.” (p.6)

“We may run early warning evaluations more frequently or adjust alert thresholds if the rate of progress suggests our safety buffer is no longer adequate.” (p.5)

3.2.2.2 Vetting of protocols by third parties (30%) 0%

Google DeepMind’s framework does not satisfy this criterion.

Quotes:
No relevant quotes found.

3.2.2.3 Replication of evaluations by third parties (30%) 0%

Google DeepMind’s framework does not satisfy this criterion.

Quotes:
No relevant quotes found.

3.2.3.1 Sharing of evaluation results with relevant stakeholders as appropriate (85%) 25%

Google DeepMind’s framework partially satisfies this criterion. It has a meaningful commitment to notifying authorities under some circumstances, but much weaker commitments around public transparency of KRI/KCI evaluations. It has moderate compliance on regulator/government notification, weak compliance on public disclosure of evaluations and very weak compliance on predefined transparency criteria. The disclosure obligations are conditional, discretionary, limited in scope, not tied to all KRI/KCI pairings and not framed as routine transparency.

Quotes:
“If we assess that a model has reached a CCL that poses an unmitigated and material risk to overall public safety, we aim to share relevant information with appropriate government authorities where it will facilitate safety of frontier AI.” (p.17)

“We may also consider disclosing information to other external organizations to promote shared learning and coordinated risk mitigation.” (p.17)

“Where appropriate, and subject to adequate confidentiality and security measures and considerations around proprietary and sensitive information, this information may include:
● Model information: characteristics of the AI model relevant to the risk it may pose with its critical capabilities.
● Evaluation results: such as details about the evaluation design, the results, and any robustness
tests.
● Mitigation plans: descriptions of our mitigation plans and how they are expected to reduce the
risk. (p.17)

3.2.3.2 Commitment to non-interference with findings (15%) 0%

Google DeepMind’s framework does not satisfy this criterion.

Quotes:
No relevant quotes found.

3.2.4.1 Identifying novel risks post-deployment: engages in some process (post deployment) explicitly for identifying novel risk domains or novel risk models within known risk domains (50%) 50%

Google DeepMind’s framework moderately satisfies this criterion conceptually, though the justification for why the process will successfully identify novel risks remains fairly weak and high-level. The framework explicitly recognizes that risk domains may evolve, commits to ongoing reassessment, includes post-market monitoring, includes iterative framework updates, contemplates adding new risk domains/TCLs/CCLs, and incorporates external research and incident learning. However, there only moderate evidence of post-deployment learning mechanisms and justification that the process is sufficient.

Quotes:
“As part of our broader research into and development of frontier AI models, we continue to assess whether there are other risk domains where significant or severe risks may arise and will update our approach as appropriate.” (p.5)

‘In particular, we strive to learn from our post-market monitoring mechanisms, including as part of our detection, mitigation, response and/or reporting of incidents relating to our frontier safety risk domains.” (p.6)

“Actionable insights from these processes allow us to enhance our tools, training, processes, policies, and response efforts.” (p.6)

‘We may include TCLs for additional risks in the future, as our threat modeling develops.” (p.4)

3.2.4.2 Mechanism to incorporate novel risks identified post-deployment (50%) 25%

Google DeepMind’s framework moderately satisfies this criterion. It explicitly connects post-deployment monitoring, framework updates, updating risk domains/TCLs/testing approaches and iterative reassessment. However, it still falls short of a fully mature operationalized mechanism because it does not explicitly describe cascading updates across multiple risk models, it does not provide formal escalation pathways, it does not define structured scenario-analysis procedures for novel findings and it does not explicitly discuss cross-domain propagation of newly discovered capabilities/risk pathways. So it is weak on operational detail for cross-model incorporation.

Quotes:
“The Frontier Safety Framework will be reviewed at least once a year—more frequently if we have reasonable grounds to believe the adequacy of the Framework or our adherence to it has been materially undermined. The process will involve (i) an assessment of the Framework’s appropriateness for the management of significant and severe risk, drawing on information sources such as record of adherence to the framework, relevant high-quality research, information shared through industry forums, and evaluation results, as necessary, and (ii) an assessment of our adherence to the Framework. Following this assessment, we may:
● Update our risk domains and T/CCLs, where necessary.
● Update our testing and mitigation approaches, where needed to ensure risk remains adequately assessed and addressed according to our current understanding.” (p.17)

“Actionable insights from these processes allow us to enhance our tools, training, processes, policies, and response efforts.” (p.6)