Naver

Category
Naver
Best in Class
1. Risk Identification
7%
62%
  • 1. Classification of Applicable Known Risks (40%)
    13%
    90%
    • 1. Risks from literature and taxonomies are well covered (50%)
      25%
      90%
    • 2. Exclusions are clearly justified and documented (50%)
      0%
      90%
  • 2. Identification of Unknown Risks (Open-ended red teaming) (20%)
    0%
    43%
    • 1. Internal open-ended red teaming (70%)
      0%
      50%
    • 2. Third party open-ended red teaming (30%)
      0%
      25%
  • 3. Risk modeling (40%)
    4%
    52%
    • 1. The company uses risk models for all the risk domains identified
and the risk models are published (with potentially dangerous
information redacted) (40%)
      10%
      75%
    • 2. Risk modeling methodology (40%)
      0%
      43%
      • 1. Methodology precisely defined (70%)
        0%
        50%
      • 2. Mechanism to incorporate red teaming findings (15%)
        0%
        25%
      • 3. Prioritization of severe and probable risks (15%)
        0%
        75%
    • 3. Third party validation of risk models (20%)
      0%
      50%
2. Risk Analysis and Evaluation
7%
36%
  • 1. Setting a Risk Tolerance (35%)
    3%
    22%
    • 1. Risk tolerance is defined (80%)
      3%
      28%
      • 1. Risk tolerance is at least qualitatively defined for all risks (33%)
        10%
        75%
      • 2. Risk tolerance is expressed at least partly quantitatively as a combination of scenarios (qualitative) and probabilities (quantitative) for all risks (33%)
        0%
        10%
      • 3. Risk tolerance is expressed fully quantitatively as a product of severity (quantitative) and probability (quantitative) for all risks (33%)
        0%
        0%
    • 2. Process to define the tolerance (20%)
      0%
      10%
      • 1. AI developers engage in public consultations or seek guidance from regulators where available (50%)
        0%
        10%
      • 2. Any significant deviations from risk tolerance norms established in other industries is justified and documented (e.g., cost-benefit analyses) (50%)
        0%
        10%
  • 2. Operationalizing Risk Tolerance (65%)
    9%
    44%
    • 1. Key Risk Indicators (KRI) (30%)
      15%
      59%
      • 1. KRI thresholds are at least qualitatively defined for all risks (45%)
        10%
        75%
      • 2. KRI thresholds are quantitatively defined for all risks (45%)
        0%
        50%
      • 3. KRIs also identify and monitor changes in the level of risk in the external environment (10%)
        0%
        25%
    • 2. Key Control Indicators (KCI) (30%)
      4%
      37%
      • 1. Containment KCIs (35%)
        5%
        45%
        • 1. All KRI thresholds have corresponding qualitative containment KCI thresholds (50%)
          10%
          90%
        • 2. All KRI thresholds have corresponding quantitative containment KCI thresholds (50%)
          0%
          10%
      • 2. Deployment KCIs (35%)
        5%
        43%
        • 1. All KRI thresholds have corresponding qualitative deployment KCI thresholds (50%)
          10%
          75%
        • 2. All KRI thresholds have corresponding quantitative deployment KCI thresholds (50%)
          0%
          25%
      • 3. For advanced KRIs, assurance process KCIs are defined (30%)
        0%
        50%
    • 3. Pairs of thresholds are grounded in risk modeling to show that risks remain below the tolerance (20%)
      10%
      25%
    • 4. Policy to put development on hold if the required KCI threshold cannot be achieved, until sufficient controls are implemented to meet the threshold (20%)
      10%
      50%
3. Risk Treatment
8%
38%
  • 1. Implementing Mitigation Measures (50%)
    4%
    38%
    • 1. Containment measures (35%)
      0%
      74%
      • 1. Containment measures are precisely defined for all KCI thresholds (60%)
        0%
        90%
      • 2. Proof that containment measures are sufficient to meet the thresholds (40%)
        0%
        50%
      • 3. Strong third party verification process to verify that the containment measures meet the threshold (100% if 3.1.1.3 > [60% x 3.1.1.1 + 40% x 3.1.1.2])
        0%
        25%
    • 2. Deployment measures (35%)
      10%
      40%
      • 1. Deployment measures are precisely defined for all KCI thresholds (60%)
        10%
        50%
      • 2. Proof that deployment measures are sufficient to meet the thresholds (40%)
        10%
        100%
      • 3. Strong third party verification process to verify that the deployment measures meet the threshold (100% if 3.1.2.3 > [60% x 3.1.2.1 + 40% x 3.1.2.2])
        0%
        25%
    • 3. Assurance processes (30%)
      0%
      43%
      • 1. Credible plans towards the development of assurance processes (40%)
        0%
        25%
      • 2. Evidence that the assurance processes are enough to achieve their corresponding KCI thresholds (40%)
        0%
        50%
      • 3. The underlying assumptions that are essential for their effective implementation and success are clearly outlined (20%)
        0%
        75%
  • 2. Continuous Monitoring and Comparing Results with Pre-determined Thresholds (50%)
    13%
    40%
    • 1. Monitoring of KRIs (40%)
      23%
      50%
      • 1. Justification that elicitation methods used during the evaluations are comprehensive enough to match the elicitation efforts of potential threat actors (30%)
        0%
        90%
      • 2. Evaluation frequency (25%)
        90%
        90%
      • 3. Description of how post-training enhancements are factored into capability assessments (15%)
        0%
        50%
      • 4. Vetting of protocols by third parties (15%)
        0%
        25%
      • 5. Replication of evaluations by third parties (15%)
        0%
        25%
    • 2. Monitoring of KCIs (40%)
      10%
      43%
      • 1. Detailed description of evaluation methodology and justification that KCI thresholds will not be crossed unnoticed (40%)
        25%
        50%
      • 2. Vetting of protocols by third parties (30%)
        0%
        50%
      • 3. Replication of evaluations by third parties (30%)
        0%
        25%
    • 3. Transparency of evaluation results (10%)
      0%
      64%
      • 1. Sharing of evaluation results with relevant stakeholders as appropriate (85%)
        0%
        75%
      • 2. Commitment to non-interference with findings (15%)
        0%
        0%
    • 4. Monitoring for novel risks (10%)
      0%
      38%
      • 1. Identifying novel risks post-deployment: engages in some process (post deployment) explicitly for identifying novel risk domains or novel risk models within known risk domains (50%)
        0%
        50%
      • 2. Mechanism to incorporate novel risks identified post-deployment (50%)
        0%
        50%
4. Risk Governance
18%
50%
  • 1. Decision-making (25%)
    13%
    56%
    • 1. The company has clearly defined risk owners for every key risk identified and tracked (25%)
      0%
      75%
    • 2. The company has a dedicated risk committee at the management level that meets regularly (25%)
      0%
      90%
    • 3. The company has defined protocols for how to make go/no-go decisions (25%)
      50%
      75%
    • 4. The company has defined escalation procedures in case of incidents (25%)
      0%
      75%
  • 2. Advisory and Challenge (20%)
    12%
    75%
    • 1. The company has an executive risk officer with sufficient resources (16.7%)
      0%
      75%
    • 2. The company has a committee advising management on decisions involving risk (16.7%)
      25%
      90%
    • 3. The company has an established system for tracking and monitoring risks (16.7%)
      0%
      75%
    • 4. The company has designated people that can advise and challenge management on decisions involving risk (16.7%)
      10%
      50%
    • 5. The company has an established system for aggregating risk data and reporting on risk to senior management and the Board (16.7%)
      10%
      90%
    • 6. The company has an established central risk function (16.7%)
      25%
      50%
  • 3. Audit (20%)
    5%
    58%
    • 1. The company has an internal audit function involved in AI governance (50%)
      0%
      50%
    • 2. The company involves external auditors (50%)
      10%
      90%
  • 4. Oversight (20%)
    45%
    58%
    • 1. The Board of Directors of the company has a committee that provides oversight over all decisions involving risk (50%)
      90%
      90%
    • 2. The company has other governing bodies outside of the Board of Directors that provide oversight over decisions (50%)
      0%
      90%
  • 5. Culture (10%)
    8%
    67%
    • 1. The company has a strong tone from the top (33.3%)
      25%
      50%
    • 2. The company has a strong risk culture (33.3%)
      0%
      75%
    • 3. The company has a strong speak-up culture (33.3%)
      0%
      100%
  • 6. Transparency (5%)
    32%
    75%
    • 1. The company reports externally on what their risks are (33.3%)
      10%
      90%
    • 2. The company reports externally on what their governance structure looks like (33.3%)
      75%
      90%
    • 3. The company shares information with industry peers and government bodies (33.3%)
      10%
      90%
Summary

NAVER’s framework is built around two risk categories, loss of control and misuse, rather than an operational risk-management pipeline. NAVER specifies the most concrete evaluation trigger of any provider assessed, committing to review every three months, or whenever model performance increases sixfold. Risk Governance is its strongest category overall, backed by a board-level risk committee and a disclosed governance structure. However, the framework defines no quantitative risk indicators or control thresholds, presents almost no risk modeling, sets no meaningful risk tolerance, and describes only vague mitigations that are not linked to specific indicators or risk domains.

Overview
Strengths
  • Clear frequency for conducting model evaluations, given both in terms of time periods and model performance (3.2.1.2).
  • Protocol for go/no-go decisions is relatively well-defined (4.1.3).
Weaknesses
  • Lacking risk indicators (2.2.1).
  • No justification for why certain risks are included or excluded (1.1.2).
  • No genuine risk modeling (1.3).
  • Only a few mitigations are mentioned, and they are not connected to risk indicators or risk domains (3.1).
Documents analyzed

We analyzed the policies outlined in Naver’s AI Safety Framework (ASF) from August 2024.

Naver

1. Risk Identification

Very Weak 7%

1.1 Classification of Applicable Known Risks (40%) 13%

1.1.1 Risks from literature and taxonomies are well covered (50%) 25%

They outline loss of control risks and biological/chemical risks, but not nuclear or radiological risks, nor AI R&D or manipulation, and 1.1.2 is less than 50%. They also do not break down loss of control risks further.

To improve, they should also reference literature that informs their risk identification process, as opposed to just “harms of AI that many people voice concern over”. This is to ensure risk domains highlighted by experts are not missed.

Quotes:

“The potential harms of AI that many people voice concern over broadly fall into one of two categories: “loss of control” and “misuse” risks.

The former concerns the fear of losing control over AI systems as they become more sophisticated, while the latter refers to the possibility of people deliberately manipulating these systems to catastrophic effect. AI’s technological limitations are also a key point in discussions about trust and safety.

NAVER’s AI Safety Framework defines the first category of risk as AI systems causing severe disempowerment of the human species. By this definition, this loss of control risk goes far beyond the implications of current AI-enabled automation, which stems from the concern that AI systems could spiral out of human control at the pace they are advancing. At NAVER, we take this risk seriously as we continually apply our standards to look for signs of alarm.

Our AI Safety Framework describes the second risk category as misusing AI systems to develop hazardous biochemical weapons or otherwise use them against their original purpose. To mitigate such risks, we have to place appropriate safeguards around AI technology. NAVER has taken a wide range of technological and policy actions so far and will continue to work toward achieving AI safety.”

1.1.2 Exclusions are clearly justified and documented (50%) 0%

There is no justification for why they have excluded certain categories of risk, such as nuclear or radiological risks, AI R&D and manipulation.

Quotes:

No relevant quotes found.

1.2 Identification of Unknown Risks (Open-ended red teaming) (20%) 0%

1.2.1 Internal open-ended red teaming (70%) 0%

The framework doesn’t mention any procedures pre-deployment to identify novel risk domains or risk models for the frontier model. To improve, they should commit to such a process to identify either novel risk domains, or novel risk models/changed risk profiles within pre-specified risk domains (e.g. emergence of an extended context length allowing improved zero shot learning changes the risk profile), and provide methodology, resources and required expertise.

Quotes:

No relevant quotes found.

1.2.2 Third party open-ended red teaming (30%) 0%

The framework doesn’t mention any third-party procedures pre-deployment to identify novel risk domains or risk models for the frontier model. To improve, they should commit to an external process to identify either novel risk domains, or novel risk models/changed risk profiles within pre-specified risk domains (e.g. emergence of an extended context length allowing improved zero shot learning changes the risk profile), and provide methodology, resources and required expertise.

Quotes:

No relevant quotes found.

1.3 Risk modeling (40%) 4%

1.3.1 The company uses risk models for all the risk domains identified
and the risk models are published (with potentially dangerous
information redacted) (40%) 10%

Whilst they indicate that they “determine whether an AI system […] can cause potential harm in special use cases”, suggesting some form of risk assessment that takes use cases into account, this doesn’t necessarily imply that they are conducting risk models, i.e. determining step by step causal pathways which could lead to harmful scenarios. Nonetheless, the reference to specific use cases is given partial credit here, as it shows an awareness of modeling ways in which AI systems may be used that would lead to harm.

Further, whilst “collaborate with different teams to identify and calculate the probability of risks across the entire lifecycle” doesn’t reference risk modelling explicitly, it does imply some form of identifying different risk pathways, which is given partial credit here.

Quotes:

“Determine whether an AI system designed to serve a certain purpose can cause potential harm in special use cases”

“Collaborate with different teams to identify and calculate the probability of risks across the entire lifecycle”

1.3.2 Risk modeling methodology (40%) 0%

1.3.2.1 Methodology precisely defined (70%) 0%

There is no methodology for risk modeling defined.

Quotes:

No relevant quotes found.

1.3.2.2 Mechanism to incorporate red teaming findings (15%) 0%

No mention of risks identified during open-ended red teaming or evaluations triggering further risk modeling.

Quotes:

No relevant quotes found.

1.3.2.3 Prioritization of severe and probable risks (15%) 0%

There is no indication that the most severe/probable harms are prioritized.

Quotes:

No relevant quotes found.

1.3.3 Third party validation of risk models (20%) 0%

There is no reference to third parties validating risk models.

Quotes:

No relevant quotes found.

Back to top
Naver

2. Risk Analysis and Evaluation

Very Weak 7%

2.1 Setting a Risk Tolerance (35%) 3%

2.1.1 Risk tolerance is defined (80%) 3%

2.1.1.1 Risk tolerance is at least qualitatively defined for all risks (33%) 10%

There is a very weak reference to a risk tolerance, as “AI systems causing severe disempowerment of the human species” and “misusing AI systems to develop hazardous biochemical weapons or otherwise use them against their original purpose.” However, to improve, they must set out the maximum amount of risk the company is willing to accept, for each risk domain (though they need not differ between risk domains), ideally expressed in terms of probabilities and severity (economic damages, physical lives, etc), and separate from KRIs.

Quotes:

“NAVER’s AI Safety Framework defines the first category of risk as AI systems causing severe disempowerment of the human species”

“Our AI Safety Framework describes the second risk category as misusing AI systems to develop hazardous biochemical weapons or otherwise use them against their original purpose.”

2.1.1.2 Risk tolerance is expressed at least partly quantitatively as a combination of scenarios (qualitative) and probabilities (quantitative) for all risks (33%) 0%

The risk tolerance, implicit or otherwise, is not expressed fully or partly quantitatively. To improve, the risk tolerance should be expressed fully quantitatively or as a combination of scenarios with probabilities.

Quotes:

No relevant quotes found.

2.1.1.3 Risk tolerance is expressed fully quantitatively as a product of severity (quantitative) and probability (quantitative) for all risks (33%) 0%

There is no evidence that risk tolerances are expressed partly or fully quantitatively.

Quotes:

No relevant quotes found.

2.1.2 Process to define the tolerance (20%) 0%

2.1.2.1 AI developers engage in public consultations or seek guidance from regulators where available (50%) 0%

No evidence of asking the public what risk levels they find acceptable. No evidence of seeking regulator input specifically on what constitutes acceptable risk levels.

Quotes:

No relevant quotes found.

2.1.2.2 Any significant deviations from risk tolerance norms established in other industries is justified and documented (e.g., cost-benefit analyses) (50%) 0%

No justification process: No evidence of considering whether their approach aligns with or deviates from established norms.

Quotes:

No relevant quotes found.

2.2 Operationalizing Risk Tolerance (65%) 9%

2.2.1 Key Risk Indicators (KRI) (30%) 15%

2.2.1.1 KRI thresholds are at least qualitatively defined for all risks (45%) 10%

No KRIs are given for loss of control risks, and for the misuse risk category there is only the indication of a KRI from “Determine whether an AI system designed to serve a certain purpose can cause potential harm in special use cases.” However, this provides no detail on what the KRI threshold is or what they are tracking. To improve, they should design and implement KRIs based off robust risk modelling.

Quotes:

“Determine whether an AI system designed to serve a certain purpose can cause potential harm in special use cases”

2.2.1.2 KRI thresholds are quantitatively defined for all risks (45%) 0%

KRIs are not defined quantitatively.

Quotes:

No relevant quotes found.

2.2.1.3 KRIs also identify and monitor changes in the level of risk in the external environment (10%) 0%

The KRIs only mention model capabilities.

Quotes:

No relevant quotes found.

2.2.2 Key Control Indicators (KCI) (30%) 4%

2.2.2.1 Containment KCIs (35%) 5%
2.2.2.1.1 All KRI thresholds have corresponding qualitative containment KCI thresholds (50%) 10%

The containment KCIs given are very vaguely related to KRIs, e.g. “For special use cases, make AI systems available only to authorized users” and “Open AI systems only to authorized users to mitigate risks” More detail is required on what threshold containment measures must meet, e.g. what constitutes “authorized users”, and under what risk models.

Quotes:

“For special use cases, make AI systems available only to authorized users”

“Open AI systems only to authorized users to mitigate risks”

2.2.2.1.2 All KRI thresholds have corresponding quantitative containment KCI thresholds (50%) 0%

Containment KCI thresholds given are not quantitative.

Quotes:

“For special use cases, make AI systems available only to authorized users”

“Open AI systems only to authorized users to mitigate risks”

2.2.2.2 Deployment KCIs (35%) 5%
2.2.2.2.1 All KRI thresholds have corresponding qualitative deployment KCI thresholds (50%) 10%

There is a very vague reference to deployment KCIs with “Deploy AI systems only after implementing guardrails through technological and policy actions and risks have been sufficiently mitigated” and “Ensure special-use capabilities are restricted for general use cases.” However, the deployment KCI thresholds should describe precisely what “sufficient mitigation” constitutes. More detail is required.

Quotes:

“Once AI systems are evaluated and their risks identified according to the two standards, we must implement appropriate guardrails around them. We should only deploy AI systems if those safeguards have proven effective in mitigating risks and keep an eye on the systems even after deployment through continuous monitoring. In theory, there may be cases where AI systems are used for special purposes and require safety guardrails in place, in which case AI systems should not be deployed.”

2.2.2.2.2 All KRI thresholds have corresponding quantitative deployment KCI thresholds (50%) 0%

There are no quantitative deployment KCI thresholds given.

Quotes:

No relevant quotes found.

2.2.2.3 For advanced KRIs, assurance process KCIs are defined (30%) 0%

There are no assurance processes KCIs defined. The framework does not provide recognition of there being KCIs outside of containment and deployment measures.

Quotes:

No relevant quotes found.

2.2.3 Pairs of thresholds are grounded in risk modeling to show that risks remain below the tolerance (20%) 10%

There is some awareness that KCI implementation must leave risks “sufficiently mitigated”, but justification is not given for why, if the KRI threshold is crossed but the KCI threshold is met, the residual risk remains below this risk tolerance (i.e. sufficiently mitigated).

Quotes:

“Deploy AI systems only after implementing guardrails through technological and policy actions and risks have been sufficiently mitigated”

 

2.2.4 Policy to put development on hold if the required KCI threshold cannot be achieved, until sufficient controls are implemented to meet the threshold (20%) 10%

There is a commitment to “delay” or “withhold” deployment if KCI thresholds are not met (implied by “until risks are mitigated”). However, the exact KCI thresholds required for this are not specified.

Importantly, no KCI threshold is given that would trigger putting development on hold.

Quotes:

“Delay deploying AI systems until risks are mitigated and appropriate technological and policy actions have been taken”

If the use case is General purpose and need for safety guardrails high, then “Withhold deployment until additional safety measures are taken”

If the use case is Special purpose and need for safety guardrails high, then “Do not deploy AI systems”

Back to top
Naver

3. Risk Treatment

Very Weak 8%

3.1 Implementing Mitigation Measures (50%) 4%

3.1.1 Containment measures (35%) 0%

3.1.1.1 Containment measures are precisely defined for all KCI thresholds (60%) 0%

No containment measures are described.

Quotes:

No relevant quotes found.

3.1.1.2 Proof that containment measures are sufficient to meet the thresholds (40%) 0%

No proof is provided that the containment measures are sufficient to meet the containment KCI thresholds, nor process for soliciting such proof.

Quotes:

No relevant quotes found.

3.1.1.3 Strong third party verification process to verify that the containment measures meet the threshold (100% if 3.1.1.3 > [60% x 3.1.1.1 + 40% x 3.1.1.2]) 0%

There is no mention of third-party verification that containment measures meet the threshold.

Quotes:

No relevant quotes found.

3.1.2 Deployment measures (35%) 10%

3.1.2.1 Deployment measures are precisely defined for all KCI thresholds (60%) 10%

The only deployment measures described are to “build guardrails by restricting special-use capabilities”. Much more detail is required on the measures that will actually be implemented to satisfy the deployment KCI threshold.

Quotes:

“For general use cases, build guardrails by restricting special-use capabilities”

 

3.1.2.2 Proof that deployment measures are sufficient to meet the thresholds (40%) 10%

No proof is provided that the deployment measures are sufficient to meet the deployment KCI thresholds, though there is an acknowledgment that proof is necessary before deployment.

Quotes:

“We should only deploy AI systems if those safeguards have proven effective in mitigating risks and keep an eye on the systems even after deployment through continuous monitoring.”

3.1.2.3 Strong third party verification process to verify that the deployment measures meet the threshold (100% if 3.1.2.3 > [60% x 3.1.2.1 + 40% x 3.1.2.2]) 0%

There is no mention of third-party verification of deployment measures meeting the threshold.

Quotes:

No relevant quotes found.

3.1.3 Assurance processes (30%) 0%

3.1.3.1 Credible plans towards the development of assurance processes (40%) 0%

There are no indications of plans to develop assurance processes nor mention of assurance processes in the framework. There are no indications to contributing to the research effort to ensure assurance processes are in place when they are required.

Quotes:

No relevant quotes found.

3.1.3.2 Evidence that the assurance processes are enough to achieve their corresponding KCI thresholds (40%) 0%

There is no mention of providing evidence that the assurance processes are sufficient.

Quotes:

No relevant quotes found.

3.1.3.3 The underlying assumptions that are essential for their effective implementation and success are clearly outlined (20%) 0%

There is no mention of the underlying assumptions that are essential for the effective implementation and success of assurance processes.

Quotes:

No relevant quotes found.

3.2 Continuous Monitoring and Comparing Results with Pre-determined Thresholds (50%) 13%

3.2.1 Monitoring of KRIs (40%) 23%

3.2.1.1 Justification that elicitation methods used during the evaluations are comprehensive enough to match the elicitation efforts of potential threat actors (30%) 0%

There is no description of elicitation methods, nor justification that these are comprehensive enough to match the elicitation efforts of potential threat actors.

Quotes:

No relevant quotes found.

3.2.1.2 Evaluation frequency (25%) 90%

They mention evaluation frequency in terms of time periods (every 3 months), and by performance gains (“when performance increases by 6x”), whichever is sooner. More detail could be given on what defines “performance”, i.e. on what tasks. They also mention that “the amount of computing can serve as an indicator when measuring capabilities” – to improve, they should specify evaluation frequency based on the amount of computations that have been executed during model development.

Quotes:

“The risk assessment scale examines risks in the “loss of control” category to see whether they are positively correlated with the advancement of AI systems. LLMs should be subject to periodic reviews or assessed whenever major performance improvements are made.”

Frontier AI, Evaluation cycle: “Every 3 months, or when performance increases by 6x” and “Frontier AI possesses the top capabilities that are available today or will be soon in the near future. Our goal is to have AI systems evaluated quarterly to mitigate loss of control risks, but when performance is seen to have increased six times, they will be assessed even before the three-month term is up. Because the performance of AI systems usually increases as their size gets bigger, the amount of computing can serve as an indicator when measuring capabilities.”

3.2.1.3 Description of how post-training enhancements are factored into capability assessments (15%) 0%

There is no description of how post-training enhancements are factored into capability assessments, nor safety margins given.

Quotes:

No relevant quotes found.

3.2.1.4 Vetting of protocols by third parties (15%) 0%

There is no mention of having the evaluation methodology vetted by third parties.

Quotes:

No relevant quotes found.

3.2.1.5 Replication of evaluations by third parties (15%) 0%

There is no mention of evaluations being replicated or conducted by third parties.

Quotes:

No relevant quotes found.

3.2.2 Monitoring of KCIs (40%) 10%

3.2.2.1 Detailed description of evaluation methodology and justification that KCI thresholds will not be crossed unnoticed (40%) 25%

There is an acknowledgment that safeguards msut have “proven effective in mitigating risks”, and that continuous monitoring should also verify this. More detail on the process and methodology for this monitoring however should be given.

Quotes:

If general purpose use case and low need for safety guardrails: “Deploy AI systems but perform monitoring afterward to manage risks”

“We should only deploy AI systems if those safeguards have proven effective in mitigating risks and keep an eye on the systems even after deployment through continuous monitoring.”

3.2.2.2 Vetting of protocols by third parties (30%) 0%

There is no mention of KCIs protocols being vetted by third parties.

Quotes:

No relevant quotes found.

3.2.2.3 Replication of evaluations by third parties (30%) 0%

There is no mention of control evaluations/mitigation testing being replicated or conducted by third-parties.

Quotes:

No relevant quotes found.

3.2.3 Transparency of evaluation results (10%) 0%

3.2.3.1 Sharing of evaluation results with relevant stakeholders as appropriate (85%) 0%

There is no commitment to publicly share evaluation results, nor to notify relevant government authorities if KRI thresholds are crossed.

Quotes:

No relevant quotes found.

3.2.3.2 Commitment to non-interference with findings (15%) 0%

No commitment to permitting the reports, which detail the results of external evaluations (i.e. any KRI or KCI assessments conducted by third parties), to be written independently and without interference or suppression.

Quotes:

No relevant quotes found.

3.2.4 Monitoring for novel risks (10%) 0%

3.2.4.1 Identifying novel risks post-deployment: engages in some process (post deployment) explicitly for identifying novel risk domains or novel risk models within known risk domains (50%) 0%

There is some indication that novel risks will arise from AI systems which cannot be anticipated: “Evaluation cycle: To be determined later depending on their future capabilities”. However, there is no mechanism for monitoring and identifying novel risks post-deployment.

Quotes:

“Evaluation cycle: To be determined later depending on their future capabilities”

 

3.2.4.2 Mechanism to incorporate novel risks identified post-deployment (50%) 0%

There is no mechanism to incorporate risks identified during post-deployment that is detailed.

Quotes:

No relevant quotes found.

Back to top
Naver

4. Risk Governance

Very Weak 18%

4.1 Decision-making (25%) 13%

4.1.1 The company has clearly defined risk owners for every key risk identified and tracked (25%) 0%

No mention of risk owners.

Quotes:

No relevant quotes found.

4.1.2 The company has a dedicated risk committee at the management level that meets regularly (25%) 0%

No mention of a management risk committee.

Quotes:

No relevant quotes found.

4.1.3 The company has defined protocols for how to make go/no-go decisions (25%) 50%

The framework uses clear risk assessment matrices for deployment decisions, but does not provide full detail on the basis for decision-making or who makes the decisions.

Quotes:

“Need for safety guardrails: Low High, Use cases: General purpose, Special purpose” with corresponding actions.” (p. 5)
“For special use cases, make AI systems available only to authorized users – For general use cases, build guardrails by restricting special-use capabilities”. (p. 5)
“Delay deploying AI systems until risks are mitigated and appropriate technological and policy actions have been taken”. (p. 5)
“Once AI systems are evaluated and their risks identified according to the two standards, we must implement appropriate guardrails around them. We should only deploy AI systems if those safeguards have proven effective in mitigating risks”. (p. 6)

4.1.4 The company has defined escalation procedures in case of incidents (25%) 0%

No mention of escalation procedures.

Quotes:

No relevant quotes found.

4.2. Advisory and Challenge (20%) 12%

4.2.1 The company has an executive risk officer with sufficient resources (16.7%) 0%

No mention of an executive risk officer.

Quotes:

No relevant quotes found.

4.2.2 The company has a committee advising management on decisions involving risk (16.7%) 25%

The framework references a Future AI Center which might serve an advisory capacity.

Quotes:

“The Future AI Center, which brings together different teams for discussions on the potential risks of AI systems at the field level”. (p. 7)

4.2.3 The company has an established system for tracking and monitoring risks (16.7%) 0%

No mention of a risk tracking system.

Quotes:

No relevant quotes found.

4.2.4 The company has designated people that can advise and challenge management on decisions involving risk (16.7%) 10%

The framework mentions a Future AI Center that potentially plays a role in challenging decisions.

Quotes:

“The Future AI Center, which brings together different teams for discussions on the potential risks of AI systems at the field level”. (p. 7)

4.2.5 The company has an established system for aggregating risk data and reporting on risk to senior management and the Board (16.7%) 10%

The framework references a working group that raises issues to the Board.

Quotes:

“The risk management working group whose role is to determine which of these issues to raise to the board”. (p. 7)

4.2.6 The company has an established central risk function (16.7%) 25%

While it is uncertain what exact role it plays, the framework references a risk management working group that seems to be playing somewhat of this type of role.

Quotes:

“The risk management working group whose role is to determine which of these issues to raise to the board”. (p. 7)

4.3 Audit (20%) 5%

4.3.1 The company has an internal audit function involved in AI governance (50%) 0%

No mention of an internal audit function.

Quotes:

No relevant quotes found.

4.3.2 The company involves external auditors (50%) 10%

The framework mentions external stakeholders, but not auditors or independent reviews.

Quotes:

“We work with external stakeholders to take on challenges surrounding safe AI technologies and services.” (p. 7)

4.4 Oversight (20%) 45%

4.4.1 The Board of Directors of the company has a committee that provides oversight over all decisions involving risk (50%) 90%

The framework makes clear that there is a risk management committee of the Board that makes decisions regarding risk.

Quotes:

“The board (or the risk management committee) [makes] the final decisions on the matter” (p. 7)

4.4.2 The company has other governing bodies outside of the Board of Directors that provide oversight over decisions (50%) 0%

No mention of any additional governance bodies.

Quotes:

No relevant quotes found.

4.5 Culture (10%) 8%

4.5.1 The company has a strong tone from the top (33.3%) 25%

The framework sets a fairly strong tone from the top with statements regarding the company’s view on risk.

Quotes:

“NAVER takes a human-centric approach to developing AI, and our aim is to help people benefit from AI by turning this technology into a daily tool.” (p. 1)
“Since introducing NAVER’s AI Principles in 2021, human-centered AI development has always been the focus of our efforts.” (p. 1)
“NAVER’s AI Safety Framework defines the first category of risk as AI systems causing severe disempowerment of the human species… At NAVER, we take this risk seriously”. (p. 2)
“Our AI Safety Framework is designed to address societal concerns around AI safety. We identify, assess, and manage risks at all stages of AI systems operations, from development to deployment.” (p. 2)

4.5.2 The company has a strong risk culture (33.3%) 0%

No mention of elements of risk culture.

Quotes:

No relevant quotes found.

4.5.3 The company has a strong speak-up culture (33.3%) 0%

No mention of elements of speak-up culture.

Quotes:

No relevant quotes found.

4.6 Transparency (5%) 32%

4.6.1 The company reports externally on what their risks are (33.3%) 10%

The framework references risks only at the level of loss of control and misuse.

Quotes:

“The potential harms of AI that many people voice concern over broadly fall into one of two categories: “loss of control” and “misuse” risks.” (p. 2)

4.6.2 The company reports externally on what their governance structure looks like (33.3%) 75%

The framework has a dedicated governance section which lists the main governance bodies.

Quotes:

“NAVER’s AI Safety Framework is our initiative to achieve AI governance. Under our governance, we foster collaboration between cross-functional teams to identify, evaluate, and manage risks when developing AI systems.”

NAVER’s AI governance includes:

“The Future AI Center, which brings together dierent teams for discussions on the potential risks of AI systems at the field level…The risk management working group whose role is to determine which of these issues to raise to the board…The board (or the risk management committee) that makes the final decisions on the matter”. (p. 7)

4.6.3 The company shares information with industry peers and government bodies (33.3%) 10%

The framework mentions working with external stakeholders, but not necessarily sharing relevant information.

Quotes:

“We work with external stakeholders to take on challenges surrounding safe AI technologies and services.” (p. 7)
“We partner with top universities like Seoul National University (SNU) and Korea Advanced Institute of Science & Technology (KAIST) on the technology front and participate in the SNU AI Policy Initiative on the policy front.” (p. 7)

Back to top