1.1.1 Risks from literature and taxonomies are well covered (50%) 10%
They don’t list specific risk domains that their risk management process focuses on ex ante. Rather, risk domains are identified for particular customers and use cases. However, their risk domains focus on malicious use and bias, with examples in cybersecurity, child sexual exploitation, and discrimination. More detail on why they chose to focus on these issues and how they came to identify these risks is required, especially as they differ from the industry standard.
They explicitly do not consider CBRN or loss of control risks, and explicitly do not consider “potential future risks associated with LLMs”. This is a serious limitation that requires strong justification; given the harms from loss of control or CBRN could be substantial, dismissing monitoring these risks at all requires a high amount of confidence. However, 1.1.2 scores less than 50%. Further, it shows they have not engaged with literature – for instance, there is emphasis on these risks in documents such as the International Science of AI Safety Report and current drafts of the EU AI Act Codes of Practice.
Quotes:
“One approach to risk assurance in the AI industry is focused on risks described as catastrophic or severe, such as capabilities related to radiological and nuclear weapons, autonomy, and self-replication. In this context, thresholds relating to these potential catastrophic risks are developed, and the approach described in safety frameworks is designed to assess risks that are speculated to arise when models attain specific capabilities, such as the ability to perform autonomous research or facilitate biorisk. The models are then deemed to present “unacceptable” levels of risk when certain capability levels are attained. While it is important to consider long-term, potential future risks associated with LLMs and the systems in which they are deployed, studies regarding the likelihood of these capabilities arising and leading to real-world harm are limited in their methodological maturity and transparency, often lacking clear theoretical threat models or developed empirical methods due to their nascency. For example, existing research into how LLMs may increase biorisks fails to account for entire risk chains beyond access to information, and does not systematically compare LLMs to other information access tools, such as the internet. More work is needed to develop methods for assessing these types of threats more reliably.” (pp. 14-15)
“Cohere’s approach to risk assurance, and to determining when models and systems are suffi ciently safe and secure to be made available to our customers, is focused on risks that are known, measurable, or observable today” (p. 15)
“Limitations in training data, such as unrepresentative data distributions, historically outdated representations, or an imbalance between harmful patterns and attributes on the one hand and positive patterns and attributes on the other, also impact model capabilities. If these limitations are not mitigated, models can output harmful content, such as hateful or violent content, or child sexual exploitation and abuse material (CSAM).
We therefore focus our secure AI work on risks that have a high likelihood of occurring based on the types of tasks LLMs are highly performant in, as well as the limitations inherent in how these models function. This is what we refer to as “model capabilities.”
We place potential risks arising from LLM capabilities into one of two categories:
- Risks stemming from possible malicious use of foundation AI models, such as generating content to facilitate cybercrime or child sexual exploitation
- Risks stemming from possible harmful outputs in the ordinary, non-malicious use of foundation models, such as outputs that are inaccurate in a way that has a harmful impact on a person or a group” (p. 5)
“Cohere consistently reviews state-of-the-art research and industry practice regarding the risks associated with AI, and uses this to determine our priorities. At Cohere, risks to our systems are identifi ed through a list of continuously-expanding techniques, including:
- Mitigating core vulnerabilities identifi ed by the Open Worldwide Application Security Project (OWASP)
- Internal threat modeling, which includes a review of how our customers interact with and use our models, to proactively identify potential threats and implement specific counter measures before deployment
- Monitoring established and well-researched repositories of security attacks and vulnerabilities for AI, such as the Mitre Altas database
With these methods, Cohere can identify risks such as data poisoning, model theft, inference attacks, injection attacks, and output manipulation.” (p. 6)
Potential Harm: Outputs that result in a discriminatory outcome, insecure code, child sexual exploitation and abuse, malware.
“The examples provided above consider the likelihood and severity of potential harms in the enterprise contexts in which Cohere models are deployed. A similar assessment of potential harms from the same models deployed in contexts such as a consumer chatbot would result in a different risk profile.” (p. 8)
“Preventing the generation of harmful outputs involves testing and evaluation techniques to control the types of harmful output described in Section 1, for example, child sexual abuse material (CSAM), targeted violence and hate, outputs that result in discriminatory outcomes for protected groups, or insecure code.” (p. 11)
1.1.2 Exclusions are clearly justified and documented (50%) 10%
They explicitly do not consider CBRN or loss of control risks, and explicitly do not consider “potential future risks associated with LLMs”, giving justification that “studies regarding the likelihood of these capabilities arising and leading to real-world harm are limited in their methodological maturity and transparency, often lacking clear theoretical threat models or developed empirical methods due to their nascency.” However, this reasoning requires more documentation and justification, for instance citing these studies and why they believe their reasoning to be limited. Excluding a risk that is established in taxonomies and literature carries a high burden of proof.
Quotes:
“Cohere’s approach to risk assurance, and to determining when models and systems are suffi ciently safe and secure to be made available to our customers, is focused on risks that are known, measurable, or observable today” (p. 15)
“One approach to risk assurance in the AI industry is focused on risks described as catastrophic or severe, such as capabilities related to radiological and nuclear weapons, autonomy, and self-replication. In this context, thresholds relating to these potential catastrophic risks are developed, and the approach described in safety frameworks is designed to assess risks that are speculated to arise when models attain specific capabilities, such as the ability to perform autonomous research or facilitate biorisk. The models are then deemed to present “unacceptable” levels of risk when certain capability levels are attained. While it is important to consider long-term, potential future risks associated with LLMs and the systems in which they are deployed, studies regarding the likelihood of these capabilities arising and leading to real-world harm are limited in their methodological maturity and transparency, often lacking clear theoretical threat models or developed empirical methods due to their nascency. For example, existing research into how LLMs may increase biorisks fails to account for entire risk chains beyond access to information, and does not systematically compare LLMs to other information access tools, such as the internet. More work is needed to develop methods for assessing these types of threats more reliably.” (pp. 14-15)
