Anthropic has published an Advanced AI Framework which includes proposals for how governments should address catastrophic risks from the most powerful AI models.
Four kinds of catastrophic risk are highlighted by Anthropic :
- Biological risk. If AI systems are released without safeguards, it could become substantially easier to develop biological weapons. The same capabilities that accelerate drug discovery can be used to make it cheaper and easier for attackers to develop dangerous viruses.
- Cyber risk. Frontier AI models can now find critical software vulnerabilities at large scale. Used defensively, these capabilities can secure critical systems, but they also raise the stakes for protecting essential infrastructure like hospitals and the energy grid.
- Loss of control risk. As AI systems improve, it could become much harder to control systems that act outside of their developers’ control.
- Automated R&D. AI systems are automating the research and development of AI itself, which could further amplify the three above risks.
The framework calls for government action and for regulations ‘that are carefully designed to prevent government overreach and protect innovation’.
Frontier AI developers should have to test models, be transparent to the public about their findings, submit them to independent evaluation, and maintain a robust security programme.
