Generative artificial intelligence (GenAI) red-teaming must evolve by adopting lessons from cybersecurity if it is to keep pace with rising risks, according to a new report from the Software Engineering Institute (SEI).
Red-teaming, which tests systems by simulating attacks, is increasingly used to probe generative AI models. However, SEI researchers warn that current approaches are inconsistent and lack clear standards, undermining confidence in AI safety across industry, government, and consumers.
The study, What Can Generative AI Red-Teaming Learn from Cyber Red-Teaming?, identifies major challenges, including fragmented evaluation methods, limited threat modelling, and weak mitigation strategies. By contrast, cybersecurity red-teaming has matured over decades into a structured discipline with frameworks, tooling, and community collaboration.
To close the gap, the authors recommend ten improvements for AI security, adapted from cyber practice:
- Incorporate realistic threat models.
- Expand attack surface considerations.
- Integrate cyber operational stages.
- Ensure actionable mitigations.
- Bridge the gap between evaluators and model developers.
- Develop open source tooling.
- Diversify red-teaming techniques.
- Enhance automation for scalability.
- Standardize vulnerability identification.
- Develop authoritative manuals and guidelines.
Generative AI models do not exist in a bubble. They are just one component of an increasingly complex software stack. Over decades of red-teaming software, cybersecurity practitioners have built a comprehensive ecosystem of tools, processes, and community involvement. AI security practitioners bring expertise in novel AI-specific threats. Their collaboration is imperative for assessing the entire threat landscape and ensuring the security of AI systems.
SEI researcher Keltin Grimes
