Large AI models are increasingly being used to power agent based systems which can automate complex tasks on behalf of users. AI agents bring many potential benefits but also introduce a new risk: AI agent hijacking.
A new technical article published by the US NIST looks at this area and provides four insights from the U.S. AI Safety Institute (US AISI) to help assess and manage AI agent hijacking risks.
These are:
- Continuous improvement and expansion of shared evaluation frameworks is important.
- Evaluations need to be adaptive. Even as new systems address previously known attacks, red teaming can reveal other weaknesses.
- When assessing risk, it can be informative to analyze task-specific attack performance in addition to aggregate performance.
- Testing the success of attacks on multiple attempts may yield more realistic evaluation results.
The article concludes that AI agent hijacking will continue to be a persistent challenge as agentic systems continue to evolve:
- Strengthening and expanding evaluations for agent security issues like hijacking will help users understand and manage these risks as they seek to deploy agentic AI systems in a variety of applications.
- Some defenses against hijacking attacks are available and continuing to evaluate the efficacy of these defenses against new attacks is another important area for future work in agent security.
- Developing defensive measures and practices that provide stronger protection, as well as the evaluations needed to validate their efficacy, will be essential to unlocking the many benefits of agents for innovation and productivity.
