NIST has finalized a new guidance document to assist people designing, developing, deploying, evaluating, and governing AI systems, with the issue of adversarial machine learning.
As the title suggests,‘Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations’ (NIST AI.100-2e2025) describes a taxonomy and terminology for adversarial machine learning (AML) that may aid in securing applications of artificial intelligence against manipulations and attacks.
Adversarial machine learning is basically about fooling machine learning (ML) models on purpose. In a nutshell, it’s the deployment of techniques that try to trick machine learning models by giving them specially crafted inputs — called adversarial examples — that look normal to humans but cause the model to provide incorrect outputs.
The NIST guidance states that AML challenges span different phases of ML operations such as the potential for adversarial manipulation of training data; the provision of adversarial inputs to adversely affect the performance of the AI system; and even malicious manipulations, modifications, or interactions with models to exfiltrate sensitive information from the model’s training data or to which the model has access. Such attacks have been demonstrated under real-world conditions, and their sophistication and impacts have been increasing steadily.
To taxonimize these attacks, the report differentiates between predictive and generative AI systems and the attacks relevant to each. It considers the components of an AI system including the data; the model itself; the processes for training, testing, and deploying the model; and the broader software and system contexts into which models may be embedded.
The guidance adopts the concepts of security, resilience, and robustness of ML systems from the NIST AI Risk Management Framework. Security, resilience, and robustness are gauged by risk. However, this report does not make recommendations on risk tolerance, because thi is highly contextual and specific to applications and use.
In addition to defining a taxonomy of attacks, the guidance provides corresponding methods for mitigating and managing the consequences of those attacks in the life cycle of AI systems and outlines the limitations of widely used mitigation techniques to raise awareness and help organizations increase the efficacy of their AI risk-mitigation efforts.
