The US National Institute of Standards and Technology (NIST) is launching a new testing, evaluation, validation and verification program intended to help improve understanding of artificial intelligence’s capabilities and impacts.
Assessing Risks and Impacts of AI (ARIA) aims to help organizations determine whether a given AI technology will be valid, reliable, safe, secure, private, and fair once deployed.
The ARIA program is designed to meet real-world needs as the use of AI technology grows. This new effort will support the US AI Safety Institute, expand NIST’s already broad engagement with the research community, and help establish reliable methods for testing and evaluating AI’s functionality in the real world.
Laurie E. Locascio, Under Secretary of Commerce for Standards and Technology and NIST Director
ARIA expands on the AI Risk Management Framework, which NIST released in January 2023, and helps to operationalize the framework’s risk measurement function, which recommends that quantitative and qualitative techniques be used to analyze and monitor AI risk and impacts. ARIA will help assess those risks and impacts by developing a new set of methodologies and metrics for quantifying how well a system maintains safe functionality within societal contexts.
“Measuring impacts is about more than how well a model functions in a laboratory setting,” said Reva Schwartz, NIST Information Technology Lab’s ARIA program lead. “ARIA will consider AI beyond the model and assess systems in context, including what happens when people interact with AI technology in realistic settings under regular use. This gives a broader, more holistic view of the net effects of these technologies.”
The results of ARIA will support and inform NIST’s collective efforts, including through the US AI Safety Institute, to build the foundation for safe, secure and trustworthy AI systems.
