Managing AI hallucination risk: a guide for enterprise risk managers

As artificial intelligence becomes more deeply embedded in enterprise workflows, a new class of risks has emerged which needs managing: AI hallucinations.

Hallucinations occur when AI systems generate outputs that are factually incorrect, misleading, or fabricated. These outputs may appear confident and credible, but they lack a foundation in reality. For enterprise risk managers, understanding and mitigating the impact of AI hallucinations is now a critical component of technology risk governance.

This article explores the nature of AI hallucinations, presents evidence on hallucination rates across different AI platforms, and provides a practical framework for identifying, managing, and mitigating these risks in enterprise settings.

What are AI hallucinations?

AI hallucinations are outputs from language models that contain fabricated or incorrect information. They often occur in generative AI systems when the model lacks reliable grounding for its response or misinterprets the user’s prompt. Hallucinations can take many forms:

Invented facts or citations
Incorrect calculations or logic
Misstatements of policy or law
Fabricated data in visual or tabular form.

While sometimes benign or even a positive thing in creative tasks, hallucinations pose significant risks in domains like healthcare, legal, finance, and compliance.

Evidence of hallucination rates across large language models (LLM)

Vectara, a company that provides an AI Agent and Assistant Platform for enterprises, maintains an index of LLM hallucination rates. As of April 29th 2025, this shows that LLM hallucination rates range from 0.7% for the Google Gemini-2.0-Flash-001 LLM to 29.9% for the TII falcon-7B-instruct LLM. A key fact here is that, even in the best performing LLM, 7 out of every 1000 prompts will produce hallucinations.

Why do AI hallucinations happen?

Several factors contribute to AI hallucinations:

Training data gaps: if a model hasn’t seen enough examples on a topic, it may generalize incorrectly.
Ambiguous prompts: vague or open-ended prompts lead the model to ‘guess’ what the user wants.
Overconfidence: models are trained to sound confident, even when wrong.
No grounding mechanism: without real-time access to verified data the model relies on learned patterns.

A risk management framework for AI hallucinations

Enterprise risk managers should approach AI hallucinations as part of their broader digital and operational risk portfolio. Steps include:

1. Risk identification

Where is the AI used? Map all business functions using AI tools. This may need some deep investigation due to the prevalence of shadow AI use amongst employees.
Where are the risks the highest? Identify where hallucinations could cause strategic, legal, reputational, financial, or safety risks.
What type of AI model is being used? Determine whether it’s open-source, proprietary, or fine-tuned. This is a starting point for determining potential hallucination levels.

2. Risk assessment

As usual, evaluate the likelihood and impact of hallucinations in each AI use case.
Use red-teaming or adversarial prompts to test systems under stress.
Ask: What happens if the AI information is wrong?

3. Potential mitigation strategies

Risk area	Mitigation strategy
Model selection	Use models with lower hallucination rates
Prompt design	Use structured, unambiguous prompts. Avoid open-ended requests in high-risk areas.
Retrieval-augmented generation (RAG)	Connect the AI to a database or knowledge base of trusted content. This grounds responses in fact.
Human-in-the-loop	Require human oversight for critical outputs, such as legal decisions or customer communication.
Content verification	Use cross-model validation or external fact-checking tools.
Fine-tuning	Tailor the AI to enterprise-specific content with verified data.

4. Monitoring and controls

Establish logging and audit mechanisms for AI outputs.
Create dashboards to flag high-risk outputs or deviations.
Collect user feedback to identify hallucination patterns.

5. Governance and policy

Develop an enterprise-wideAI Acceptable Use Policy.
Define accountability for AI outputs (e.g., business unit ownership).
Provide training on hallucination awareness for end users.
Include hallucination risks into the organizational risk register (s).

While AI hallucinations are an inherent challenge in generative AI, they are not unmanageable. With the right combination of model selection, prompt engineering, human oversight, and governance, enterprises can contain the operational and reputational risks of hallucinated outputs.

For risk managers, the key is to treat AI like any other complex system: with clear documentation, rigorous testing, and adaptive controls. As the technology matures, so too must the frameworks for accountability and resilience.