The risks of GenAI risk assessment: beyond the obvious

By David Honour

Resilience Forward recently highlighted the issue of probabilistic drift, an inherent characteristic of most large language models. This isn’t a design flaw; it is an aspect of probability. However, it results in inconsistent answers to the same question or prompt – and that creates an issue if organizations are relying on generative AI for risk assessments or decision support. That variability becomes a risk factor in itself.

Probabilistic drift isn’t the only area where relying on GenAI for risk assessments, horizon scanning, and related scenario planning brings its own risks. The attraction is obvious: AI offers speed, clarity, and the ability to synthesise vast quantities of information into neat, authoritative insights. But here lies the paradox. Because AI communicates in a voice of confidence, charm, and apparent empathy, distortions can be subtle, persuasive, and easily overlooked.

The dark psychology of AI

Generative AI does not have intent, ego, or emotion. Yet the style of its outputs often mirrors what psychology calls the Dark Triad: narcissism, Machiavellianism, and psychopathy. These traits, when embodied in human leaders, can be simultaneously attractive and dangerous. The same is true when echoed in AI systems.

Narcissistic echoes: AI presents insights with certainty, vision, and authority. As a result, it is highly believable, even when it presents conjecture as fact; and its responses often lack nuance. Where it lacks solid information, AI will make a best guess rather than giving an I’m not sure answer. However, this guess will often be perceived as fact by the recipient.
Machiavellian echoes: AI reframes and persuades, steering conversations in ways that feel constructive. This is effective for engagement – but can also be manipulative.
Psychopathic echoes: AI remains calm, unemotional, and detached. It has no empathy, although it may appear empathetic at a surface level. When assessing information that requires an understanding of human impacts, GenAI will fail to do this effectively.

These echoes are not accidental. They reflect training data drawn largely from academic, corporate, and thought-leadership sources – domains where confidence, persuasion, and detachment often dominate. In other words: the culture of leadership and risk thinking is already saturated with Dark Triad energy; and AI mirrors it back to us.

In addition, developers often tune models to prioritise helpfulness, coherence, and confidence, which reinforces the narcissistic tendency to not admit error or ignorance, regardless of the training data’s subject matter.

Hidden risks for risk professionals

When applied to horizon scanning and risk assessment, the above creates several hidden dangers:

False certainty: overconfidence in outputs can encourage professionals into narrow thinking, discouraging the exploration of uncertainty.
Manipulated perception: AI’s framing of risks can skew horizon scanning toward dominant narratives, creating its own blind spots.
Illusion of trust: simulated empathy and mirroring build rapport, lowering critical challenge functions.
Narrow horizons: weak signals outside mainstream discourse are easily overlooked, reinforcing groupthink. From an AI perspective the more an idea or concept is repeated by reputable sources, the more credence it is given. There is little room for those on the fringe who think outside the box.
De-skilling and dependency: over time, professionals may lose their sceptical edge as they grow reliant on AI structuring and analysis.

The danger is not dramatic bias in a single answer. It is the slow shaping of professional judgement by an endlessly charming manipulator!

Mitigations: guardrails for AI in risk work

If AI echoes Dark Triad styles, then the controls mirror those we would apply to human leaders with such traits:

Ensure there is human challenge: pair AI outputs with structured human dissent to challenge overconfidence. Any mission-critical decision-making that uses GenAI must also include an appropriately trained human capable of assessing and verifying the information being generated.
Force uncertainty: prompt AI for ranges, alternative framings, and blind spots, not just polished conclusions.
Diversify sources: use multiple AI systems alongside expert opinion and play AI systems off against each other, assessing the responses that come back with human insight.
Audit regularly: test AI-generated risk assessments against independent benchmarks.
Educate users: train risk professionals to recognise how AI’s charm, confidence, and apparent empathy can shape perceptions.

Conclusion

The risk is real: while AI has a role in risk assessment, organizations may absorb not just its speed and power, but also its distortions. This is a meta-risk that could leave organizations blind to the very uncertainties they sought to understand.

Generative AI is not malevolent. But it is persuasive, authoritative, and charming. Without boundaries, risk professionals may find themselves seduced by its credibility – and in doing so, undermining the very resilience they seek to build.