AI hallucination is a property of how models work, not a rare fault. Four defence layers, the UK cases behind them, and what to brief your speaker on.
Link copied
AI hallucination is not a rare malfunction. It is a property of how language models work: they generate the most probable next token, and probability is not truth. That distinction is why the defence has to be built into the system rather than bolted on afterwards.
In June 2025 the Divisional Court of the King's Bench Division heard two cases together under its Hamid jurisdiction. In the first, counsel had cited five authorities that did not exist. In the second, a schedule of forty-five citations was put before the court in an £89.4 million claim, and eighteen of them referred to cases that had never been decided.
The court's framing in Ayinde v Haringey and Al-Haroun v Qatar National Bank was precise: AI tools produce arguments "which are not then checked, so that false information (typically a fake citation or quotation) is put before the court." The barrister in the first case and the law centre instructing her were each ordered to pay £2,000 in wasted costs, with referrals to the Bar Standards Board and the Solicitors Regulation Authority.
Fourteen months later the SRA published a warning notice of its own.
The Solicitors Regulation Authority issued "Misuse of AI" on 17 August 2026. Its definition is worth quoting because it is a regulator's, not a vendor's:
> AI tools can produce 'hallucinations', generating fictitious cases, references or seemingly factual assertions that may appear convincing despite having no basis in fact.
The notice flags two distinct risks, and most commentary only reports the first. The second is that confidential client information is being pasted into public AI tools. On that point the SRA is blunt: doing so "will likely breach client confidentiality and as a result, legal professional privilege may be permanently waived and unable to be recovered."
It also confirms that reports have reached the regulator from senior members of the judiciary, and cites three 2026 decisions: a tribunal finding that putting client and Home Office letters into an open tool "is to place this information on the internet in the public domain", a family court finding that an unregistered barrister misled the court through AI-generated content, and a Chancery Division case.
The point for anyone outside the legal sector is that a professional regulator has now written down what a reasonable standard of care looks like around these tools. Other regulators tend to follow that template.
The most useful research here comes from Stanford's RegLab and HAI. Their study of legal queries, published in the Journal of Legal Analysis, found hallucination rates between 58% and 88% on specific, verifiable legal questions, and at least 75% on questions about a court's core holding.
Two things about that figure matter more than the number itself.
First, it moves. The widely circulated "69% to 88%" range came from the January 2024 preprint; the peer-reviewed version reports 58% once GPT-4 is included. Any single quoted rate has a shelf life.
Second, and more importantly for anyone buying tools: the same team tested commercial legal research products that use retrieval rather than raw generation. Those tools, marketed with "hallucination-free" language, still produced hallucinated output between 17% and 33% of the time.
That is the finding to sit with. Retrieval reduces the rate substantially. It does not eliminate it. A defence built on one layer is a defence built on a vendor's marketing claim.
Each layer is insufficient alone. Together they move hallucination from an unmanaged risk to a measured one.
Layer | What it does | Where it fails on its own |
1. Grounding (RAG) | Answers are drawn from an approved document store rather than the model's parametric memory, with citations shown | Retrieval can still surface the wrong passage, and the model can still misread it — 17-33% in tested commercial tools |
2. Human in the loop | Outputs above a defined error-cost threshold require sign-off from someone competent to judge them | Fails when the reviewer defers to the model rather than exercising independent judgement |
3. Ensemble verification | Two independent systems run in parallel; divergence triggers escalation | Doubles cost, so it is only justified above a high risk threshold |
4. Audit trail | Prompt, context, model version, retrieval results and output are logged, timestamped and reproducible | Retrospective only. It tells you what happened, not that it is happening now |
Layer one is technical. Retrieval-augmented generation points the model at an approved corpus — policy documents, product documentation, contractual terms — and requires it to answer from what it finds. The Moffatt v Air Canada decision in February 2024 is the clearest illustration of what its absence costs: an airline chatbot invented a bereavement fare policy, and the tribunal awarded damages on the basis that a company is responsible for everything on its website. The tribunal called the argument that the chatbot was a separate entity "a remarkable submission", and asked why a customer should have to check one part of a website against another.
Layer two is procedural. The question is not whether to use human review but where the threshold sits. Build an error-cost matrix: when the financial, legal or reputational cost of a wrong output crosses a defined line, that output goes through a person. The critical design point is independence. A reviewer who trusts the model is not a control; they are a formality. Ayinde is what layer two failing looks like in practice.
Layer three is expensive and narrow. Two independent systems running the same task, with divergence between them treated as its own alarm signal, is justified where a single wrong output is intolerable: clinical recommendations, transaction approval, regulatory interpretation. The disagreement is the value. It surfaces cases that a single system would have passed silently.
Layer four is the time dimension. Six months after a decision, can you reconstruct why the system produced that output? Audit trails are also the only way to learn where hallucination clusters: which topics, which prompt structures, which model versions. Those patterns exist nowhere else.
This is worth stating precisely, because the shorthand circulating in the market is wrong in both directions.
The UK has no AI statute. As of September 2026 there is no AI-specific legislation covering the technology; AI is regulated through existing frameworks in the context in which it is used. The pro-innovation, regulator-led approach set out in 2023 remains the operative model, and the only bill in Parliament is a Private Member's Bill that has had a first reading and no further stages. That is why the SRA notice matters so much: in the absence of a statute, regulator guidance is the standard.
The EU AI Act has not been "pushed to 2027". Regulation (EU) 2026/1744 of 8 July 2026 moved the high-risk obligations: Annex III systems to 2 December 2027, Annex I embedded systems to 2 August 2028. What did not move: the prohibitions, in force since February 2025; general-purpose AI model obligations, since August 2025; and the Article 50 transparency obligations, live since August 2026. A UK company selling into the EU is already subject to the parts that bite first.
The practical reading for a UK organisation: the timeline for the heaviest obligations widened, the obligation to have controls did not.
Sessions on this topic fail in a predictable way. They explain what hallucination is, everyone nods, and nothing changes in the workflow the following Monday.
Three things separate a session that changes practice from one that raises awareness:
Speakers on our UK roster work at different points of this problem: Toju Duke on responsible AI governance and how organisations build oversight structures; Henry Ajder on synthetic media and where authenticity breaks down; Daniel Hulme on the technical frontier and what the systems can and cannot do. These are examples of matching logic rather than recommendations, and no availability or fee is implied by their appearance here.
The governance end of that spectrum is the subject of this talk:
Toju Duke — Responsible AI at Google (RE·WORK)
If the audience is a leadership team rather than a technical one, the framing shifts from defence architecture to judgement, and our piece on the five competencies leadership needs in the AI era covers that angle.
1. Which of our AI outputs go to a customer, a court, a regulator or a counterparty without a person reading them first?
2. What is the error-cost threshold above which sign-off is mandatory, and is it written down?
3. If we were asked in six months why the system produced a particular output, could we reconstruct it?
4. Does the person signing off have the standing to disagree with the model?
If you would like a speaker who can hold that conversation with a risk or compliance audience rather than a general one, talk to us and we will shortlist against your brief.
It is output that presents non-existent information as fact — a fabricated case citation, an invented statistic, a policy that was never written. The Solicitors Regulation Authority defines it as AI "generating fictitious cases, references or seemingly factual assertions that may appear convincing despite having no basis in fact". The model does not know it is fabricating; both true and false outputs come from the same probability distribution.
No. It can be reduced substantially and measured, but not removed, because it is a property of probabilistic generation rather than a fixable bug. Even commercial legal research tools built on retrieval, marketed as hallucination-free, were found to hallucinate between 17% and 33% of the time in Stanford RegLab testing.
Retrieval-augmented generation makes the model answer from an approved document store rather than its own trained memory, and shows the source alongside the answer. It is the first line of defence and the most cost-effective, but retrieval can still surface the wrong passage, which is why it is a layer rather than a solution.
Above a defined error-cost threshold, set by the financial, legal and reputational cost of a wrong output. The threshold must be written down, and the reviewer must be competent to disagree with the model. Every published failure involves a professional who stopped checking because the output looked authoritative.
Not as of September 2026. The UK has no AI-specific statute; AI is regulated through existing frameworks according to the context of use, under the pro-innovation approach set out in 2023. The only bill before Parliament is a Private Member's Bill that has completed a first reading. In practice, regulator guidance such as the SRA's August 2026 warning notice sets the standard.
Partly. Regulation (EU) 2026/1744 moved high-risk obligations under Annex III to 2 December 2027 and Annex I to 2 August 2028. The prohibitions have applied since February 2025, general-purpose AI model obligations since August 2025, and transparency obligations under Article 50 since August 2026. UK organisations operating in the EU are already within scope of the earlier phases.
After Dinner Speakers
Contents
