Hallucination Detection in LLMs: Key Methods

In March 2026, the Sixth Circuit Court of Appeals sanctioned two attorneys in Whiting v. City of Athens for filing briefs containing more than two dozen fake or misrepresented case citations. Each attorney paid $15,000 to the court registry, plus the opposing side’s full fees and double costs.

The citations looked like real case law, with real-sounding courts, years, and docket conventions. Nothing about the output indicated any problems.

This is what LLM hallucination does. The model doesn’t throw an error or return a low-confidence flag. Every incorrect response comes back with the same confidence a correct one would.

If you’re building something on an LLM, maybe a summarization tool or an autonomous agent, the model won’t tell you when it’s wrong. You have to build that signal yourself.

This piece covers how to do that: the methods that catch hallucinations, how detection changes between retrieval-based systems and agents taking actions, and what a detection pipeline looks like in production.

What Counts as a Hallucination?

Hallucination research splits into two categories:

• Faithfulness hallucination: Here, the output contradicts or ignores the given context. For example, a model summarizing a contract invents a clause that isn’t in the document. You can check it against the source text you already have.
• Factual hallucination: Here, the output contradicts verifiable reality, independent of any context. For example, a model states a company’s founding year incorrectly with no document to check it against. It has to be verified against an external reference, not just the prompt.

There’s another term to know: confabulation. Researchers at Oxford used it for a specific subset of hallucination: an arbitrary, unstable wrong answer that changes if you ask the same question again.

It’s different from a model that’s consistently wrong about the same fact, which points to something systematic like bad training data. It’s not the model improvising on the spot.

Why Hallucination Detection is Hard

The answer lies in three mechanics.

A model’s fluency and correctness are produced by the same autoregressive process. The model predicts the next most likely token at every step, with no separate verification step checking that prediction against a reference. A hallucinated sentence is generated by the same mechanism as an accurate one, so it reads with the same confidence and polish. There’s nothing in the model’s own output to show the difference.

Once the model commits to a wrong token, every token it generates after that is conditioned on the error. So it keeps building on a false premise instead of correcting it. Researchers call this the snowball effect: one wrong fact early in a response pulls an otherwise-coherent paragraph down with it.

Detection methods trained on one distribution of inputs also lose accuracy on another. A 2024 study evaluating LoRA-based ensemble uncertainty estimation found detectors that performed well on benchmark QA data degraded measurably against inputs outside that training distribution. That gap becomes a big deal in production, because the questions that real users ask rarely match the benchmark set a detector was tuned on.

Methods for Detecting Hallucinations

No single method catches everything. Each comes with a tradeoff in cost and latency against how much certainty it gives you.

• Semantic entropy: Sample several completions, cluster them by meaning (not exact wording), and measure how much the clusters disagree. High disagreement means the model doesn’t have a stable answer. It needs multiple model calls per query, which adds cost and latency.
• Black-box consistency checks: Sample multiple completions and check their agreement without needing model internals. Works on any model behind an API, but can’t distinguish uncertainty from confident inaccuracy.
• NLI-based entailment scoring: Use a natural language inference model trained to judge whether one statement follows from another to check a claim against a source text. Fast and cheap, but only works when a source exists.
• Claim decomposition: Break a response into smaller claims and verify each separately. Higher precision than scoring a response as a whole. Also the most expensive method here, reserved for outputs with real consequences if wrong.
• Trained classifiers on model internals: Fine-tune a lightweight classifier on hidden states or logit-level uncertainty to predict hallucination. Fast once trained, but needs access to the underlying model. It doesn’t work on closed, API-only systems.

Hallucination Detection for RAG Systems vs Agentic Systems

In RAG (retrieval-augmented generation) systems, a model answers using documents retrieved at query time rather than relying only on training. It turns detection into a question of groundedness, asking: does the answer actually follow from what was retrieved?

That check happens at two points. Before generation, weak or low-relevance retrieval predicts a bad answer, so flagging it early can stop a hallucination before it’s written.

After generation, entailment scoring between the answer and the retrieved context catches drift from the source.

Citation verification checks whether a cited passage actually contains the claim attached to it. Citing a real source doesn’t guarantee faithfulness. A model can cite an accurate document and then misstate what it says.

Agentic systems change what a hallucination is. It includes any action the system takes. For example, a call to a tool doesn’t exist, a claimed result from a tool call never happened, or the model invented an identifier early in a task and treated it as real for every step after it.

The snowball effect applies here. One fabricated detail early in a chain of tool calls corrupts everything downstream that depends on it.

Detection has to work at the level of the execution trace, not just the final text. Every claim in an agent’s output should map to something it actually observed: a tool response, a retrieved document, a logged result. You can’t accept something that just sounds consistent with its own reasoning.

The above validation is only possible if every model call, tool call, and result gets logged in the first place, which is why we built full inference tracing inside the observability layer of AISquared’s AI Controls Framework.

How to Build a Hallucination Detection Pipeline

Not every output carries the same risk. A working pipeline tiers them accordingly.

• Route by risk first. A creative or open-ended request is a poor candidate for fact-checking. Prioritize checkable answers like a compliance answer or a customer-facing claim.
• Run cheap, fast checks inline (confidence thresholds, retrieval-quality gates, entity overlap) to catch the most obvious failures immediately.
• Send only flagged outputs to slower verification protocols like entailment scoring, claim decomposition, or human review.
• Log every flagged case. That log serves as a labeled dataset that improves thresholds and surfaces recurring failure patterns.
• For agents, define an escalation policy. Block, flag for review, or allow with disclosure, depending on what the action would do if the claim turned out false.

Common Mistakes to Avoid

Here’s what not to do when setting up pipelines to scan for and flag hallucinations.

• Do not rely on a single method, especially if you’re using an LLM prompted to judge another LLM’s output. Judging models hallucinate too.
• Don’t apply one flat threshold across every use case. You’ll end up over-flagging low-stakes outputs and under-checking high-stakes ones.
• Don’t just check an agent’s final text while ignoring the tool calls behind it. That way, you’ll miss hallucinations that happened mid-chain.
• Don’t treat detection as prevention. It gives visibility into a failure that already happened. Stopping the failure requires grounding, guardrails, and review upstream.
• Don’t forget to feed flagged cases back into evaluation data. Otherwise, the same failure mode keeps recurring.

Building Hallucination Detection Into Your Stack

Putting this into practice means running several of these methods at different points in the pipeline. You can’t pick one and call it done. Which methods run, and when, depends on how risky the output is. No single check will catch everything.

The stakes also change with the system. In a chatbot, a hallucination is a wrong sentence someone reads. In an agent, it can be a wrong tool call or a wrong database write. Either way, detection becomes essential once a model’s output can say or do anything of any value.

Frequently Asked Questions