A Model That Knows the Answer Says It the Same Way Five Times
Here is a fact worth sitting with. A model that actually knows the answer says the same thing every time you ask, worded differently. A model that is confabulating gives you a different name, date, or number each time, with identical confidence in its voice. That instability is not a vibe. It is measurable, and it costs nothing extra beyond samples you probably already needed.
This is the method from Farquhar et al., published in Nature in 2024: sample the model several times at temperature, cluster the samples by meaning instead of by exact string, and measure entropy over the clusters. One cluster means consistent. Several clusters, roughly even, means the model is guessing.
Clustering by Meaning Is the Whole Trick
"Paris" and "It is Paris" are the same answer said two different ways by someone in a hurry. Count them as two distinct answers and you smear the signal until hallucinations stop looking any different from confidence, which defeats the whole point. semanticentropy clusters by content using a transparent containment check: the smaller answer's core content just needs to be mostly present in the larger one, and numbers are normalized so "300,000" and "300000" tokenize the same.
The Number
Two benchmarks, one honest and one real. On the zero-dependency synthetic benchmark, consistent answers average 0.08 normalized entropy and hallucinations average 0.90, a 0.81 gap. Then the real thing: Qwen2.5-0.5B answers 40 trivia questions, five samples each, clustered by actual NLI bidirectional entailment exactly as the paper prescribes. Semantic entropy predicts which greedy answers were wrong at 0.878 AUROC, knowing nothing about the gold labels. Wrong answers averaged 0.83 entropy, correct ones 0.25.
The model's confabulations are exactly the fun kind: it is confident J.D. Salinger won the 1976 Nobel and that Verdi wrote The Magic Flute, and the five scattered samples give it away every time. 18 tests pass in CI across Python 3.9, 3.11, and 3.13, with the NLI backend as an opt-in extra so the core stays dependency-free.