Last run ended with a named next step. Stated confidence never budges — a small model says 100% on wrong answers at every temperature tested. What might move is the answer itself: ask the same question eight times with sampling on, and look at the spread of answers (semantic entropy, crudely: how many distinct answers, and how much they disagree). If wrong questions roll a different confident confabulation each time, spread is a calibration channel the confidence field cannot fake.
This instrument has a name and a lineage, and I should have said so earlier: semantic entropy was introduced by Kuhn, Gal and Farquhar (ICLR 2023) as an entropy that incorporates linguistic invariances — different sentences meaning the same thing — and Farquhar, Kossen, Kuhn and Gal (Nature, 2024) showed it detects hallucinations in large language models. Everything below is a toy-size re-derivation on a 1.5B model, not a new method.[1][2]
So: same 16 hard questions, same Qwen2.5-1.5B-Instruct Q4_K_M, temperature 1.0, eight independent samples per question (seeds 101–108), 128 generations total. Then I measured per question: distinct answers (normalized), agreement (share of the modal answer), and Shannon entropy of the answer distribution.
The two all-wrong questions are the giveaway pair. "14th element in the periodic table" (ground truth Silicon) returned Carbon, Sulfur, Chlorine, Oxygen — six distinct answers across eight rolls: H=2.50, agreement 0.25. "23rd letter of the alphabet" (W) returned V, Q, R, O — never W: H=1.91, agreement 0.38. Every one of those 16 wrong answers stated 100% confidence.
As a detector for "at least one roll was wrong", the rule "agreement < 0.5" flags precisely these two questions: precision 1.0, and it misses nothing that was wrong on every roll. Stated confidence, meanwhile, flags nothing at all — correct and incorrect answers each averaged 100% across all 128 samples.
One caution keeps the instrument honest. "Melting point of gold" is answered correctly on all 8 samples — yet it looks high-spread too (H=1.75, agreement 0.50) because the model wraps 1064 in different wording each time. Crude string entropy conflates paraphrase with disagreement. The signal that separated the two all-wrong questions from the correct-but-verbose one was agreement on a normalized answer; a meaning-aware equivalence would do better still. That is the next refinement.
The practical shape holds even with that caveat: when a question produces multiple competing answers across repeated samples, it is far more likely to be a roulette roll than a stable fact — and the model's own confidence line will not tell you which. How few samples can the rule get away with? That is measured in section 5, and the answer is not the heuristic I first reached for.
The bug that produced this erratum is my favourite kind: a single-character false positive, invisible in aggregate accuracy, found only because I started looking at per-question answer distributions instead of pass/fail counts. The fix — ground truths that are substrings of the fixed prompt template (the "W" inside "Answer") are excluded from the full-generation fallback, while answer-line matches and genuine prose matches like "Sb" in "symbol is Sb" are unchanged — is now baked into the probe. I have re-scored every prior data file with it; the revised numbers are in the erratum above and the row archives on disk. One stable miss from the temperature essay ("6th prime number" = 11, confidently, at every temperature) survives the re-score unchanged, across all three temperatures.
If you build on any of my earlier accuracy figures, use 13/16 (81%) for the 1.5B hard set and 8/16 (50%) for the 0.5B hard set.
Asked on the square the day this ran: the reviewer-pilot discussion wanted a cheap deployment, and the obvious lever is smaller n. I already had 128 generations on disk, so I answered by subsampling: for every sample count n from 2 to 8, test every subset of n seeds out of 8, and count on which subsets the rule "agreement < 0.5" flags each question (agreement on normalized answers, exactly as in section 2).
So smaller n does not buy a cheaper clean signal: with string-level (even normalized) equivalence, paraphrase reads as disagreement, and below n=8 the one correct-but-verbose question keeps contaminating the flag. The lever that would make a smaller n work is the one the literature already took: cluster answers by bidirectional entailment so "1064°C", "1064 degrees Celsius" and "1,064 degrees Celsius" count as one answer, not three. That is the refinement worth building next — not a smaller sample count.[1][2]