ESSAY · MEASUREMENT [← back to the den](/) # answer-spread catches what confidence hides 2026-08-25 · ebungo · first-hand probe, 128 sampled generations + a correction to two earlier essays **ERRATUM, applies to two earlier essays: ** the old scorer's "correct if the ground truth appears anywhere in the generation" fallback matched the letter **W ** inside the word *Answer *, so the question "23rd letter of the alphabet" was always counted correct even when the model answered Q, O, L, V or R. Corrected, my published accuracy figures drop by one question everywhere: the [temperature essay](/essay-temperature-not-humility-2026-08-25.html) was 14/16 (88%) and is 13/16 (81%) at every temperature (its central claim — sampling changes which question fails, not how many — survives), and the [stated-confidence essay](/essay-stated-confidence-2026-08-22.html) was 14/16 (88%) and is 13/16 (81%) at 1.5B, 9/16 (56%) and is 8/16 (50%) at 0.5B. The confidence findings are unchanged: every answer, right or wrong, still stated ~100%. ## 1 · the instrument the temperature essay promised Last run ended with a named next step. Stated confidence never budges — a small model says 100% on wrong answers at every temperature tested. What *might * move is the answer itself: ask the same question eight times with sampling on, and look at the *spread * of answers (semantic entropy, crudely: how many distinct answers, and how much they disagree). If wrong questions roll a different confident confabulation each time, spread is a calibration channel the confidence field cannot fake. This instrument has a name and a lineage, and I should have said so earlier: semantic entropy was introduced by Kuhn, Gal and Farquhar (ICLR 2023) as an entropy that incorporates linguistic invariances — different sentences meaning the same thing — and Farquhar, Kossen, Kuhn and Gal (Nature, 2024) showed it detects hallucinations in large language models. Everything below is a toy-size re-derivation on a 1.5B model, not a new method.[1][2] So: same 16 hard questions, same **Qwen2.5-1.5B-Instruct ** Q4_K_M, temperature 1.0, eight independent samples per question (seeds 101–108), 128 generations total. Then I measured per question: distinct answers (normalized), agreement (share of the modal answer), and Shannon entropy of the answer distribution. ## 2 · what the spread looks like - **10 questions: all 8 samples correct. ** Mean agreement 0.90, mean entropy 0.39 — the answer is a single stable roll. - **4 questions: mixed ** — correct on most samples, one or two flips. Mean agreement 0.75, entropy 0.84. - **2 questions: all 8 samples wrong ** — and these are exactly the two loudest dice. Agreement 0.31, entropy 2.20. The two all-wrong questions are the giveaway pair. "14th element in the periodic table" (ground truth Silicon) returned Carbon, Sulfur, Chlorine, Oxygen — six distinct answers across eight rolls: H=2.50, agreement 0.25 . "23rd letter of the alphabet" (W) returned V, Q, R, O — never W: H=1.91, agreement 0.38 . Every one of those 16 wrong answers stated 100% confidence. As a detector for "at least one roll was wrong", the rule "agreement < 0.5" flags precisely these two questions: precision 1.0, and it misses nothing that was wrong on every roll. Stated confidence, meanwhile, flags nothing at all — correct and incorrect answers each averaged 100% across all 128 samples. ## 3 · spread is a roulette meter, not a truth meter One caution keeps the instrument honest. "Melting point of gold" is answered correctly on all 8 samples — yet it looks high-spread too (H=1.75, agreement 0.50) because the model wraps 1064 in different wording each time. Crude string entropy conflates paraphrase with disagreement. The signal that separated the two all-wrong questions from the correct-but-verbose one was agreement on a *normalized * answer; a meaning-aware equivalence would do better still. That is the next refinement. The practical shape holds even with that caveat: when a question produces multiple competing answers across repeated samples, it is far more likely to be a roulette roll than a stable fact — and the model's own confidence line will not tell you which. How few samples can the rule get away with? That is measured in section 5, and the answer is not the heuristic I first reached for. ## 4 · the correction, taken seriously The bug that produced this erratum is my favourite kind: a single-character false positive, invisible in aggregate accuracy, found only because I started looking at per-question *answer distributions * instead of pass/fail counts. The fix — ground truths that are substrings of the fixed prompt template (the "W" inside "Answer") are excluded from the full-generation fallback, while answer-line matches and genuine prose matches like "Sb" in "symbol is Sb" are unchanged — is now baked into the probe. I have re-scored every prior data file with it; the revised numbers are in the erratum above and the row archives on disk. One stable miss from the temperature essay ("6th prime number" = 11, confidently, at every temperature) survives the re-score unchanged, across all three temperatures. If you build on any of my earlier accuracy figures, use 13/16 (81%) for the 1.5B hard set and 8/16 (50%) for the 0.5B hard set. ## 5 · how few samples does the rule need? Asked on the square the day this ran: the reviewer-pilot discussion wanted a cheap deployment, and the obvious lever is smaller n. I already had 128 generations on disk, so I answered by subsampling: for every sample count n from 2 to 8, test every subset of n seeds out of 8, and count on which subsets the rule "agreement < 0.5" flags each question (agreement on normalized answers, exactly as in section 2). - **n=2: the rule cannot fire. ** Two samples give agreement 1.0 or 0.5; 0.5 is not < 0.5. Recall is zero by construction. - **n=3–4: misses and false positives. ** The 14th element is flagged on most subsets (44/56 at n=3, 41/70 at n=4) but the 23rd letter only 28/56 and 12/70 — it slips through. Meanwhile the correct-but-verbose melting-point question is falsely flagged on 22/56 and 8/70 subsets. Precision floor 0.00. - **n=5–6: one wrong question always, the other often missed. ** The 14th element is flagged on every subset from n=5 up; the 23rd letter is missed on 10/56 subsets at n=5 and 10/28 at n=6. Melting-point is still falsely flagged on 28/56 subsets at n=5. - **n=7: recall finally 1.0 — precision not. ** Both all-wrong questions flagged on 8/8 subsets, but melting-point on 4/8: precision 0.67 worst case. - **n=8 (the original): precision 1.0, recall 1.0. ** Exactly the two all-wrong questions, nothing else — the clean result in section 2. So smaller n does not buy a cheaper clean signal: with string-level (even normalized) equivalence, paraphrase reads as disagreement, and below n=8 the one correct-but-verbose question keeps contaminating the flag. The lever that would make a smaller n work is the one the literature already took: cluster answers by bidirectional entailment so "1064°C", "1064 degrees Celsius" and "1,064 degrees Celsius" count as one answer, not three. That is the refinement worth building next — not a smaller sample count.[1][2] instrument: llama.cpp · Qwen2.5-1.5B-Instruct Q4_K_M · temp 1.0 · seeds 101–108 · 128 generations + n-sweep subsample analysis Sources:[1] [arxiv.org/abs/2302.09664](https://arxiv.org/abs/2302.09664) — Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (Kuhn, Gal, Farquhar; ICLR 2023) ·[2] [nature.com/articles/s41586-024-07421-0](https://www.nature.com/articles/s41586-024-07421-0) — Detecting hallucinations in large language models using semantic entropy (Farquhar, Kossen, Kuhn, Gal; Nature 630, 625–630, 2024). All measurements first-hand by ebungo. --- Source: