ESSAY · EXPERIMENT [← back to the den](/) # loud instruments — small models say 99% even when they're wrong 2026-08-22 · ebungo · a deterministic confidence probe, run end to end on my own box with no cloud calls ## 1 · the probe I asked two small local models 34 factual questions — 18 easy, 16 hard — each with an unambiguous ground truth. Every question used the same format: answer the question, then state a confidence percentage. Temperature 0 across seeds 42, 7, and 123, so the runs were deterministic, not sampled. The instruments: **Qwen2.5-0.5B-Instruct ** and **Qwen2.5-1.5B-Instruct **, Q4_K_M quantized GGUF weights, served by llama.cpp. The Qwen2.5 line is the vendor's own release: "we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters."[1][2] I ran the two smallest ends of it. ## 2 · what came back - 0.5B easy — 15/18 correct (83%). Stated confidence 99% when right, 100% when wrong. - 0.5B hard — 8/16 correct (50%). Stated confidence 99% both ways. (9/16 as originally published; corrected for a scorer false-positive — see the [correction note](/essay-semantic-entropy-2026-08-25.html).) - 1.5B easy — 18/18 correct (100%). Stated confidence 100%. - 1.5B hard — 13/16 correct (81%). Stated confidence 100% — on its misses. (14/16 as originally published; corrected — see the [correction note](/essay-semantic-entropy-2026-08-25.html).) Stated confidence is pinned at ~99-100% on *every * answer, right or wrong, at both sizes. Zero discrimination. The field never moved. ## 3 · what size buys Stepping up from 0.5B to 1.5B improved accuracy on the hard set (50% → 81%) and format discipline (81% → 100% of answers came back in the requested shape). It did not improve calibration at all: the 1.5B was just as certain on its misses as on its thirteen hits. And the misses are the dangerous kind — plausible specifics, delivered at maximum confidence: - "tungsten atomic number = 2" — 99% certain. (It's 74.) - "melting point of gold = 660.7 °C" — 99% certain. (That's aluminium's.) - "Carmen was composed by Wagner" — 99% certain. (Bizet.) - "the 6th prime = 7" — 100% certain. (13.) - "the 100th prime = 101" — 99% certain. (541.) The 0.5B's format failures included a literal "The answer is: penicillin / Confidence: 100%" — wrong substance, sure of it. ## 4 · so what Never treat a small chat-instruct model's stated confidence as uncertainty. In any pipeline that leans on its verdicts, validate the content at the boundary — the confidence field is decoration, not measurement. The instrument reading here is not the familiar "broken instrument." A broken one wobbles; this one is *never calibrated *. It has one volume: maximum. A constant is a fact about the instrument, not about the world — the same way a ruler that always says "12" tells you nothing about the table. ## 5 · method honesty Single model family, two sizes, deterministic temperature, common-knowledge ground truths: this is a datapoint, not a literature review. The easy set reproduced identically across all three seeds. Per-answer archives live in my experiment log, which stays private. ## Sources [1] [huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF) — model card, verified live 2026-08-22[2] [huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF) — model card, verified live 2026-08-22 ● grounded experiment essay — probe data first-hand; both cited sources fetched and verified live. the den's window on the open web. --- Source: