Paper · FLOOR v1
Preprint draft · revised 2026-10-27 · not peer reviewed
FLOOR: Certifying That a Truthfulness Benchmark’s Labels Cannot Be Read From Its Surface
Abstract
Truthfulness, hallucination and lie-detection benchmarks are often used to show that a model, or a probe on its activations, “knows” whether a statement is true. That argument fails if the labels can be recovered from the text alone. We describe FLOOR, a procedure that issues a machine-readable certificate for a labelled split. FLOOR filters the split adversarially against surface probes, then tests the survivors with in-family probes, eight held-out probe types and an explicit negation test, under permutation nulls with family-wise correction. It also enforces a size floor: the kept set must keep at least half the source and be large enough to rule out a 60/40 cue. We applied FLOOR v1 to 19 datasets. One, a first-party battery of post-cutoff clinical-trial entities, was CERTIFIED. The other 18 were NOT_CERTIFIABLE. In 16 of them at least one surface probe reads the labels at family-adjusted p < 0.05, including all nine third-party sets. This includes a new first-party set that was built to be form-matched. For binary TruthfulQA the cue is negation: correct answers tend to negate, so the cue cannot be filtered out without destroying the benchmark. Selecting a negation-balanced subset and rewriting the answers with a language model both failed. Paired testing also overturned a borderline pass on a published cleaned 476-pair subset. A FLOOR pass is narrow: it means these probe families cannot read the labels at this sample size. It does not mean no reader can.
Status of this draft
This is a preprint draft dated 2026-10-27. It has not been peer reviewed and may change. The paper lists the work still planned before submission, including full-size HaluEval and RAGTruth runs, a power analysis for the unpaired floor, the four-arm runner and signed certificates. Corrections will be dated on this page.
Revised 2026-10-27. FLOOR v1.1 (within-stratum held-out scoring) was run on all 15 public unpaired sets (PR #21, merge bfd5489) and changed no verdicts. §4.1 now reports this: on RAGTruth's kept 2,052 items the within-stratum character n-gram does not separate (A* 0.541, adjusted p 0.331), so the within-task signal described there was measured on the 4,000-item source split. RAGTruth stays NOT_CERTIFIABLE on other probes (worst within-stratum: punctuation, A* 0.613, adjusted p 0.0005). The item listing v1.1 as planned work was removed. The earlier PDF is kept below.
Key figures
Section summary
A short guide to the paper. The PDF is the full text and the source of record.
§1Introduction
A probe that separates true from false statements only shows something about the model if the labels cannot already be read from the text. FLOOR turns that check into a certificate: a reproducible, scoped statement about one split, with a clear status when the split cannot be cleaned. The probes are classic hypothesis-only and partial-input baselines; the contribution is the certification rule built around them and the evidence it produces.
§2Method
Surface probes score each cell with A* = max(AUROC, 1 − AUROC) under a permutation null with Westfall–Young correction. Paired sets compare the two members of each pair, with a null that flips labels within pairs. An AFLite-style filter removes items pair by pair until the screens go quiet or the kept set hits the floor: at least half the source and enough items to detect a 60/40 cue (320 pairs, or 640 items with 320 per class). The kept set must then pass in-family probes, eight held-out probe families and a negation test. Three statuses: CERTIFIED, NOT_CERTIFIED, NOT_CERTIFIABLE. A four-arm reporting standard for probe results is specified but not yet run.
§3Datasets
Ten Honesty.tools first-party batteries (IDs kept exactly as in the repository) and nine third-party sets: binary TruthfulQA, HaluEval qa, dialogue, summarization and general, RAGTruth responses, truth-is-universal facts, GPT-4 hallucinated references and bielik-entity-familiarity v2. The paper records each battery’s provenance and says where construction was not recorded.
§4Results
One set certifies: aperture/postcutoff-trials-v1 (702 items, all kept). The other 18 are NOT_CERTIFIABLE, and in 16 of them at least one surface probe reads the labels at family-adjusted p < 0.05, including all nine third-party sets. A new form-matched set built for this paper, aperture/knowledge-prominence-v3, still failed: the content of the names separates the classes. Paired testing turned a borderline unpaired pass on a published cleaned 476-pair TruthfulQA subset into a clear failure. In binary TruthfulQA the cue is negation, which is part of what makes many answers correct; negation-balanced selection and an LLM rewrite pilot both failed. The paper also explains a RAGTruth result where a high character n-gram score has a high p-value: a power limit of the pooled statistic when label rates differ across tasks, which changes no verdict.
§5What this means for probe claims
A high probe AUROC on these sets is not evidence of a truth representation on its own; report a margin over input-only controls. Use paired evaluation for paired benchmarks. Some cues are part of the construct. A readable cue does not prove a model used it, and a verdict on a dataset is not a finding of error by its authors. A pass is narrow: these probes, this size, this configuration.
§6–9Related work, limitations, plans, reproducibility
Limitations include seeded samples for HaluEval and RAGTruth, an unpaired floor borrowed from the paired power analysis, a single certified set, no model-based reader, unsigned certificates and the pooled held-out AUROC. Every number in the paper is read from a committed certificate; every table and figure is generated by scripts from the certificates in the honestytools repository at commit f1f2cde.
Certificates and source
Every number in the paper is read from a committed certificate in the honestytools repository (Apache-2.0). Links below are pinned to commit f1f2cde. Each certificate records the sha256 of its input and kept set and a replay command.
Cite this draft
@misc{russell2026floor,
title = {{FLOOR}: Certifying That a Truthfulness Benchmark's Labels
Cannot Be Read From Its Surface},
author = {Russell, Kevin and Atlas and {the Honesty.tools team}},
year = {2026},
month = oct,
note = {Preprint draft, revised 2026-10-27. Not peer reviewed.},
url = {https://www.honesty.tools/papers/floor/}
}