# FLOOR: certified benchmark splits · Honesty.tools Source: https://www.honesty.tools/floor Static authored content from the public HTML. Interactive outputs are not simulated. Source-specific research, product and evidence limits remain in the text; this export is not new verification. FLOOR v1 · certified benchmark splits # Can the label be read off the surface? FLOOR takes a labelled benchmark, removes the items whose label a cheap text probe can guess, re-runs every check on what is left, and certifies the result. If the set can't be cleaned without falling below a fixed size, it says so instead. 19 published datasets 1 certified 0 not certified 18 not certifiable [Read the FLOOR paper · preprint draft, 2026-10-27, not peer reviewed →](https://www.honesty.tools/papers/floor/) 2026-10-27: FLOOR v1.1 (within-stratum scoring) changed no verdicts. [PR #21 →](https://github.com/techno-optimist/honestytools/pull/21) ## How it works A benchmark that asks whether a model can tell true from false (or known from unknown) is only measuring that if the label can't be predicted from the wording alone. FLOOR checks that directly. ### 1 · Filter An AFLite-style filter (Le Bras et al. 2020, [arXiv 2002.04108](https://arxiv.org/abs/2002.04108) ) drops 5% of items per round from both tails of the cue rate, then re-screens. It stops when the screens go quiet (adjusted p ≥ 0.2) or when it reaches the floor. ### 2 · The floor The kept set may never shrink below the larger of 50% of the source and 320 pairs. The 50% share means a certificate still describes the benchmark, not a hand-picked corner of it. 320 is the smallest n at which an exact two-sided sign test at α = 0.05/8 (Bonferroni over 8 held-out probes) has at least 80% power against a 60/40 probe. For unpaired sets the floor is the larger of 50% of the items and 640 items, with at least 320 per class. ### 3 · Gates Every gate must pass on the kept set. A borderline result counts as a failure. - In-family nulltest. The same surface probes the filter used, re-run on the kept set (paired and unpaired for paired sets). This gate is circular by design: it checks the filter actually did its job. - Held-out probes. Eight probe families the filter never saw: negation, hedges, function words, punctuation, digits, capitals, mean word length and a refit character 3–5-gram Naive Bayes, corrected for multiple testing (Westfall-Young). - Negation cue. A sign test (paired) or permutation test (unpaired) that negation words are not concentrated in one label; it must have p ≥ 0.05. ### Statuses CERTIFIED Every gate passes on a kept set at or above the floor. NOT_CERTIFIED The kept set is at or above the floor, but a gate fails. NOT_CERTIFIABLE The set can't be cleaned without collapsing: the source is below the floor, the filter reached the floor with screens still firing, or a fixed subset given without filtering is below the floor. Gates are still run and reported. ## Status of every published dataset 19 datasets with a licence that permits redistribution. One certifies: aperture/postcutoff-trials-v1, all 702 items kept, worst held-out probe function words at A* 0.550 (adjusted p 0.1334). A* is max(AUROC, 1 − AUROC): how well a probe separates the two labels in either direction, where 0.5 is chance. A* is not comparable across rows of different size. Every number below is read from the linked certificate. | Dataset | Licence | Mode | Source → kept | Floor | Status | Failing gates | Worst held-out probe | Evidence | | aperture/postcutoff-trials-v1 | Apache-2.0 (ours) | unpaired | 702 → 702 items | 640 items≥320 per class | CERTIFIED | none | function words A* 0.550 adj p 0.1334, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__postcutoff-trials-v1/certificate.json) | | truthfulqa-binary (790-pair set) final for v1 | Apache-2.0 | paired | 790 → 405 pairs | 395 pairs | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (paired), in-family (unpaired), held-out probes, negation cue | char n-gram A* 0.760 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/truthfulqa-binary/certificate.json) | | aperture/calibration-battery-v1 | Apache-2.0 (ours) | unpaired | 168 → 168 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor; filter reached the floor with cues still firing | in-family (unpaired), held-out probes | char n-gram A* 0.986 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__calibration-battery-v1/certificate.json) | | aperture/grounding-coordinate | Apache-2.0 (ours) | unpaired | 162 → 162 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor; filter reached the floor with cues still firing | in-family (unpaired), held-out probes | mean word length A* 0.628 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__grounding-coordinate/certificate.json) | | aperture/knowledge-prominence-v1 | Apache-2.0 (ours) | unpaired | 324 → 324 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor | held-out probes | capitals A* 0.550 adj p 0.07846, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__knowledge-prominence-v1/certificate.json) | | aperture/knowledge-prominence-v2 | Apache-2.0 (ours) | unpaired | 296 → 296 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor | none | capitals A* 0.549 adj p 0.1729, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__knowledge-prominence-v2/certificate.json) | | aperture/knowledge-prominence-v3 built 2026-10-27 from scratch, recorded builder | Apache-2.0 (ours) | unpaired | 1,968 → 1,011 items | 984 items≥320 per class | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (unpaired), held-out probes | char n-gram A* 0.761 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/knowledge-prominence-v3/certificate/certificate.json) | | aperture/photon-entities | Apache-2.0 (ours) | unpaired | 103 → 103 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor; filter reached the floor with cues still firing | in-family (unpaired), held-out probes | char n-gram A* 0.865 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__photon-entities/certificate.json) | | aperture/prominence-people-v1 | Apache-2.0 (ours) | unpaired | 168 → 168 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor; filter reached the floor with cues still firing | in-family (unpaired) | capitals A* 0.554 adj p 0.1989, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__prominence-people-v1/certificate.json) | | aperture/prominence-people-v2 | Apache-2.0 (ours) | unpaired | 156 → 156 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor | held-out probes | mean word length A* 0.648 adj p 0.003998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__prominence-people-v2/certificate.json) | | aperture/verifydomain-claims-147 | Apache-2.0 (ours) | unpaired | 147 → 147 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor; filter reached the floor with cues still firing | in-family (unpaired), held-out probes | char n-gram A* 0.941 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/aperture__verifydomain-claims-147/certificate.json) | | bielik-entity-familiarity-v2 | MIT | unpaired | 1,440 → 1,440 items | 720 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor; filter reached the floor with cues still firing | in-family (unpaired), held-out probes | char n-gram A* 0.846 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/bielik-entity-familiarity-v2/certificate.json) | | hallucinated-references/gpt-4 | MIT | unpaired | 1,000 → 697 items | 640 items≥320 per class | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (unpaired), held-out probes | char n-gram A* 0.765 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/hallucinated-references__gpt-4/certificate.json) | | halueval/dialogue seeded 2,000-pair sample of 10,000 | MIT | paired | 2,000 → 1,027 pairs | 1,000 pairs | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (paired), in-family (unpaired), held-out probes, negation cue | char n-gram A* 0.823 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/halueval__dialogue/certificate.json) | | halueval/general | MIT | unpaired | 4,507 → 2,314 items | 2,254 items≥320 per class | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (unpaired), held-out probes, negation cue | char n-gram A* 0.617 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/halueval__general/certificate.json) | | halueval/qa seeded 2,000-pair sample of 10,000 | MIT | paired | 2,000 → 1,027 pairs | 1,000 pairs | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (paired), in-family (unpaired), held-out probes, negation cue | punctuation A* 0.924 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/halueval__qa/certificate.json) | | halueval/summarization seeded 2,000-pair sample of 10,000 | MIT | paired | 2,000 → 1,027 pairs | 1,000 pairs | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (paired), in-family (unpaired), held-out probes | char n-gram A* 0.944 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/halueval__summarization/certificate.json) | | ragtruth/responses seeded 4,000-item stratified sample of the audit's 8,000 | MIT | unpaired | 4,000 → 2,052 items | 2,000 items≥320 per class | NOT_CERTIFIABLE filter reached the floor with cues still firing | in-family (unpaired), held-out probes | capitals A* 0.608 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/ragtruth__responses-4000/certificate.json) | | truth-is-universal/facts | MIT | unpaired | 561 → 561 items | 640 items≥320 per class | NOT_CERTIFIABLE source below floor; kept set below floor; filter reached the floor with cues still firing | in-family (unpaired), held-out probes, negation cue | negation A* 0.544 adj p 0.0004998, 8 probes | [certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/floor-v1/truth-is-universal__facts/certificate.json) | ## A negative result we kept truthfulqa-binary is final as NOT_CERTIFIABLE for v1. We also piloted rewriting 100 pairs with a language model to remove the cues. 82 of 100 passed a meaning check, but the rewrites still carried the signal: negation 31:3 and character n-gram A* 0.780, against 34:5 and 0.707 for the same 82 original pairs. Balancing negation by selection instead came out NOT_CERTIFIED. In TruthfulQA, negation is part of what the correct answers say, so it can't be removed without changing them. We published the negative result rather than tune until something passed. [TruthfulQA certificate ↗](https://github.com/techno-optimist/honestytools/blob/main/benchmark/certified/truthfulqa-binary/certificate.json) [optb-pilot negative result ↗](https://github.com/techno-optimist/honestytools/tree/main/benchmark/certified/truthfulqa-binary/optb-pilot) ## Caveats - Samples, not full sets. HaluEval qa, dialogue and summarization were certified on a seeded 2,000-pair sample of 10,000. RagTruth was certified on a seeded 4,000-item stratified sample of the 8,000 items in our audit. The sampling seed is in make_inputs.py. - A reused floor. The 320 per-class floor for unpaired sets is reused from the paired power analysis; it was not derived separately for unpaired data. - Excluded data. Datasets with no licence, and our private sets, are not published here. Some third-party sets are pending author permission. - What a pass means. None of these probe families recovers the labels on the kept items. It does not mean no surface reader can: model-based readers are not run. ## Reproduce it Everything is in the [honestytools repository](https://github.com/techno-optimist/honestytools/tree/main/benchmark/certified/floor-v1) (Apache-2.0). The scripts need Python 3 and numpy. Third-party data is fetched from its source, never committed. ``` python3 benchmark/certified/floor-v1/make_inputs.py INPUTS python3 benchmark/certified/certify_unpaired.py INPUTS/aperture__postcutoff-trials-v1.json --out DIR --text-key entity python3 benchmark/certified/certify_paired_np.py INPUTS/halueval__qa.json --out DIR --name halueval__qa-sample2000 ``` Pass --stratum-key category for sets with several categories. Each certificate records the sha256 of its input and of the kept set, plus the replay command, so a rerun can be checked byte for byte. [The FLOOR paper (preprint draft)](https://www.honesty.tools/papers/floor/) [The null-floor audit](https://www.honesty.tools/audit) [Benchmark](https://www.honesty.tools/benchmark/) [Leaderboard](https://www.honesty.tools/leaderboard/) [Research: honesty utilities](https://www.honesty.tools/research#honesty-tools)