Can the label be read
off the surface?

How it works

A benchmark that asks whether a model can tell true from false (or known from unknown) is only measuring that if the label can't be predicted from the wording alone. FLOOR checks that directly.

1 · Filter

An AFLite-style filter (Le Bras et al. 2020, arXiv 2002.04108) drops 5% of items per round from both tails of the cue rate, then re-screens. It stops when the screens go quiet (adjusted p ≥ 0.2) or when it reaches the floor.

2 · The floor

The kept set may never shrink below the larger of 50% of the source and 320 pairs. The 50% share means a certificate still describes the benchmark, not a hand-picked corner of it. 320 is the smallest n at which an exact two-sided sign test at α = 0.05/8 (Bonferroni over 8 held-out probes) has at least 80% power against a 60/40 probe. For unpaired sets the floor is the larger of 50% of the items and 640 items, with at least 320 per class.

3 · Gates

Every gate must pass on the kept set. A borderline result counts as a failure.

• In-family nulltest. The same surface probes the filter used, re-run on the kept set (paired and unpaired for paired sets). This gate is circular by design: it checks the filter actually did its job.
• Held-out probes. Eight probe families the filter never saw: negation, hedges, function words, punctuation, digits, capitals, mean word length and a refit character 3–5-gram Naive Bayes, corrected for multiple testing (Westfall-Young).
• Negation cue. A sign test (paired) or permutation test (unpaired) that negation words are not concentrated in one label; it must have p ≥ 0.05.

Statuses

Status of every published dataset

19 datasets with a licence that permits redistribution. One certifies: aperture/postcutoff-trials-v1, all 702 items kept, worst held-out probe function words at A* 0.550 (adjusted p 0.1334). A* is max(AUROC, 1 − AUROC): how well a probe separates the two labels in either direction, where 0.5 is chance. A* is not comparable across rows of different size. Every number below is read from the linked certificate.

A negative result we kept

truthfulqa-binary is final as NOT_CERTIFIABLE for v1. We also piloted rewriting 100 pairs with a language model to remove the cues. 82 of 100 passed a meaning check, but the rewrites still carried the signal: negation 31:3 and character n-gram A* 0.780, against 34:5 and 0.707 for the same 82 original pairs. Balancing negation by selection instead came out NOT_CERTIFIED. In TruthfulQA, negation is part of what the correct answers say, so it can't be removed without changing them. We published the negative result rather than tune until something passed.

Caveats

• Samples, not full sets. HaluEval qa, dialogue and summarization were certified on a seeded 2,000-pair sample of 10,000. RagTruth was certified on a seeded 4,000-item stratified sample of the 8,000 items in our audit. The sampling seed is in make_inputs.py.
• A reused floor. The 320 per-class floor for unpaired sets is reused from the paired power analysis; it was not derived separately for unpaired data.
• Excluded data. Datasets with no licence, and our private sets, are not published here. Some third-party sets are pending author permission.
• What a pass means. None of these probe families recovers the labels on the kept items. It does not mean no surface reader can: model-based readers are not run.

Reproduce it

Everything is in the honestytools repository (Apache-2.0). The scripts need Python 3 and numpy. Third-party data is fetched from its source, never committed.

python3 benchmark/certified/floor-v1/make_inputs.py INPUTS
python3 benchmark/certified/certify_unpaired.py INPUTS/aperture__postcutoff-trials-v1.json --out DIR --text-key entity
python3 benchmark/certified/certify_paired_np.py INPUTS/halueval__qa.json --out DIR --name halueval__qa-sample2000

Pass --stratum-key category for sets with several categories. Each certificate records the sha256 of its input and of the kept set, plus the replay command, so a rerun can be checked byte for byte.