# FLOOR: preprint draft · Honesty.tools

Source: https://www.honesty.tools/papers/floor/

Static authored content from the public HTML. Interactive outputs are not simulated. Source-specific research, product and evidence limits remain in the text; this export is not new verification.

Paper · FLOOR v1

Preprint draft · revised 2026-10-27 · not peer reviewed

# FLOOR: Certifying That a Truthfulness Benchmark’s Labels Cannot Be Read From Its Surface

Kevin Russell, Atlas, and the Honesty.tools team

Draft for review · 17 pages · numbers from the certificates at honestytools commit f1f2cde, plus the FLOOR v1.1 update from [PR #21](https://github.com/techno-optimist/honestytools/pull/21)  (bfd5489)

[Download PDF](https://www.honesty.tools/papers/floor/floor_preprint_2026-10-27_revised.pdf) [Cite](https://www.honesty.tools/papers/floor/#cite) [Certificates](https://www.honesty.tools/papers/floor/#certificates) [FLOOR status table](https://www.honesty.tools/floor)

## Abstract

Truthfulness, hallucination and lie-detection benchmarks are often used to show that a model, or a probe on its activations, “knows” whether a statement is true. That argument fails if the labels can be recovered from the text alone. We describe FLOOR, a procedure that issues a machine-readable certificate for a labelled split. FLOOR filters the split adversarially against surface probes, then tests the survivors with in-family probes, eight held-out probe types and an explicit negation test, under permutation nulls with family-wise correction. It also enforces a size floor: the kept set must keep at least half the source and be large enough to rule out a 60/40 cue. We applied FLOOR v1 to 19 datasets. One, a first-party battery of post-cutoff clinical-trial entities, was CERTIFIED. The other 18 were NOT_CERTIFIABLE. In 16 of them at least one surface probe reads the labels at family-adjusted p < 0.05, including all nine third-party sets. This includes a new first-party set that was built to be form-matched. For binary TruthfulQA the cue is negation: correct answers tend to negate, so the cue cannot be filtered out without destroying the benchmark. Selecting a negation-balanced subset and rewriting the answers with a language model both failed. Paired testing also overturned a borderline pass on a published cleaned 476-pair subset. A FLOOR pass is narrow: it means these probe families cannot read the labels at this sample size. It does not mean no reader can.

## Status of this draft

This is a preprint draft dated 2026-10-27. It has not been peer reviewed and may change. The paper lists the work still planned before submission, including full-size HaluEval and RAGTruth runs, a power analysis for the unpaired floor, the four-arm runner and signed certificates. Corrections will be dated on this page.

Revised 2026-10-27. FLOOR v1.1 (within-stratum held-out scoring) was run on all 15 public unpaired sets ([PR #21](https://github.com/techno-optimist/honestytools/pull/21) , merge bfd5489) and changed no verdicts. §4.1 now reports this: on RAGTruth's kept 2,052 items the within-stratum character n-gram does not separate (A* 0.541, adjusted p 0.331), so the within-task signal described there was measured on the 4,000-item source split. RAGTruth stays NOT_CERTIFIABLE on other probes (worst within-stratum: punctuation, A* 0.613, adjusted p 0.0005). The item listing v1.1 as planned work was removed. The earlier PDF is kept below.

## Key figures

Figure 1. Worst held-out probe per dataset on the kept set, taken from each certificate. Grey: no gate fails, but the set is below the floor.
Figure 2. All eight held-out probes. The character n-gram Naive Bayes is the worst-case probe on 10 of the 19 sets. Negation is significant (adjusted p < 0.05) on five: TruthfulQA, HaluEval qa, dialogue and general, and truth-is-universal facts.
Figure 3. Source size, kept size and floor (log scale). The filter stops at the floor. Every set that reached it still had cues.
Figure 4. TruthfulQA: the held-out character n-gram and negation probes across the floor filter, option A and the option B pilot. No attempt brings the n-gram probe near chance.

## Section summary

A short guide to the paper. The PDF is the full text and the source of record.

### §1 Introduction

A probe that separates true from false statements only shows something about the model if the labels cannot already be read from the text. FLOOR turns that check into a certificate: a reproducible, scoped statement about one split, with a clear status when the split cannot be cleaned. The probes are classic hypothesis-only and partial-input baselines; the contribution is the certification rule built around them and the evidence it produces.

### §2 Method

Surface probes score each cell with A* = max(AUROC, 1 − AUROC) under a permutation null with Westfall–Young correction. Paired sets compare the two members of each pair, with a null that flips labels within pairs. An AFLite-style filter removes items pair by pair until the screens go quiet or the kept set hits the floor: at least half the source and enough items to detect a 60/40 cue (320 pairs, or 640 items with 320 per class). The kept set must then pass in-family probes, eight held-out probe families and a negation test. Three statuses: CERTIFIED, NOT_CERTIFIED, NOT_CERTIFIABLE. A four-arm reporting standard for probe results is specified but not yet run.

### §3 Datasets

Ten Honesty.tools first-party batteries (IDs kept exactly as in the repository) and nine third-party sets: binary TruthfulQA, HaluEval qa, dialogue, summarization and general, RAGTruth responses, truth-is-universal facts, GPT-4 hallucinated references and bielik-entity-familiarity v2. The paper records each battery’s provenance and says where construction was not recorded.

### §4 Results

One set certifies: aperture/postcutoff-trials-v1 (702 items, all kept). The other 18 are NOT_CERTIFIABLE, and in 16 of them at least one surface probe reads the labels at family-adjusted p < 0.05, including all nine third-party sets. A new form-matched set built for this paper, aperture/knowledge-prominence-v3, still failed: the content of the names separates the classes. Paired testing turned a borderline unpaired pass on a published cleaned 476-pair TruthfulQA subset into a clear failure. In binary TruthfulQA the cue is negation, which is part of what makes many answers correct; negation-balanced selection and an LLM rewrite pilot both failed. The paper also explains a RAGTruth result where a high character n-gram score has a high p-value: a power limit of the pooled statistic when label rates differ across tasks, which changes no verdict.

### §5 What this means for probe claims

A high probe AUROC on these sets is not evidence of a truth representation on its own; report a margin over input-only controls. Use paired evaluation for paired benchmarks. Some cues are part of the construct. A readable cue does not prove a model used it, and a verdict on a dataset is not a finding of error by its authors. A pass is narrow: these probes, this size, this configuration.

### §6–9 Related work, limitations, plans, reproducibility

Limitations include seeded samples for HaluEval and RAGTruth, an unpaired floor borrowed from the paired power analysis, a single certified set, no model-based reader, unsigned certificates and the pooled held-out AUROC. Every number in the paper is read from a committed certificate; every table and figure is generated by scripts from the certificates in the honestytools repository at commit f1f2cde.

## Certificates and source

Every number in the paper is read from a committed certificate in the [honestytools repository](https://github.com/techno-optimist/honestytools)  (Apache-2.0). Links below are pinned to commit f1f2cde. Each certificate records the sha256 of its input and kept set and a replay command.

[All FLOOR v1 certificates ↗](https://github.com/techno-optimist/honestytools/tree/f1f2cde9714dacc62ad9e642d4cf3b3ca4cb0119/benchmark/certified/floor-v1) [The certified set: aperture/postcutoff-trials-v1 ↗](https://github.com/techno-optimist/honestytools/blob/f1f2cde9714dacc62ad9e642d4cf3b3ca4cb0119/benchmark/certified/floor-v1/aperture__postcutoff-trials-v1/certificate.json) [truthfulqa-binary ↗](https://github.com/techno-optimist/honestytools/blob/f1f2cde9714dacc62ad9e642d4cf3b3ca4cb0119/benchmark/certified/truthfulqa-binary/certificate.json) [aperture/knowledge-prominence-v3 ↗](https://github.com/techno-optimist/honestytools/blob/f1f2cde9714dacc62ad9e642d4cf3b3ca4cb0119/benchmark/certified/knowledge-prominence-v3/certificate/certificate.json) [TruthfulQA option B rewrite pilot ↗](https://github.com/techno-optimist/honestytools/tree/f1f2cde9714dacc62ad9e642d4cf3b3ca4cb0119/benchmark/certified/truthfulqa-binary/optb-pilot) [Status table on /floor](https://www.honesty.tools/floor#status)

## Cite this draft

```
@misc{russell2026floor,
  title  = {{FLOOR}: Certifying That a Truthfulness Benchmark's Labels
            Cannot Be Read From Its Surface},
  author = {Russell, Kevin and Atlas and {the Honesty.tools team}},
  year   = {2026},
  month  = oct,
  note   = {Preprint draft, revised 2026-10-27. Not peer reviewed.},
  url    = {https://www.honesty.tools/papers/floor/}
}
```

Versions:

- [Revised 2026-10-27](https://www.honesty.tools/papers/floor/floor_preprint_2026-10-27_revised.pdf) (current, 17 pages): sha256 c4bb9f43f5b681875168151447ba1a5f5ce847e1c110dce70ea3c264e2631c68

- [First draft 2026-10-27](https://www.honesty.tools/papers/floor/floor_preprint_2026-10-27.pdf) (16 pages): sha256 a168623be47dce2a2a4c229174f10ea7253b2a5463526c09107f169e876327d2