# PRE-REGISTRATION (LOCKED): Epoch Benchmark Reviews × Honesty.tools /audit cue test, batch 1

**Status: LOCKED.** Locked 2026-10-27 13:03:35Z (2026-10-27 07:03:35 MDT) by Ford for Kevin Russell / Honesty.tools (P42).
Authors: Kevin Russell and the Honesty.tools team. Drafted by Abe (PREREG_DRAFT.md); open items settled and locked by Ford.

| Pinned input | sha256 |
|---|---|
| `SCOPE.md` (this folder) | `1962f2c1f5815f4acf73f881bb2740d5727439551314d5c5bb9b52b909bd793a` |
| `PREREG_DRAFT.md` (Abe's draft, superseded by this file) | `08b0f28355c90cd749565575c642dee6f657cf56478670b626f33672dc1d4b84` |
| `raw/SHA256SUMS` (Epoch page snapshots) | `e809e2116bddc414be21bd6bca0aefaaf628168c0e0c408909b470984472fc6f` |

The hash of this file is in `PREREG.md.sha256`. This file is never edited after the lock. Any change
goes in a separate, hashed `DEVIATION_n.md` written before the affected step, saying what changed and why.

## Data-touch statement

**No benchmark item or label was downloaded, opened or inspected by Ford in settling this pre-registration.**
Ford used only: Hugging Face Hub API metadata (`/api/datasets/<id>`, `?full=true`, `/tree/main`: revision sha, gating,
licence, schema, row counts, file names, sizes, LFS sha256), the GraphWalks dataset card (whose example prompt is a
toy graph, `uvwx -> alke` etc., not a benchmark item), the Epoch review snapshots already in `raw/`, and the
nulltest source and docs. The HLE card body and the HF dataset-viewer `/info` and `/size` endpoints were refused
without authentication (gated) and were not pursued further. No HLE data was accessed by anyone.

**Disclosure of pre-lock exposure by the drafter (from SCOPE.md §3–4, not by Ford):**
- `raw/simpleqa_verified.csv` (B2, 1,000 rows, sha256 b5db2115…92f5) was downloaded into this folder before the lock.
  Consequence: none for any verdict, because B2 is not run (O3).
- `openai/graphwalks` `graphwalks_128k_and_shorter.parquet` was downloaded to /tmp before the lock, and three facts were
  recorded: 750 rows (parents 400, bfs 350), 113 tasks with an empty answer, and that 3 node ids checked are
  md5(str(int))[:10]. No cue statistic was computed. Consequence: the B3 design (text = node id) was chosen with knowledge
  of the id format. The B3 result carries this disclosure wherever it is shown.

## 0. What is being compared (unchanged from draft)

- **Epoch's axis**: Epoch's published verdict (Verified / Flawed / Not enough info): are tasks and answer keys correct and is scoring sound?
- **Our axis**: the /audit cue-test verdict (DISQUALIFIED / PASS): can model-free surface probes recover the scored labels from item text alone?
- These are **different properties**. A benchmark can have correct labels and still give them away, or wrong labels and not give them away.
  The crosswalk does not test either party's method against the other.

## 1. Benchmarks in scope (batch 1, fixed)

| ID | Benchmark | Epoch verdict (theirs) | Data (pinned) | Status at lock |
|---|---|---|---|---|
| B1 | Humanity's Last Exam, **multiple-choice, text-only subset** | Flawed (Aug. 11, 2026), https://epoch.ai/benchmarks/hle/review | `cais/hle` @ `5a81a4c7271a2a2a312b9a690f0c2fde837e4c29`, file `data/test-00000-of-00001.parquet` (274,276,147 B; last data commit 021a3d71 "Minor update - 17 questions", 2025-05-20). Licence MIT (card metadata). 2,500 test rows (card `dataset_info`). | Runnable after access (O2) |
| B2 | SimpleQA Verified | Verified (Sep. 10, 2026), https://epoch.ai/benchmarks/simple-qa-verified/review | `google/simpleqa-verified` @ `0dc97e0d28d8233463e005cdc4475cc2a13ba2dc` | **Not applicable** (O3) |
| B3 | GraphWalks (derived node-membership label) | Verified (Sep. 29, 2026), https://epoch.ai/benchmarks/graphwalks/review | `openai/graphwalks` @ `be6cc6ecf9b4d495b07d1ff53d2a16598e90fed7`, both files: `graphwalks_128k_and_shorter.parquet` (22,128,284 B, sha256 54036036c91d8e04bb2a5fcd9e36f8e2a852cacece5dfc2b1ee40e3a6182b516) and `graphwalks_256k_to_1mil.parquet` (296,903,814 B, sha256 537879431c72a42e3b500f80efc3047e7facb90390b6063d33679b4320985911). Licence MIT (card). | Runnable (O4) |

Every downloaded file is hash-checked against the sha256 above before use; a mismatch stops that benchmark (reported, not repaired).
Nothing is added after lock. Further benchmarks go in a new pre-registration (batch 2).

## 2. The test and its threshold (as /audit defines it; nothing new)

**Primary tool:** `nulltest.py` **v1.4** (`VERSION = "1.4"`, line 1301), sha256
`0e103dc41a3c208d2652646f2b8acc8b701ed7799402ca1ea0cb856de5ddf46c`. The same bytes are at
techno-optimist/honesty.tools-site @ 211b569 `Aperture/web/static/verifier/nulltest.py` and at
techno-optimist/honestytools @ fb3dcc5 `nulltest.py` (both checked at lock; the raw.githubusercontent copy of honestytools main hashes identically).
The run aborts if the tool's sha256 differs.

Thresholds are the tool's own (no constant of ours):
- `evaluate(..., n_perm=600, seed=20260724, min_cell=12, alpha=0.05)` (nulltest.py lines 582–583); CLI defaults `--n-perm 600`, `--alpha 0.05` (lines 1568, 1573). Defaults are used unchanged.
- A* = max(AUROC, 1−AUROC) per (probe, stratum) cell; worst cell, not mean (module docstring rules 1–2). Six probes in v1.4: lexicality, words, length, shape, alpha_first, alpha_last (docstring; `NULLS`, line 258). Note: the verifier README's probe table still lists a seventh `stratum` probe from v1.3; the v1.4 code and docstring govern.
- Null: "labels shuffled within stratum (stratified permutation), every probe refit" (line 613); Westfall–Young min-p (line 612); B raised automatically to ≥ 2(K+1)/alpha (lines 597–604).
- Verdict: `"DISQUALIFIED" if worst["adj_p"] < alpha else "PASS"` (line 716); NOT_SCORABLE if no cell scores (lines 620–627). BORDERLINE: `alpha/2 <= adj_p < 2*alpha` (line 723).
- Power caveat as published on honesty.tools/audit ("What this test can see at all", audit.html line 224ff): "Below n≈700 an artifact must cover roughly 60% of the fabricated half before this test sees it … a PASS at small n establishes nothing."
- Binding framing (docstring): "A PASS IS NOT A CERTIFICATE OF QUALITY."

## 3. Decisions on the seven open items

### O1. B1 analysis mode: **unpaired nulltest.py v1.4 is primary; paired mode is exploratory only.**
Primary command (B1): `python3 nulltest.py b1_items.jsonl --text-key text --label-key label --stratum-key category --json` (`--jobs N` allowed; per the tool's help it changes wall clock only).
Exploratory (no effect on any verdict, labelled "exploratory, paired 0.1" wherever shown): `python3 nulltest_paired.py b1_pairs.jsonl --text-key text --label-key label --stratum-key category --pair-key id --json`, using `nulltest_paired.py` PAIRED_VERSION 0.1, sha256 `2fbbebca77abd8b494272801fa3f957e1f910d7a10148a7c6ccc10116e4f37d3` (honestytools @ fb3dcc5; added in e17c955). Pairing rule: per question, the keyed option (label 0) against **one** distractor (label 1) drawn with `random.Random(20261030)` (O5).
**Reasons:** (1) every row on /audit is scored with the unpaired tool, and the v1.4 path is the one with a measured false-positive rate (README, "Measured false-positive rate") and JS parity; the README itself calls paired mode "experimental … versioned separately and … not part of the JS-parity core". (2) Paired mode requires exactly one label-0 and one label-1 item per pair and drops other groups (nulltest_paired.py docstring and `make_pairs`), so k-option questions would need a pairing rule that discards k−2 options; that is our construction, not the benchmark's. (3) The --explode convention on /audit (README, "Question-answer benchmarks") already scores answers pooled across questions. **Correction to the draft:** paired mode *is* public, in techno-optimist/honestytools (public repo, main @ fb3dcc5); it is only absent from honesty.tools-site. That does not change the decision.

### O2. HLE access, MC subset, option parser, image questions.
- **Source: `cais/hle` at the pinned revision only.** It is gated "auto" (HF API `gated: "auto"`): it needs an HF account to accept the dataset's terms. **That acceptance is an account action and needs Kevin's explicit OK** (who accepts, with which HF account); if it is not given, B1 is reported as "not run: no data access" and gets no dot.
- **Ungated mirrors are not used.** Checked by metadata only: `macabdul9/hle_text_only` (MIT tag, 2,370 rows, last modified 2025-02-18, card says "3,000 questions") predates the 2025-05-20 data update of `cais/hle` and so is not the pinned benchmark; `jxcai-scale/hle-public-questions` (one CSV "Public Prompt Set - Public Set 04_03_2025 (2500 questions)", no licence in card metadata) is a third-party file of uncertain contents. In addition the official card asks users to help "protect the integrity of this benchmark by not publicly sharing, re-uploading, or distributing the dataset", which argues against relying on re-uploads. A text-only subset is instead derived from the official file by rule (below).
- **MC subset:** rows whose `answer_type`, lowercased with non-letters removed, equals `multiplechoice`. The field and its values are unverified at lock (gated); if no such value exists, B1 stops and needs a DEVIATION before anything is scored.
- **Image questions: excluded from the primary analysis** (rows with a non-empty `image` field). Reason: the option text of an image question may be meaningful only together with the image, and the task scope prefers text-only; no strong reason to include them was given in the draft. Their count is reported. They are not run in any arm.
- **Option parser (fixed now, written before data access):** within `question`, take the **last** contiguous block of lines matching `^\s*\(?([A-Z])[\.\):]\s+(.*\S)\s*$`. Each match is one option; the option text is group 2 with the letter prefix removed (so the letter itself is never in the probed text). The keyed option is the one whose letter equals `answer.strip().upper()` with any surrounding parentheses or trailing period removed. A question is **dropped and counted** (never hand-repaired) if: fewer than 2 options parse, letters are not consecutive from A, the key matches 0 or >1 options, or two options have identical text. If more than 10% of text-only MC questions are dropped, B1 is reported as "not run: parser failure" with the count; the parser may be changed only by a DEVIATION written before any nulltest run on B1.
- **Items:** one per option; text = option text only (never the question, per the --explode convention); label 0 = keyed, 1 = unkeyed; stratum = `category`; group = `id`. The class balance is about 1:(k−1) and is reported with the majority-class rate.
- **Exploratory descriptive only:** the distribution of keyed letters (A, B, C…) is reported as counts. It is not a nulltest input and does not affect any verdict.
- **Redistribution:** because of the HLE card's request, B1's constructed item files, `--explain` tables and WHY strings stay **internal** (DGX). Published for B1: the builder script, all sha256s, and the verdict numbers (verdict, worst cell, A*, adj p, n, family size, n_perm).
- Scope wording (from SCOPE.md): Epoch's Flawed verdict covers HLE as a whole (48 sampled questions, 6 from each of 8 categories, epoch_review_hle.txt line 145); our dot is labelled "HLE (MC, text-only subset)".

### O3. SimpleQA Verified: **not applicable. No exploratory arm in batch 1.**
Recorded as "cue test not applicable: single-class answer key" and kept off the grid, with the reason. **Reason:** the tool needs two classes per item (README: "A benchmark with more than two classes has to be filtered to two first; the tool refuses to pick"; the same logic applies to one class), and each SimpleQA Verified row has one reference answer and no wrong answers (SCOPE.md §4). The only binary columns (`multi_step`, `requires_reasoning`) are metadata, not correctness. Constructed negatives (another question's answer) would make the label ours, not the benchmark's, so neither a PASS nor a DISQUALIFIED would describe SimpleQA Verified. No principled two-class design exists in the published data, so no exploratory arm is run (this also avoids building anything on the CSV the drafter already downloaded).

### O4. GraphWalks: **node-membership construction confirmed, both files in scope, empty-answer tasks excluded, bounded seeded subsample.**
- **Files: both** (750 + 400 tasks). Reason: Epoch's verdict covers 1,150 tasks ("an independent solver reproduced the stated answer for all 1,150 tasks", epoch_review_graphwalks.txt line 148), and the >128k subset is "200 prompts at 256k tokens and 200 prompts at 1M tokens" (line 241); testing only the 128k file would test a subset of what Epoch verified. The file is recorded per item and reported descriptively; it is not a stratum (more strata enlarge the family and cost power, README "A wider family costs power").
- **Answer field:** the card schema names the column `answer` ("A list of node ids"); the draft called it `answer_nodes`. Use whichever single one of the two exists; if both or neither exist, B3 stops for a DEVIATION.
- **Graph parser (fixed now):** the final graph is the last contiguous block of lines matching `^\s*(\S+)\s*->\s*(\S+)\s*$` that precedes the last line containing `Operation:` (the card says the prompt is "A 3-shot example followed by the graph and the operation", so the last block is the task's graph). The queried node is the node named in the final operation line (the token after "node", trailing punctuation stripped). A task is **dropped and counted** if the queried node or any answer node is not in the final graph's node set. If more than 10% of non-empty tasks are dropped, B3 is "not run: parser failure"; parser changes only by DEVIATION before any B3 nulltest run.
- **Empty-answer tasks: excluded** (count reported; SCOPE.md counts 113 of 750 in the 128k file). Reason: they contribute only label-1 items, so no within-task contrast exists and the within-task shuffle control is undefined for them. The card's grader scores an empty set as correct (n_golden == n_sampled == 0 gives 1.0), so they are legitimate tasks, just outside a two-class test.
- **Items, per task:** up to 5 answer nodes (label 0) and the same number of non-answer nodes (label 1) from the final graph, excluding the queried node; if a set is larger, draw with `random.Random(20261028)` (O5). text = the node id string only; stratum = `problem_type` (bfs / parents); group = task row index within its file, prefixed with the file name. The cap bounds each task's weight and the 1:1 balance keeps the classes even; both numbers are design choices, not measurements.
- **Reading rule:** node ids observed by the drafter are md5(str(int))[:10] (3 of 3 checked; SCOPE.md §4). A cue that requires decoding the hash is outside the six probes; a PASS is worded "no surface cue detected at this n" and says this. The dot is labelled "GraphWalks (derived node-membership label)" and carries the pre-lock-exposure disclosure.

### O5. Seeds (all fixed; Python 3 `random.Random(seed)`, applied to items sorted by (group, item text, label) first so the draw is order-independent).
| Use | Seed |
|---|---|
| nulltest permutations and folds | **20260724**, the tool's built-in default (`evaluate` and `_fold_of`); v1.4 has no CLI flag to change it, and it is not changed |
| Shuffled-label control (within-group label permutation) | **20261027** |
| B3 subsample of answer / non-answer nodes | **20261028** |
| Planted-cue check, seed s = 0..3 | **20261029 + s** |
| B1 exploratory paired arm, one-distractor draw | **20261030** |
| `--selftest` | tool default `--seed-base 9000` |

### O6. Controls (required unless marked otherwise).
1. **Planted-cue check (positive control, required, gates validity).** On a copy of each constructed primary file, append the single character `x` to a fraction f of the label-1 items, chosen by seed 20261029+s. This is the tool's own power procedure (nulltest.py `measure_power` and its procedure comment, lines 862–926: append ONE character, DISQUALIFIED in ≥3 of 4 seeds). f is read from the published power figures, taking the more conservative (larger) of the /audit page (n=156/324/702 → 60%, n=1496 → 30%) and the code's `_MDE` (60/60/30/15%): **f = 60% if n < 1496, f = 30% if n ≥ 1496**, n being the constructed file's item count. Run the identical primary command on each of the 4 copies. **If fewer than 3 of 4 are DISQUALIFIED, that benchmark's run is INVALID**: reported as "INVALID: planted cue not detected", no verdict, no dot.
2. **Shuffled-label control (negative control, required).** Permute labels **within group** (within question for B1, within task for B3) with seed 20261027 and run the identical command. Expected PASS. If DISQUALIFIED, that benchmark's real verdict is **void** ("control failed"), no dot. Note: the tool's measured false-positive rate is about 5% (README: pooled 4.6%), so this control can void a clean benchmark by chance about 1 time in 20; that cost is accepted rather than adding post-hoc reruns.
3. **Tool self-test (pipeline smoke test, recorded, not a gate).** `python3 nulltest.py --selftest` once on the DGX before real runs; output hash recorded. The README says it "usually returns INDISTINGUISHABLE, which is the honest answer", so it cannot gate.

Order per benchmark: build items → hash → planted-cue (4 runs) → shuffled control → primary run. The primary verdict is read only if both controls are satisfied; all outputs are kept and published (subject to O2's HLE rule) whatever they show.

### O7. Locking.
- **Lock = this file plus its hash.** Ford wrote PREREG.md, computed sha256 into `PREREG.md.sha256`, and pushed both to the DGX (`~/honesty-tools/runs/epoch-crosswalk/`) with a verified hash match. This is a private lock; there is no public timestamp until publication. No Rekor anchoring and no signing (not authorised for this work).
- **Publication:** Abe publishes the locked bytes on the site through the normal content approval; the published file must hash to `PREREG.md.sha256`. No commit to a public repo is made by this lock.
- **Run gate:** no benchmark data is downloaded or built until (a) this hash is on the DGX and (b) the Goodfire three-feature run has finished. No nulltest is run on benchmark data until, in addition, (c) the locked file is published by Abe with a matching hash, and (d) for B1, Kevin has OK'd the HLE terms acceptance.
- **Deviations:** any change after lock is a new hashed `DEVIATION_n.md`, written before the step it affects; PREREG.md is never edited.
- **Outside contact:** any note to Epoch or to dataset authors needs Kevin's explicit OK.

## 4. Four-box assignment (mechanical; from the draft, with the outcomes above)

Each benchmark is placed by its **primary** run only, after both controls are satisfied.

| | Our verdict **DISQUALIFIED** | Our verdict **PASS** |
|---|---|---|
| **Epoch: Verified** | VG: "Epoch Verified; labels recoverable from surface" | VP: "Epoch Verified; no surface cue detected at this n" |
| **Epoch: Flawed** | FG: "Epoch Flawed; labels recoverable from surface" | FP: "Epoch Flawed; no surface cue detected at this n" |

- Epoch's axis is Epoch's verdict, copied with its review date and URL as of lock. If Epoch changes a verdict later, both are shown, dated.
- PASS is worded "no surface cue detected at this n", never "not gameable" or "clean"; n < ~700 carries the /audit power caveat. BORDERLINE keeps its box with the mark.
- Not placed on the grid, but listed with the reason: "not applicable" (B2), "not run" (access or parser), "INVALID" (planted cue missed), "control failed" (shuffled control fired), NOT_SCORABLE.
- Dot labels: "HLE (MC, text-only subset)"; "GraphWalks (derived node-membership label)".

## 5. Framing (binding, from the draft)

- Epoch's verdict is always shown as **theirs**, quoted with a link, never restated or re-scored.
- Where the axes disagree, they **measure different properties** (answer-key correctness vs surface recoverability). Neither is presented as wrong; no "gotcha" framing.
- A DISQUALIFIED makes only the /audit statement: "a model-based score on it needs a surface control to be interpretable." It is not a finding of error or misconduct.
- Every outcome is published, including all-PASS, not applicable, not run and INVALID.

## 6. Outputs (hashed together in SHA256SUMS)

Per benchmark: builder script, constructed item files (B3 public; B1 internal per O2), exact commands, `--json` reports for primary, shuffled control and the 4 planted-cue copies, `--explain` tables (B1 internal), the self-test output, the drop counts (parser, image, empty-answer), and the Epoch snapshots in `raw/`.