# PRE-REGISTRATION (DRAFT): Epoch Benchmark Reviews × Honesty.tools /audit cue test, batch 1

Status: **DRAFT, not locked.** Items marked **[FORD]** must be settled before the lock. The lock means:
hash this file, commit it, and publish it on the page **before any batch-1 data is scored**.
Authors: Kevin Russell, Atlas, and the Honesty.tools team. Drafted 2026-10-27 06:56 MDT.
Scope and sources: `SCOPE.md` in this folder.

## 0. What is being compared

- **Epoch's axis** is Epoch's published verdict on the benchmark: Verified, Flawed or Not enough info. It
  answers Epoch's question: are the tasks and answer keys correct, and is the scoring sound?
- **Our axis** is the /audit cue-test verdict, DISQUALIFIED or PASS. It answers a different question: can a model-free surface
  probe recover the scored labels from the items' text alone?
- These are **different properties**. A benchmark can have correct labels and still give them away, or
  have wrong labels and not give them away. The crosswalk does not test either party's method against the other.

## 1. Benchmarks in scope (batch 1, fixed at lock)

| ID | Benchmark | Epoch verdict (theirs, verbatim) | Data | Pinned revision |
|---|---|---|---|---|
| B1 | Humanity's Last Exam, **multiple-choice subset only** | Flawed (Aug. 11, 2026), https://epoch.ai/benchmarks/hle/review | `cais/hle` test split, `answer_type` = multipleChoice **[FORD: confirm field value and count after access]** | 5a81a4c7271a2a2a312b9a690f0c2fde837e4c29 |
| B2 | SimpleQA Verified | Verified (Sep. 10, 2026), https://epoch.ai/benchmarks/simple-qa-verified/review | `google/simpleqa-verified` `simpleqa_verified.csv`, 1,000 rows, sha256 b5db21155444763543fe31b67e7cf28ce2bb225742a5b889421c5f182e2f92f5 | 0dc97e0d28d8233463e005cdc4475cc2a13ba2dc |
| B3 | GraphWalks | Verified (Sep. 29, 2026), https://epoch.ai/benchmarks/graphwalks/review | `openai/graphwalks` **[FORD: 128k file only (750 rows, sha256 54036036…b516) or both files]** | be6cc6ecf9b4d495b07d1ff53d2a16598e90fed7 |

Nothing else is added after lock. Further benchmarks go in a new pre-registration (batch 2).

## 2. The test and its threshold (as /audit already defines it; nothing new)

Tool: `nulltest.py` **v1.4** from techno-optimist/honesty.tools-site @ 211b569, path
`Aperture/web/static/verifier/nulltest.py`, sha256 `0e103dc41a3c208d2652646f2b8acc8b701ed7799402ca1ea0cb856de5ddf46c`.
Pin this sha in the lock.

The threshold as defined in the tool (no constant of ours):
- `evaluate(..., n_perm=600, seed=20260724, min_cell=12, alpha=0.05)` (nulltest.py line 582–583; CLI defaults
  `--n-perm 600` and `--alpha 0.05`, lines 1568 and 1573).
- The statistic is A* = max(AUROC, 1−AUROC) per (probe, stratum) cell, two-sided. The worst cell is used, not the mean
  (module docstring, rules 1–2).
- The null is "labels shuffled within stratum (stratified permutation), every probe refit" (line 613). The family-wise
  correction is `westfall_young_min_p` (line 612). B is raised automatically to at least 2(K+1)/alpha (lines 597–604).
- **Verdict:** `"DISQUALIFIED" if worst["adj_p"] < alpha else "PASS"` (line 716).
- **BORDERLINE flag:** `alpha/2 <= worst adj_p < 2*alpha` (line 723).
- **Power limit**, as published on honesty.tools/audit ("What this test can see at all"): "Below n≈700 an artifact must
  cover roughly 60% of the fabricated half before this test sees it … a PASS at small n establishes nothing."
- The docstring's framing is binding on how we word results: "A PASS IS NOT A CERTIFICATE OF QUALITY."

**[FORD]** Paired mode. For B1, the natural unit is a question with one keyed option and k−1 others. Settle one of:
(a) unpaired v1.4 with `--stratum-key category` (public, reproducible today); or
(b) paired mode (`nulltest_paired.py` PAIRED_VERSION 0.1, sha256 2fbbebca…37d3, the "honestytools 0.6.0" used for the
TruthfulQA cross-checks on /audit). Paired mode is **not in the public repo**, so publish it first or don't use it. It
also requires exactly one label-0 and one label-1 member per pair, so k-option questions need a pairing rule
(e.g. keyed option against each distractor, or against one seeded distractor).
Pick one **primary** analysis. The other, if run, is labelled exploratory.

## 3. Item construction per benchmark (fixed at lock)

**B1 HLE MC.** One item per answer option. text = the option text only (never the question, per the `--explode`
convention); label 0 = keyed option, label 1 = unkeyed; stratum = `category`; group key = `id`.
- **[FORD]** Write the option parser from the actual `question` format after access. Report how many questions don't parse
  and drop them with a count, never repairing them by hand.
- **[FORD]** Image questions: keep them (the option text is still text) or exclude them. Record the count either way.
- The class balance is about 1:(k−1). The majority-class rate is reported, as on other /audit rows.

**B2 SimpleQA Verified.** The data has **no negative class** (one reference answer per question). **[FORD] choose one:**
- (a) **Recommended: not applicable.** Record B2 as "cue test not applicable: single-class answer key" and keep it off the
  grid. This is honest and costs nothing.
- (b) Build the negatives ourselves, e.g. the reference answer of another question with the same `answer_type`, chosen
  with a seeded draw, with text = question + answer. Caveat that must go on the page: the negatives are ours, so a
  DISQUALIFIED would partly measure our construction, and the six probes barely see question–answer fit, so a PASS
  would say little.
- Not acceptable: using `multi_step` or `requires_reasoning` as the label. That would test whether metadata leaks, not
  whether the scored answer can be recovered.

**B3 GraphWalks.** There is no native two-class label, so we derive one from the benchmark's own data. For each task, parse
the final graph's edge list (excluding the 3-shot examples in `prompt`). Items are the nodes of that graph: label 0 = in
`answer_nodes`, label 1 = not. text = the node id; stratum = `problem_type`; group key = the task.
- **[FORD]** Confirm this construction, or record B3 as not applicable. Tasks with an empty answer (113 of 750 in the 128k file)
  contribute only label-1 items. Exclude them or keep them, and decide whether to subsample non-answer nodes (seeded) so
  the classes stay bounded.
- Note for the record: the node ids checked are md5(str(int))[:10] (3 of 3 checked). A cue that requires decoding the hash is
  outside the six probes. If such a cue exists, this test would not see it, and the PASS reading must say so.

## 4. Controls

1. **Shuffled-label control (negative control, required for every benchmark run).** Take the exact constructed item file,
   permute the labels **within group key** (within question for B1, within task for B3) with a seed fixed at lock
   **[FORD: seed]**, and run the identical command. Expected: PASS. If the control comes out DISQUALIFIED, that benchmark's
   real verdict is **void** and is reported as "control failed". It is not placed on the grid.
2. **Tool self-test (pipeline check).** Run `python3 nulltest.py --selftest` (the planted-failure control) on the DGX build
   before the real runs, and record its output hash.
3. **Positive control (optional, recommended) [FORD].** Plant a one-character marker on a fixed fraction of label-1 items
   in a copy of each constructed file, following the /audit power procedure. This confirms the construction can register a cue
   at this n. Fraction and seed are fixed at lock.

## 5. Four-box assignment rule (mechanical, fixed at lock)

Each benchmark is placed by the **primary** run only, after its shuffled-label control has passed.

| | Our verdict **DISQUALIFIED** | Our verdict **PASS** |
|---|---|---|
| **Epoch: Verified** | Box VG: "Epoch Verified; labels recoverable from surface" | Box VP: "Epoch Verified; no surface cue detected at this n" |
| **Epoch: Flawed** | Box FG: "Epoch Flawed; labels recoverable from surface" | Box FP: "Epoch Flawed; no surface cue detected at this n" |

Rules:
- The Epoch axis is Epoch's verdict, copied verbatim with its review date and URL as of the lock date. If Epoch changes a
  verdict later, we show both, dated, and don't move the dot silently.
- A PASS is worded "no surface cue detected at this n" and never "not gameable" or "clean".
- A PASS with n below about 700 items carries the published /audit power caveat on the dot.
- BORDERLINE results (line 723) keep their box but carry the BORDERLINE mark, as on /audit.
- **Not enough info** benchmarks don't go on the 2×2. They are listed beside it with our verdict only, if we ran one.
  (There are none in batch 1.)
- A benchmark recorded as "not applicable" (possibly B2 and B3) is listed with the reason and gets no dot.
- B1 is tested on the MC subset only. Its dot is labelled "HLE (MC subset)". The page says Epoch's Flawed verdict covers HLE
  as a whole and that Epoch's sample was not limited to MC.

## 6. How results are framed (binding)

- Epoch's verdict is always shown as **theirs**, quoted with a link, and never restated, re-scored or summarised in our words.
- Where the two axes disagree, we say they **measure different properties** (answer-key correctness against surface
  recoverability). Neither verdict is presented as wrong, and the result isn't framed as a "gotcha".
- A DISQUALIFIED here makes the /audit statement: "a model-based score on it needs a surface control to be interpretable." It
  is not a finding of error or misconduct.
- Every outcome is published, including all-PASS and "not applicable".

## 7. Outputs (hashed and published together)

For each benchmark: the constructed item file, the exact command, the `--json` report, the `--explain` feature tables, the
shuffled-control report and the self-test output, all with sha256 in a `SHA256SUMS`, plus the Epoch page snapshots (in `raw/`).

## 8. Open items for Ford (all must be closed before lock)

- O1: Choose unpaired v1.4 or paired mode as primary for B1. If paired, publish `nulltest_paired.py` first and set the k-option pairing rule.
- O2: HLE access on the DGX account; the MC field value and count; the option parser; how image questions are handled.
- O3: SimpleQA Verified: not applicable (recommended), or a constructed-negatives design with the caveat.
- O4: GraphWalks: confirm the node-membership construction (or not applicable), the file(s) in scope, empty-answer tasks and the subsample rule.
- O5: Seeds for the shuffled-label control and any subsampling. Keep the tool's default `seed=20260724` for the test itself unless Ford says otherwise.
- O6: Whether to add the positive control, and its marker fraction.
- O7: The lock procedure: who hashes and commits, and where the locked prereg is published (a /audit note or its own page). Any
  site change needs to go through Abe's normal content approval; any note to Epoch needs Kevin's explicit OK.