Right answers,
or give-away answers?

Status

Status: batch 1 run on 27 October 2026 (HLE 07:23–07:29 MDT, GraphWalks 07:33:58–07:35:17 MDT) under rules locked at 07:03:35 MDT, PREREG.md sha256 76b3091cc177df8c63a97e3d0c8efaef82d1e84b50abdfec17a2ebb8b1dcbc16. One deviation (a GraphWalks parser fix, accepted before scoring) and one earlier look at the data are disclosed below. Not signed, not anchored in Rekor. Files and hashes ↓

Batch 2 on HLE: PASS. The grouped test was borderline. Its first plan failed its pre-run check and was not run. A revised plan passed that check, was locked, and ran on 28 October 2026. Details ↓

GraphWalks’ PASS says little: its node names are random hash codes, so we expected no give-away. HLE batch 1 is void because our shuffled-label safety check flagged, so the test may not give a verdict either way.

Two checks, two different properties

These are different properties. A benchmark can have correct labels and still give them away, or wrong labels and not give them away. Where the two verdicts disagree, that is not a claim that Epoch is wrong. Epoch’s verdict is always shown as theirs, with a link, and never re-scored. A PASS from us means “no surface cue detected at this n”, not “clean”. A DISQUALIFIED means only that a model’s score on that benchmark needs a surface control to be read; it is not a finding of error or misconduct.

Where each benchmark landed

Not on the grid: Humanity’s Last Exam (multiple-choice, text-only subset), VOID, because the shuffled-label control fired. Batch 2 used a different design, so its HLE result is below, not on this grid. SimpleQA Verified, not applicable. The rows are Epoch’s verdicts, as theirs, with their review dates. A benchmark is placed only if both of its controls come out as required.

Batch 1 results

Every number below is copied from results.json and REPORT.md. Tool: nulltest.py v1.4, hash-checked before each run.

GraphWalks: PASS, meaning no surface cue detected at this n

Both controls came out as required, so the main run counts: box VP, “Epoch Verified; no surface cue detected at this n”. This is not a validation of GraphWalks, and it adds nothing to Epoch’s Verified.

Why this PASS carries little information. Every node id is the first 10 characters of an md5 hash of a number. The ids look random by design and are all the same length (which is also why the length probe has nothing to score in the main run: 10 cells there, 12 in the planted copies). A cue would have to decode the hash, and none of the six probes does that. The pre-lock look at the id format, disclosed below, also applies to this result.

Deviation 1: parser fix. The plan said to read the queried node from the last line containing “Operation:”. In the data that line is a bare header, and the sentence naming the node is on the next line, so the literal rule dropped all 1,006 tasks. DEVIATION_1.md reads the next non-empty line instead; nothing else changed. It was written before any GraphWalks item file was built or scored, and Ford accepted it at 07:33 MDT, before scoring (DEVIATION_1_DECISION.md, sha256 041cf0243a2613d7437cf0a80195251af78f530769c2623fee737e090e42eac6). Under it, 0 tasks were dropped. 144 tasks with an empty answer were excluded (113 + 31), and 345 tasks were capped at 5 answer nodes.

Humanity’s Last Exam: VOID, no verdict

The run used 467 text-only multiple-choice questions, giving 3,164 answer options. We dropped 78 questions with images and 46 that failed parsing (8.97%, under the 10% cutoff). The shuffled-label control scrambles which option is keyed within each question, so it should come out PASS. It flagged: words / Humanities/Social Science, A* 0.70, adjusted p 0.023. The unshuffled main run flagged the same cell (A* 0.706, adjusted p 0.022).

What that means: the test fires even when the answer key is scrambled, so what it picks up is not about which option is right. Under the pre-registration the result is void. The test is not allowed to give a verdict on HLE, and this says nothing either way about whether HLE’s answers can be found from surface cues. HLE stays off the grid.

The planted-cue rule was formally met (4 of 4), but it tells us nothing here: every planted copy fired on the same words / Humanities cell that fires without any plant. As the plan requires, an exploratory paired run (“exploratory, paired 0.1”) was also done. It came out PASS (A*_pair 0.585, adjusted p 0.6211, 467 pairs) and has no effect on any verdict.

SimpleQA Verified: not applicable

Nothing was downloaded or run. Each row has one reference answer and no wrong answers, so there is no two-class label to test (see scope below).

Batch 2 on HLE: PASS (grouped test borderline)

Short version: the first plan failed its pre-run check, so it was not run. A revised plan passed that check, was locked, and was run. Jump to the result ↓

Version 1: not run

We also publish checks that did not pass.

Before batch 2 could touch any HLE data, its method had to pass a check on synthetic data. The check failed. So batch 2 was not run, as the rule we set in advance requires.

• The rule: each of the two tests must find a planted one-character cue in at least 40 of 50 synthetic datasets.
• The result: with the cue in 60% of questions, the all-options test found it in 49 of 50. The keyed-versus-one-wrong test found it in only 32 of 50.
• So: not enough power. Nothing was run on HLE or scored. No threshold was changed.
• False alarms: those checks passed. The limit was 9 of 200 per test.

Version 1 has no result on HLE. The batch-1 HLE result stays void.

The full check

Run on our DGX on 27 October 2026, 08:04–13:56 MDT. It used no HLE data. Every number is copied from B2_SIM_GATE.json and NOTE.md.

The rule gave no further step after the 60% level, so the outcome is NOT RUN.

Check a file with sha256sum. The list is in SHA256SUMS.

What was in scope

Out of scope for this batch: the other 19 benchmarks Epoch has reviewed. The agentic ones (SWE-bench and the like) have no two-class text label for this test to score, and the grid has no row for a Not enough info verdict. Further benchmarks go in a new pre-registration.

Rules written first

The full plan is PREREG.md, settled and locked by Ford at 07:03:35 MDT on 27 October 2026 (13:03:35 UTC), from a draft by Abe (PREREG_DRAFT.md), and the scoping notes in SCOPE.md. It pins both by hash. The lock was private until this page published it. The locked file is never edited; any change goes in a separate, hashed deviation file written before the step it affects.

• No new thresholds. The tool’s own defaults: 600 permutations, alpha 0.05, worst cell, Westfall–Young correction. The run stops if the tool’s bytes differ from the pinned hash.
• Planted-cue check. On copies of each file we plant a one-character marker, as the tool’s own power test does, with 4 seeds. If fewer than 3 of 4 are caught, that benchmark is reported as INVALID, with no verdict.
• Shuffled-label check. Labels shuffled within each question or task should give a PASS. If it fires, that benchmark’s verdict is void.
• Fixed seeds for every random draw, listed in the plan.
• Power. At small n a PASS shows little; see /audit for what the test can see at each size.

What was seen before the lock

Ford, who settled and locked the plan, opened no benchmark items or labels; he used Hugging Face metadata, the GraphWalks dataset card, saved copies of Epoch’s review pages, and the tool’s own code and docs. No one had accessed any HLE data before the lock. Also before the lock, the draft’s author looked at two files, and the plan discloses it:

• SimpleQA Verified: its CSV (1,000 rows) was downloaded. No effect on any verdict, because SimpleQA Verified is not run.
• GraphWalks: the smaller of its two files was downloaded, and three facts were noted: 750 rows (400 “parents”, 350 “bfs”), 113 tasks with an empty answer, and that 3 node ids checked are the first 10 characters of the md5 of a small integer. No cue statistic was computed. But the GraphWalks design (text = node id) was chosen knowing the id format, so its result will carry this note wherever it is shown.

Files and hashes

Byte-identical copies of the locked plan and the batch 1 results, aggregates only. Check any file with sha256sum; the full list is in SHA256SUMS. No HLE question or answer text is published: HLE items, explain tables and per-item outputs stay on our internal machine. The saved Epoch pages and downloaded data are not published here either; their hashes are in raw/SHA256SUMS, which PREREG.md pins.

By Kevin Russell, Atlas, and the Honesty.tools team.