# Epoch Benchmark Reviews × /audit crosswalk: SCOPE

Status: research only. No site change, post or outreach was made. Compiled 2026-10-27 06:56 MDT.
Source snapshots are in `raw/`, with hashes in `raw/SHA256SUMS`.

## 1. Epoch's reviewed list, checked against Ford's count

Source: https://epoch.ai/data/benchmark-reviews-documentation/included-benchmarks, fetched 2026-10-27
(`raw/epoch_included-benchmarks_2026-10-27.html`). The table has **22 rows**, which matches Ford.
They break down as **Verified 5, Flawed 12, Not enough info 5**.

| Benchmark | Verdict | Review date | Review URL |
|---|---|---|---|
| Chartography | Not enough info | Oct. 21, 2026 | (no review link on the index page) |
| TasteVal | Not enough info | Oct. 21, 2026 | (no review link on the index page) |
| CMT-Benchmark v1.0.0 | Flawed | Oct. 14, 2026 | https://epoch.ai/benchmarks/cmt-benchmark/review |
| CritPt v1 | Flawed | Oct. 13, 2026 | https://epoch.ai/benchmarks/critpt/review |
| Remote Labor Index | Not enough info | Oct. 12, 2026 | (no review link on the index page) |
| Surface Evolver Bench | Not enough info | Oct. 9, 2026 | (no review link on the index page) |
| GraphWalks | Verified | Sep. 29, 2026 | https://epoch.ai/benchmarks/graphwalks/review |
| OSWorld 2.1 | Flawed | Sep. 28, 2026 | https://epoch.ai/benchmarks/osworld-2-1/review |
| ExploitBench v0.1 | Verified | Sep. 12, 2026 | https://epoch.ai/benchmarks/exploitbench/review |
| FrontierCode | Not enough info | Sep. 11, 2026 | (no review link on the index page) |
| Berkeley Function Calling Leaderboard (BFCL) v4 | Flawed | Sep. 10, 2026 | https://epoch.ai/benchmarks/berkeley-function-calling-leaderboard/review |
| HealthBench Professional | Flawed | Sep. 10, 2026 | https://epoch.ai/benchmarks/healthbench-professional/review |
| SimpleQA Verified | Verified | Sep. 10, 2026 | https://epoch.ai/benchmarks/simple-qa-verified/review |
| PostTrainBench v1.1 | Verified | Sep. 9, 2026 | https://epoch.ai/benchmarks/post-train-bench/review |
| DeepSWE v1.1 | Flawed | Sep. 7, 2026 | https://epoch.ai/benchmarks/deepswe/review |
| Terminal-Bench 4.0.0 | Flawed | Sep. 4, 2026 | https://epoch.ai/benchmarks/terminal-bench-4/review |
| SWE-bench Verified | Flawed | Sep. 3, 2026 | https://epoch.ai/benchmarks/swe-bench-verified/review |
| SWE-Bench Pro | Flawed | Sep. 1, 2026 | https://epoch.ai/benchmarks/swe-bench-pro/review |
| Humanity's Last Exam | Flawed | Aug. 11, 2026 | https://epoch.ai/benchmarks/hle/review |
| Lech Mazur Writing | Flawed | Aug. 10, 2026 | https://epoch.ai/benchmarks/lech-mazur-writing/review |
| TextQuests | Flawed | Aug. 10, 2026 | https://epoch.ai/benchmarks/textquests/review |
| WeirdML v2 | Verified | Aug. 10, 2026 | https://epoch.ai/benchmarks/weirdml/review |

I only read the reasons for the three batch-1 benchmarks (per the steering note). The other 19 rows
carry the verdict as listed and nothing else.

**Already on /audit:** none of the 22. I searched audit.html and the verifier JSONs for
HLE, SimpleQA and GraphWalks, and the 24 targets in `nulltest_audit_manifest.json` are all
honesty, hallucination or deception sets (TruthfulQA, HaluEval, RAGTruth, Apollo, geometry-of-truth and others).

## 2. Batch 1 (Ford's ranking): Epoch's reasons, quoted exactly

### Humanity's Last Exam: **Flawed** (Aug. 11, 2026)
Source: https://epoch.ai/benchmarks/hle/review
> "After auditing a random sample of 48 questions, we found that 22 (46%) questions had substantial accuracy-altering errors, leading us to designate the benchmark as Flawed."
> "We reviewed the original benchmark, Humanity's Last Exam, not Humanity's Last Exam-Rolling or Humanity's Last Exam-Verified."
> "This is a partial review. We stopped inspecting once we discovered errors that exceeded our published threshold for a flawed benchmark."

Scope caveat: Epoch reviewed the whole benchmark and sampled 6 from each of 8 categories. Our run
would cover only the multiple-choice subset, so the crosswalk must say the verdict is Epoch's for
HLE as a whole, not for the MC subset. One of Epoch's representative errors is itself an MC item
("The author's answer key ends with the formula printed in option E, but the answer key says D").

### SimpleQA Verified: **Verified** (Sep. 10, 2026)
Source: https://epoch.ai/benchmarks/simple-qa-verified/review
> "SimpleQA Verified successfully measures whether a model can recall obscure facts with no tool usage."
> "In a random, topic-stratified sample of 50 questions, we found 5 (10%) with score-affecting defects. [...] This error rate is well below our 20% threshold for declaring a benchmark flawed, but still worth being aware of."
> "All questions and answers have been public since 2024 as part of SimpleQA, so the benchmark is not contamination-resistant"

### GraphWalks: **Verified** (Sep. 29, 2026)
Source: https://epoch.ai/benchmarks/graphwalks/review
> "GraphWalks tests a model's ability to find and combine information across very long contexts."
> "The tasks were all well constructed and an independent solver reproduced the stated answer for all 1,150 tasks."

## 3. Data availability (Hugging Face API, 2026-10-27)

| Benchmark | Dataset | Access | License (card) | Size | Revision sha |
|---|---|---|---|---|---|
| HLE | `cais/hle` | **gated: auto** (needs an HF login and accepting the terms; I did not do this) | mit | test split 2,500 rows; parquet ~274 MB (images included) | 5a81a4c7271a2a2a312b9a690f0c2fde837e4c29 |
| SimpleQA Verified | `google/simpleqa-verified` | open | mit | `simpleqa_verified.csv` 349,184 B, **1,000 rows** (downloaded, sha256 b5db2115…92f5, in `raw/`) | 0dc97e0d28d8233463e005cdc4475cc2a13ba2dc |
| GraphWalks | `openai/graphwalks` | open | mit | `graphwalks_128k_and_shorter.parquet` 22.1 MB, **750 rows** (parents 400, bfs 350; downloaded to /tmp only, sha256 54036036…b516); `graphwalks_256k_to_1mil.parquet` 296.9 MB (not downloaded, row count unverified) | be6cc6ecf9b4d495b07d1ff53d2a16598e90fed7 |

HLE schema (from the card): id, question, image, answer, answer_type, rationale, raw_subject, category, ….
The MC subset size and the way options appear in `question` are **unverified** because the set is gated.

## 4. Can the /audit cue test run on these?

How /audit works (honesty.tools-site @ 211b569, `Aperture/web/static/verifier/nulltest.py` v1.4,
last changed d034407, sha256 0e103dc4…f37d3, plus its README.md):
- It needs **one text field and one two-class label per item**, in JSON, JSONL, CSV or TSV. The README says
  "A benchmark with more than two classes has to be filtered to two first; the tool refuses to pick."
- For QA sets, `--explode FIELD=LABEL` turns each answer into its own item. The question it asks is
  "can the answer be told right from wrong **without the question**?"
- It runs six model-free probes (lexicality, words, length, shape, alpha_first, alpha_last) and takes the worst
  (probe × stratum) cell, two-sided. The null is a stratified label permutation with every probe refit,
  with Westfall–Young min-p family-wise correction.
- Paired mode (`--pair-key`, "honestytools 0.6.0" on /audit) exists in `/tmp/floorv1-dgx/code/nulltest_paired.py`
  (sha256 2fbbebca…37d3). It is **not in the public site repo** (`git ls-files | grep paired` is empty).

| Benchmark | Native two-class label? | Fit |
|---|---|---|
| HLE MC subset | Yes, by construction. Each MC question has one keyed option and k−1 unkeyed ones. Explode the options with correct=0 and incorrect=1, and use the question id as the pair/group key. | **Runnable**, once access is granted and the option format has been parsed and checked. Best fit of the three. |
| SimpleQA Verified | **No.** Each row is a question and its single reference answer, with no wrong answers in the data. The only binary columns (`multi_step`, `requires_reasoning`) are metadata, not correctness. | **Not natively runnable.** We would have to build the negatives ourselves (open item O3). |
| GraphWalks | **No two-class label.** `answer_nodes` is a set of node ids, and 113 of 750 tasks (128k file) have an empty answer. A two-class version could come from the benchmark's own data: for each task, answer nodes against the graph's other nodes. | **Runnable only after we derive the classes** (O4). Low prior for a surface cue: the node ids I checked are md5 prefixes of small integers (1679091c5a = md5("6")[:10], c9f0f895fb = md5("8"), d3d9446802 = md5("10")). That is 3 ids, so it's an observation, not a finding. |

Agentic benchmarks (SWE-bench, OSWorld, Terminal-Bench and the like) are a poor fit, as Ford said: they have no
two-class text label.

**Overlap count:** 22 reviewed, 0 already on /audit. 3 are in batch 1. Of those, 1 is natively runnable (HLE MC,
gated), 1 is runnable with a derived label (GraphWalks) and 1 is not natively runnable (SimpleQA Verified).