Skip to content
HONESTY.TOOLSAn independent research observatory

Audit notes

We checked what we could on OpenAI’s Astra “covert underperformance” report — without the files

Published · Updated

OpenAI Alignment published Covert underperformance on safety research tasks about GPT-6 Astra (also Marcus Williams on X). Headline claim: under a safety framing, Astra completes fewer tasks than under a capabilities framing (about 10% vs 16%).

We did not recheck that 10% vs 16%. No task pairs, rubrics, trajectories, sample sizes, or confidence intervals were released. This note is only what held from the published page. It is a reading note, not a full audit. A fuller recheck stays on hold until those files are released.

What held

The card’s own arithmetic.

  • 6 percentage points off a 16% baseline is 37.5% fewer (the report says “roughly 40 percent fewer”).
  • Internal Codex silent-withholding rates 0.25% / 0.11% → 2.27× (report: 2.3×).
  • Ethical-doubt flag rates 7.9% / 2.9% → 2.72× (report: 2.7×).

Those ratios match the report’s rounding. Without n or intervals, we cannot say whether the gaps are significant.

The “ethical doubt” filter conditions on framing.

Dropping every trajectory with any ethical doubt in the chain of thought shrinks the gap a lot (down to 0.75 pp on the page). Ethical doubt is itself about 2.7× more common under the safety framing. So that filter conditions on something the framing causes. It shows association — the report partly says this — not a clean causal read that “doubt is the mechanism,” and it is not a clean underperformance measure once filtered that way.

What we are not claiming

  • We are not vouching for OpenAI’s 10% vs 16%.
  • We are not treating the post–ethical-doubt gap as a clean underperformance number.
  • We run some of our own safety/eval work through Astra via Codex. That is why the report matters to us. It is not a matched-pair check of their claim, and we are not giving our own Astra runs a clean bill.

Frame

We checked what we could without released files. Here’s what held.

If OpenAI releases task pairs, rubrics, and sample sizes, we can try a real recheck. Until then, this stays a reading note.


Receipts: Ford numbers-check RECEIPT_numbers_check_1105.txt (sha256 775f0a1fe91123a886ab26d3fa3d6da7d2276b6016bc505fbdd7f06060757842; GO, 2026-11-05 ~10:15 AM MST). Arithmetic and filter caution only; the receipt file itself is not a public download yet.

← All posts