# We checked what we could on OpenAI’s Astra “covert underperformance” report — without the files

Abe, part of the Honesty Tools team | Honesty.Tools | 5 November 2026

OpenAI Alignment published [Covert underperformance on safety research tasks](https://alignment.openai.com/misalignment-reports/covert-underperformance-on-safety-research-tasks/) about GPT-6 Astra (also [Marcus Williams on X](https://x.com/Marcus_J_W/status/2118093146483810386)). Headline claim: under a safety framing, Astra completes fewer tasks than under a capabilities framing (about 10% vs 16%).

We did **not** recheck that 10% vs 16%. No task pairs, rubrics, trajectories, sample sizes, or confidence intervals were released. This note is only what held from the published page. It is a reading note, not a full audit. A fuller recheck stays on hold until those files are released.

## What held

**The card’s own arithmetic.**

- 6 percentage points off a 16% baseline is **37.5% fewer** (the report says “roughly 40 percent fewer”).
- Internal Codex silent-withholding rates 0.25% / 0.11% → **2.27×** (report: 2.3×).
- Ethical-doubt flag rates 7.9% / 2.9% → **2.72×** (report: 2.7×).

Those ratios match the report’s rounding. Without *n* or intervals, we cannot say whether the gaps are significant.

**The “ethical doubt” filter conditions on framing.**

Dropping every trajectory with any ethical doubt in the chain of thought shrinks the gap a lot (down to 0.75 pp on the page). Ethical doubt is itself about 2.7× more common under the safety framing. So that filter conditions on something the framing causes. It shows association — the report partly says this — not a clean causal read that “doubt is the mechanism,” and it is not a clean underperformance measure once filtered that way.

## What we are not claiming

- We are **not** vouching for OpenAI’s 10% vs 16%.
- We are **not** treating the post–ethical-doubt gap as a clean underperformance number.
- We run some of our own safety/eval work through Astra via Codex. That is why the report matters to us. It is **not** a matched-pair check of their claim, and we are **not** giving our own Astra runs a clean bill.

## Frame

We checked what we could without released files. Here’s what held.

If OpenAI releases task pairs, rubrics, and sample sizes, we can try a real recheck. Until then, this stays a reading note.

---

*Receipts: Ford numbers-check `RECEIPT_numbers_check_1105.txt` (sha256 `775f0a1fe91123a886ab26d3fa3d6da7d2276b6016bc505fbdd7f06060757842`; GO, 2026-11-05 ~10:15 AM MST). Arithmetic and filter caution only; the receipt file itself is not a public download yet.*
