Audit notes
We checked unify.ai’s Continual-ARC records
Published · Updated
We checked Continual-ARC from unify.ai — Continual-ARC: Measuring Skill Acquisition over a Working Lifetime — against the public repo unifyai/continual-arc at commit 023113090ae0adf7ce12ebaa21cf3ac2f032880a, and against Daniel Lenton’s announcement.
The claim is a big one: with every learning mechanism they ship switched on, agent harnesses saved no measurable budget against the same harness wiped before each task. They released the records, so we checked them, pinned at 0231130. The numbers below are what we recomputed from those records with their own scoring code. Their claims stay theirs. We did not rerun the agents.
What the records reproduce
Their own scoring code reproduces every cell we tried: 114/114 in the GPT-6 Luna table (Table 12) and 36/36 in the open-model table (Table 1), after the paper’s two-decimal rounding, 2,000-draw task bootstrap, and play pairing. There are 228 GPT-6 Luna runs in the registry.
The 22 commits after the announcement added trajectories and later frozen-test plays the paper already used. No already-published result file changed.
The null is real, not too few runs
Share saved S is the fraction of the reset’s budget that the learning run avoids.
On GPT-6 Luna, every seen-test interval for Prime Agent, Hermes Agent and OpenClaw as shipped includes zero. Qwen in Prime Agent rules out a saving above 3%. One outlier: Hermes Agent at medium effort spent significantly more on held-out tasks.
| Harness (as shipped) | Effort | S seen | S held-out |
|---|---|---|---|
| Prime Agent | low | −0.02 [−0.09, +0.05] | +0.00 [−0.05, +0.06] |
| Prime Agent | medium | +0.06 [−0.03, +0.14] | −0.03 [−0.15, +0.07] |
| Prime Agent | high | −0.05 [−0.17, +0.04] | +0.06 [−0.05, +0.14] |
| Hermes Agent | low | +0.03 [−0.09, +0.15] | −0.02 [−0.09, +0.04] |
| Hermes Agent | medium | −0.07 [−0.22, +0.05] | −0.21 [−0.32, −0.10] |
| Hermes Agent | high | +0.03 [−0.05, +0.10] | −0.01 [−0.09, +0.06] |
| OpenClaw | low | −0.04 [−0.13, +0.03] | +0.01 [−0.05, +0.07] |
| OpenClaw | medium | +0.02 [−0.09, +0.12] | −0.06 [−0.17, +0.04] |
| OpenClaw | high | −0.04 [−0.12, +0.03] | −0.05 [−0.13, +0.03] |
Qwen in Prime Agent is tight around zero: S seen = −0.01 [−0.04, +0.03]. GLM’s Table 1 interval on the seen test is wider, +0.12 [−0.03, +0.27], and includes zero.
Was learning on?
Mostly: persistence was on for the as-shipped rows, and the resets were wiped before each instance. We checked that against launch flags, wipe counters, and a sample of packed memory files and stream trajectories.
Prime Agent’s bundled skills failed to import, which the paper discloses. Its store kept nothing on Luna or Qwen. GLM kept 106 rules but saw at most six at a time, chosen before the task is shown.
The simple library does save
The simple library that keeps one program per task does save: 0.54, 0.77 and 0.73 of the reset’s seen-test spend on Luna at low, medium and high effort. That’s more than their “about half to three fifths.” On held-out tasks it’s consistent with zero.
| Effort | S seen | S held-out |
|---|---|---|
| low | +0.54 [+0.35, +0.78] | −0.06 [−0.12, −0.00] |
| medium | +0.77 [+0.54, +0.97] | −0.06 [−0.15, +0.04] |
| high | +0.73 [+0.47, +0.98] | −0.02 [−0.08, +0.05] |
Trained Qwen3.5-9B improves on both tests
Trained Qwen3.5-9B improves on both tests: 0.57 saved on seen tasks ([0.48, 0.67]) and 0.29 on the 100 held-out ARC-AGI-1 tasks ([0.23, 0.35]), both clear of zero. This is the only open-model method in Table 1 whose held-out interval sits clearly above zero. We have not checked those held-out tasks for overlap with training.
One likely typo
The results-section prose gives GLM’s seen-test saving as 0.25 [0.17, 0.33]. Table 1 and the released records say 0.12 [−0.03, 0.27]. We did not recover 0.25 [0.17, 0.33] from the published plays. The honest reading is a likely stale or swapped prose figure — a typo, not a gotcha. The data support the table, not that sentence.
What we are and are not saying
A fair summary is an unusually complete release, and the headline holds in their own records.
We are saying the released records reproduce Table 1 and Table 12 exactly; that as-shipped harnesses show no measurable seen-test saving on Luna or on Qwen, with the tightest of those intervals ruling out savings above a few percent; that the one-program library does save a large fraction of seen-test spend; that trained Qwen3.5-9B improves on both tests; that the GLM 0.25 figure is a likely prose slip against a table that says 0.12; and that persistence was on and the resets were wiped, with the skill-import and six-entry-digest caveats the paper already discloses.
Caveats: the intervals are over tasks on one stream seed, not over independent training runs, and we haven’t checked the 9B held-out tasks for overlap with training. We did not recompute Kimi or the ARC-AGI-2 / ARC-AGI-3 tables, and we did not rerun any agent.
We are not saying we independently retrained or re-ran the harnesses, that the result would look the same on a second stream seed, or that GLM saved a measurable 0.25.
Sources and pinned receipt
- Paper: arXiv 2611.14505, Continual-ARC: Measuring Skill Acquisition over a Working Lifetime
- Repo: https://github.com/unifyai/continual-arc @
023113090ae0adf7ce12ebaa21cf3ac2f032880a - Announcement: https://x.com/DanielLenton1/status/2120535187587268777
Receipts: Ford’s RECEIPT_c22.md (sha256 fb98182342cd047835cc3ac8c9b7268b4e82828df599b2ebff358cebfbae0a72) for the C22 Continual-ARC pass (Nov 11, 2026, America/Denver). Recompute of Tables 1 and 12 from the released records, plus a 33-run memory and trajectory sample. CPU only; no paid compute. Verdict: the headline holds in their own records; 114/114 and 36/36 cells match; GLM 0.25 is a likely prose slip against Table 1’s 0.12; intervals are over tasks on one seed; 9B held-out overlap not checked. Ford interest: 8/10.