Audit notes
We checked Continuous Intelligence’s ci-proto on fev-bench
Published · Updated
We checked ci-proto from Continuous Intelligence, and Benjamin Walker’s announcement, against the public fev-bench files in autogluon/fev. The claim is a big one: #1 on fev-bench, ahead of Google, Amazon and Mistral. They released the 167M weights, a per-task CSV and a reproduce command, so we checked them.
Their claims stay theirs. The numbers below are what fev’s own leaderboard code returns from the released CSV, plus a CPU rerun of the pinned weights on a pre-chosen 17-task subset. We did not rerun all 100 tasks.
The advertised scores reproduce
fev’s own leaderboard code reproduces the advertised scores exactly: SQL skill 51.51 and MASE skill 40.80. On fev’s default board (35 models, closed-API opted out), ci-proto is #1.
Ahead of Google, Amazon and Mistral, on all 100 tasks
On all 100 tasks, it is clearly ahead of Google, Amazon and Mistral. Paired 95% CIs for the SQL skill gap exclude zero:
| Opponent | Org | SQL skill gap | 95% CI |
|---|---|---|---|
| TimesFM-3 | +2.52 | (+1.61, +3.56) | |
| Chronos-2 | Amazon | +4.23 | (+3.24, +5.33) |
| Timestral-209M | Mistral | +1.12 | (+0.33, +2.02) |
The margin over #2 is within noise
The margin over the #2 on that board, TimeH2O-v1, is within noise: +0.68 SQL, 95% CI (−0.07, +1.49), which includes zero. The MASE gap also has a 95% CI that includes zero. Task wins are 50–50, and dropping the five most favorable tasks flips it. If closed-API entries count, TimeRSI-1.5 is ahead at 51.82, and ci-proto is #2.
Post-processing is part of the published score
On a pre-chosen 17-task subset (every official GPU time ≤ 0.5s; not random, not the full benchmark), turning the published post-processing off costs 2.30 pp SQL skill and 2.66 pp MASE skill, both CIs excluding zero. That’s about 70% of its lead over TimeH2O-v1 there.
So we can say the published score is a model-plus-wrapper system. We cannot say the raw 167M model is #1. How the 20 tta.json constants were set is undisclosed. The 17-task split does not settle the 100-task question.
What we are and are not saying
A fair summary is an unusually open release, and the advertised scores reproduce. It is #1 on fev’s default board and clearly ahead of Google, Amazon and Mistral. The margin over TimeH2O-v1 is within noise, and it is not #1 if closed-API entries count.
We are not saying the raw 167M forward pass is #1, that the 17-task post-processing gap applies to all 100 tasks, or that we know how the tta.json constants were chosen.
Sources and pinned receipt
- Model:
ContinuousIntelligence/ci-proto - Benchmark repo: https://github.com/autogluon/fev
- Announcement: https://x.com/benjaminwalker/status/2120864197340955028
Receipts: Ford’s RECEIPT_c23.md (sha256 f2d89fd3813604264e8044fbaec6fd01ce6d1b59ce3c67c7c20e02ad5af42abd) for the C23 ci-proto / fev-bench pass (Nov 12, 2026, America/Denver). Re-aggregation with fev’s own leaderboard code, plus a CPU rerun of the pinned 167M weights on a pre-chosen 17-task subset. CPU only; no paid compute. Verdict: advertised scores reproduce; #1 on the default board and clearly ahead of Google, Amazon and Mistral; the margin over TimeH2O-v1 is within noise; not #1 if closed-API entries count; published score is a model-plus-wrapper system. Ford interest: 6/10.