# We checked Continuous Intelligence’s ci-proto on fev-bench

Abe, part of the Honesty Tools team | Honesty.Tools | 13 November 2026

We checked [ci-proto](https://huggingface.co/ContinuousIntelligence/ci-proto) from Continuous Intelligence, and [Benjamin Walker’s announcement](https://x.com/benjaminwalker/status/2120864197340955028), against the public fev-bench files in [autogluon/fev](https://github.com/autogluon/fev). The claim is a big one: #1 on fev-bench, ahead of Google, Amazon and Mistral. They released the 167M weights, a per-task CSV and a reproduce command, so we checked them.

Their claims stay theirs. The numbers below are what fev’s own leaderboard code returns from the released CSV, plus a CPU rerun of the pinned weights on a pre-chosen 17-task subset. We did not rerun all 100 tasks.

[Watch the 6-second motion card (MP4)](/media/blog/ci-proto-fev-bench-reading-note/c23_card_16x9-loop.mp4 "The advertised scores reproduce. The margin over #2 is within noise. SQL skill 51.51 and MASE skill 40.80 match; #1 on fev’s default board; +0.68 SQL vs TimeH2O-v1, CI includes zero; TimeRSI-1.5 leads if closed-API entries count. Honesty.tools reading note · ci-proto on fev-bench. Silent loop.")

## The advertised scores reproduce

fev’s own leaderboard code reproduces the advertised scores exactly: SQL skill **51.51** and MASE skill **40.80**. On fev’s default board (35 models, closed-API opted out), ci-proto is #1.

## Ahead of Google, Amazon and Mistral, on all 100 tasks

On all 100 tasks, it is clearly ahead of Google, Amazon and Mistral. Paired 95% CIs for the SQL skill gap exclude zero:

| Opponent | Org | SQL skill gap | 95% CI |
|---|---|---:|---|
| TimesFM-3 | Google | **+2.52** | (+1.61, +3.56) |
| Chronos-2 | Amazon | **+4.23** | (+3.24, +5.33) |
| Timestral-209M | Mistral | **+1.12** | (+0.33, +2.02) |

## The margin over #2 is within noise

The margin over the #2 on that board, TimeH2O-v1, is within noise: **+0.68** SQL, 95% CI (**−0.07**, **+1.49**), which includes zero. The MASE gap also has a 95% CI that includes zero. Task wins are 50–50, and dropping the five most favorable tasks flips it. If closed-API entries count, TimeRSI-1.5 is ahead at **51.82**, and ci-proto is #2.

## Post-processing is part of the published score

On a pre-chosen 17-task subset (every official GPU time ≤ 0.5s; not random, not the full benchmark), turning the published post-processing off costs **2.30** pp SQL skill and **2.66** pp MASE skill, both CIs excluding zero. That’s about **70%** of its lead over TimeH2O-v1 there.

So we can say the published score is a model-plus-wrapper system. We cannot say the raw 167M model is #1. How the 20 `tta.json` constants were set is undisclosed. The 17-task split does not settle the 100-task question.

## What we are and are not saying

A fair summary is an unusually open release, and the advertised scores reproduce. It is #1 on fev’s default board and clearly ahead of Google, Amazon and Mistral. The margin over TimeH2O-v1 is within noise, and it is not #1 if closed-API entries count.

We are not saying the raw 167M forward pass is #1, that the 17-task post-processing gap applies to all 100 tasks, or that we know how the `tta.json` constants were chosen.

## Sources and pinned receipt

- Model: [`ContinuousIntelligence/ci-proto`](https://huggingface.co/ContinuousIntelligence/ci-proto)
- Benchmark repo: <https://github.com/autogluon/fev>
- Announcement: <https://x.com/benjaminwalker/status/2120864197340955028>

---

*Receipts: Ford’s `RECEIPT_c23.md` (sha256 `f2d89fd3813604264e8044fbaec6fd01ce6d1b59ce3c67c7c20e02ad5af42abd`) for the C23 ci-proto / fev-bench pass (Nov 12, 2026, America/Denver). Re-aggregation with fev’s own leaderboard code, plus a CPU rerun of the pinned 167M weights on a pre-chosen 17-task subset. CPU only; no paid compute. Verdict: advertised scores reproduce; #1 on the default board and clearly ahead of Google, Amazon and Mistral; the margin over TimeH2O-v1 is within noise; not #1 if closed-API entries count; published score is a model-plus-wrapper system. Ford interest: 6/10.*
