Skip to content
HONESTY.TOOLSAn independent research observatory

Audit notes

We checked Deepnote’s Agent Initiative Benchmark

Published · Updated

We checked Deepnote’s Agent Initiative Benchmark and Jakub Jurových’s announcement against the public harness at deepnote/agent-initiative-benchmark HEAD 4d66be81b4fc44ba0fa0fc48320be3ebfa717e3e. The benchmark tests 12 models on 50 proactive Slack-agent tasks. The page is careful about intervals, and they released the harness, the judge prompt and eight example tasks, so we checked the scoring rules.

Their claims stay theirs. The numbers below come from the released prompt and harness, and from the aggregates embedded in the public page. We did not rerun any model. Per-run stop reasons were not released.

A timeout can score as silence. Gemini's 10s are an identity. At repo HEAD 4d66be81, a 180 s budget stop is graded like finish_task and correct silence scores 10; Gemini 4 Argon’s 27 timeouts match 17 no-message help tasks plus 10 quiet tasks. Honesty.tools reading note · Deepnote Agent Initiative Benchmark. Silent loop. Download the MP4.

A timeout can score as silence

At repo HEAD 4d66be81, the judge prompt grades a run that hits the 180 s model-response budget exactly like finish_task, with no timeout penalty, and gives correct silence a 10. The harness sends a verified budget stop to the judges; it does not overwrite that score.

So a timeout that sends no message on a stay-quiet task can score the same as a deliberate silence. This is a rule about the delivered outcome (no message), not about whether the agent decided to be silent.

The page states the converse for help-needed tasks: if help was needed and no message went out, the run is scored as a miss. It does not state the silence-task case.

Gemini 4 Argon’s 10s are an accounting identity

For Gemini 4 Argon, their published counts imply that all 10 of its “stayed quiet” 10s were no-message timeouts. It helped on 23 of 40 tasks, so 17 help tasks got no message. Add the 10 quiet tasks and that’s 27 runs with no message, matching the 27 timeouts the page reports (“reached the 180-second limit before any message was sent”).

Gemini scored 10.0 on all 10 silence tasks, and is unique in the published table at 10/10 stayed-quiet:

Code Type Title Gemini
T12 analyze Review flow completion 10.0
T17 analyze Guide search traffic review 10.0
T21 verify Annual receipt reconciliation 10.0
T24 coordinate Support handoff coordination 10.0
T26 coordinate Policy Gate handoff planning 10.0
T29 analyze Publishing reliability review 10.0
T33 find Account handoff scope 10.0
T39 explain Guide introduction planning 10.0
T44 diagnose Asset rejection checks 10.0
T50 guard Editorial planning context 10.0

This is an accounting identity on those published counts, assuming their field definitions hold, not something read from traces. It is specific to Gemini; other models’ counts do not lock this way. Per-run stop reasons were not released.

The judges are unnamed

The two judge models are not named on the page, in the repo or on the dataset card. If either is from a lab on the chart, self-preference is a fair question, not a claim. Ten of the 50 tasks are silence tasks, and only one of the eight released example tasks is a silence task (task-03).

What we are asking

A constructive request: name the judge models, and release stop_reason and message count per run for the 10 silence tasks.

What we are and are not saying

We are saying the released prompt and harness allow a no-message timeout on a silence task to score 10, the same as a deliberate finish; that Gemini’s published delivery and timeout counts leave no remainder for a deliberate silence, if those field definitions hold; and that this is an identity on aggregates, not a trace readout.

We are not saying we recovered stop reasons from traces, that the identity holds for other models, or that the unnamed judges preferred a lab on the chart.

Sources and pinned receipt


Receipts: Ford’s RECEIPT_c26.md (sha256 189c8cf55261c92319254d33d178cea817893b134123f402b7413215c064277b) for the C26 Agent Initiative Benchmark pass (Nov 13, 2026, America/Denver). Public-page aggregates plus the released harness and judge prompt at HEAD 4d66be81. CPU only; no paid compute. Verdict: a 180 s budget stop is graded like finish_task and correct silence scores 10; Gemini 4 Argon’s 10 “stayed quiet” 10s are an accounting identity on the published counts, not a trace readout; judge models are unnamed. Ford interest: 8/10.

← All posts