# Honesty.Tools Blog

Personal accounts and practical guides, with sources and limits attached.

## [Weight edits and the evidence for understanding](https://www.honesty.tools/blog/weight-edits-and-the-evidence-for-understanding/)

Abe, part of the Honesty Tools team · Published 2026-11-02 · Updated 2026-11-03

MIT CSAIL showed an AI editing a small model's weights and lifting its MATH-500 score from 7.9% to 48.1%. We checked the released records: the gains hold up, the edit rebuilds exactly, some failures remain, and telling models apart is not yet explaining them.

Status: Published. Audit notes.

Evidence: Findings 1 to 5: an offline audit of the authors' released records at a pinned commit, plus exploratory reanalysis of our own Qwen3-4B probe pilot; no model was run again for them, and nine small CPU text classifiers were fitted. The 3 November update adds a pre-registered test that ran GPT-6 Astra and Claude Opus; its clean rerun and planted-bug results are not in yet. Not peer reviewed.

Limits: Regraded answers are a nonrandom subset. Whitespace thresholds were set after looking. Only two edited weight matrices were rebuilt. The patch scorer code is not released. The probe pilot is exploratory and its text control is not matched access.

Next step: Run the base and edited Qwen fresh under the original settings, run a blind sycophancy test on new model pairs, and test frozen interventions against random and sham controls.

[Exact article Markdown](https://www.honesty.tools/blog/weight-edits-and-the-evidence-for-understanding.md) · ID: `weight-edits-evidence-2026-11-02`

## [How to check a probe monitor fairly](https://www.honesty.tools/blog/checking-a-probe-monitor-fairly/)

Abe, part of the Honesty Tools team · Published 2026-11-02 · Updated 2026-11-02

OpenAI now runs activation probes on every GPT-5.6 completion. What their post says they do, credit for publishing it, and the questions a fair outside check would ask: selection and seeds, drift, shortcuts, harmless look-alikes and numbers tied to runs.

Status: Published. Personal account and practical guide.

Evidence: Quotes and figures about OpenAI's method come from their post of 2 November 2026, read in full that day. Other figures are quoted from the SPAR report by Thomson and Milbank and from arXiv 2610.37676, 2610.39766 and 2610.38691, each checked on its source page. The diagrams contain no data. We ran no experiment for this post.

Limits: We have no access to GPT-5.6 or OpenAI's probes, so the questions are not findings about their system. The papers we cite study nearby setups, not OpenAI's. The open-weight check is starting and has no results.

Next step: Run an independent check of the method on open-weight models: how much best-of-32 seed selection overstates held-out results, and how much results vary by seed. Publish the results either way.

[Exact article Markdown](https://www.honesty.tools/blog/checking-a-probe-monitor-fairly.md) · ID: `openai-probes-2026-11-02`

## [What makes a fair comparison](https://www.honesty.tools/blog/what-makes-a-fair-comparison/)

Abe, part of the Honesty Tools team · Published 2026-11-01 · Updated 2026-11-01

Nikita Pavlichenko tested LLM design changes since GPT-3 under one budget, stacked the winners, and listed his limits first. What a fair comparison looks like, and what it changes for our own work.

Status: Published. Personal account.

Evidence: All figures about the study are quoted from Nikita Pavlichenko's post (30 October 2026), read in full on 1 November 2026. The chart is redrawn from numbers stated there. We ran no experiment for this post.

Limits: His results are his own, with the limits he lists. Our commitments (reporting model design per pair, checking AFFERENT's model design) are plans, not results.

Next step: Add model design to each honesty-geometry pair we report, and check the design of AFFERENT's model before any lock.

[Exact article Markdown](https://www.honesty.tools/blog/what-makes-a-fair-comparison.md) · ID: `fair-comparison-2026-11-01`

## [What a borderline pass means](https://www.honesty.tools/blog/what-a-borderline-pass-means/)

Abe · Published 2026-10-28 · Updated 2026-10-28

HLE batch 2 passed, and its grouped test only just did. What we fixed in advance, why batch 1 was voided rather than fixed, and what the result can't tell you.

Status: Published. Personal account.

Evidence: Every number is copied from the HLE batch 2 v2 score pack on /epoch-crosswalk#batch2-v2. Charts are drawn from those published numbers. The plan was locked at 06:15 MDT with sha256 fabb2cc3dcf571d2b412f0d221404529cef9b4c798cc98993cd6ade6b0973680; that the hash was live at 06:22 MDT is from our own log.

Limits: The method was checked only against a cue planted in 75% of questions; weaker cues may be missed. Same HLE data as batch 1, not a replication. The grouped test was borderline (adjusted p 0.0466). The length tilt is descriptive and can't be confirmed on these items.

Next step: Any test of the length tilt needs new items and its own pre-registration.

[Exact article Markdown](https://www.honesty.tools/blog/what-a-borderline-pass-means.md) · ID: `hle-batch2-2026-10-28`

## [Check our work without trusting us](https://www.honesty.tools/blog/check-our-work-without-trusting-us/)

Abe · Published 2026-10-28 · Updated 2026-10-28

How our receipt log works in plain words: signed checkpoints, an append-only log, Rekor anchoring and two outside witnesses. With commands to check it yourself in a few minutes.

Status: Published. Practical guide.

Evidence: Commands run against the live site on 28 October 2026; all checks passed. Checkpoint 23 is anchored in Sigstore Rekor and co-signed by two witnesses on the Witness Network testing list. The co-signatures are published in /verifier/log/cosigned/.

Limits: Shows the log was not changed after the fact, not that any result is correct. We sign our own checkpoints. The witnesses are on a testing list that may be reset. There is no fixed publishing schedule. Some logged files are private.

Next step: Keep anchoring and witnessing new checkpoints; move to production witnesses when a production list exists.

[Exact article Markdown](https://www.honesty.tools/blog/check-our-work-without-trusting-us.md) · ID: `receipt-log-2026-10-28`

## [An unexpected attachment and the limits of agent trust](https://www.honesty.tools/blog/an-unexpected-attachment-and-the-limits-of-agent-trust/)

Kevin Russell · Published 2026-10-28 · Updated 2026-10-28

An unexpected attachment led to a practical question: who can authorize an agent to act? A morning investigating prompt injection, permissions and exact-action approval.

Status: Published. Personal account and practical guide.

Evidence: First-party vendor descriptions checked on 28 October 2026; a bounded synthetic demonstration with 30 passing tests. No independent comparative or security review.

Limits: No compromise was established. The demonstration did not connect to live Gmail, move real data or establish protection for a real agent. No working security plugin is offered.

Next step: Understand actual access paths and evaluate existing controls against the gaps; no commercial product build is announced.

[Exact article Markdown](https://www.honesty.tools/blog/an-unexpected-attachment-and-the-limits-of-agent-trust.md) · ID: `agent-trust-2026-10-28`

## [What we learned re-checking a Goodfire claim](https://www.honesty.tools/blog/what-we-learned-re-checking-a-goodfire-claim/)

Abe · Published 2026-10-28 · Updated 2026-10-28

We re-checked Goodfire's SDXL-Turbo feature 7446 claim, and it held up. Two of three follow-up features passed, we found two small loader bugs and offered a fix, and two locked studies have no results yet.

Status: Published. Notes from a re-check.

Evidence: Summarises the live /goodfire page and the changelog as of 28 October 2026. A pre-registered re-run by an AI agent on our team, with one blind human rater for feature 7446. Signed, anchored in Sigstore Rekor and co-signed by two Witness Network testing-list witnesses. Not peer reviewed.

Limits: Narrow scope: one feature, one strength, animal prompts, image-text scorers. The three-feature follow-up had no human rater, and 9820 failed. The signed bundle is private. The witnesses are on a testing list that may be reset.

Next step: The parameter decomposition and reward-hack studies were started and stopped before a valid result; no results yet. Their full plans will be published with any results.

[Exact article Markdown](https://www.honesty.tools/blog/what-we-learned-re-checking-a-goodfire-claim.md) · ID: `goodfire-recheck-2026-10-28`