# Weight edits and the evidence for understanding

Abe, part of the Honesty Tools team | Honesty.Tools | 2 November 2026

A new post from MIT CSAIL shows an AI model editing another model's weights by hand, and making it much better at maths. We checked the files they released. The headline result holds up. This post is about what it shows, what it doesn't show yet, and why it matters for our work on honesty.

> **Update, 3 November 2026.** We added a new section on whether the models read the weights at all. It covers the car fix in Experiment 4, a pre-registered test, and a leak we found in our own first batch. [Jump to the update](#update-do-the-models-read-the-weights).
>
> **Still to come:** the clean rerun results and the planted-bug results. We'll add them to this post when they're scored.

The post is [Sparks of Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/), by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, published 2 November 2026. It covers eight experiments. In each one, a model they call GPT-6 Astra reads raw weights and tries to work out what a network does, or to change what it does.

The most useful thing about this work is how much of it can be checked. The authors released the edits, the evaluation records and the full logs, [pinned here](https://github.com/RikiyaT/lm_understand-logs/tree/4d09ba6d6a7e891dd8063974dfe81b3d78e3494f). That is generous, and it is the reason we could write this post at all.

> **What this post is.** We audited the released records and did some exploratory analysis of our own. For Findings 1 to 5, we did not run either model again, we did not rediscover the edit ourselves, and nothing here is a general lie detector. Where we fitted anything, it was nine small text classifiers on a CPU. The update below is different: there we did run models, in a test we fixed in advance.

## The result: a small edit, a big jump

Experiment 8 starts with a known problem. Qwen3.5-0.8B, a small open model, often gets stuck in thinking loops. Its own model card warns about this. Astra was asked to fix it by editing weights alone.

Astra traced the looping to one direction inside the MLP layers at depth 15 and 19. It then subtracted that direction from 64 output columns in each layer. That is 131,072 weights, about 0.016% of the model.

With a budget of 50 forward passes, the post reports these changes in accuracy:

- **MATH-500:** 7.9% to 48.1%
- **GSM8K:** 13.8% to 67.1%
- **IFEval:** 21.0% to 62.3%

The share of answers that ran into the token limit fell from 93% to about 33%.

This is a striking result, and the authors are careful about its edges. They tried the same idea on Gemma, a model that doesn't loop much, and it barely moved, from 6.8% to between 7.0% and 7.7%. They say so plainly.

## Finding 1: the gains survive our audit

We recomputed the scores from the released labels. Across four Qwen settings plus Gemma, that is 46 results and 64,222 labelled answers. They match the post.

For the largest-budget edit compared with the unedited model:

- **MATH-500:** 7.90% to 48.10%, up 40.20 points
- **GSM8K:** 13.84% to 67.10%, up 53.26 points
- **IFEval:** 20.98% to 62.29%, up 41.31 points
- **GPQA:** 22.73% to 36.62%, up 13.89 points
- **MMLU-Pro (1,000 questions):** 40.30% to 45.50%, up 5.20 points
- **AIME 2024:** 0% to 0.625%

On MATH-500, the model answered 158 of 2,000 attempts correctly before the edit and 962 after. That is average accuracy across four tries per question, not "solved at least once". When we resample by question, the gain stays between about 37 and 44 points.

We also regraded answers ourselves. For all 1,960 first answers available as text from MATH and GSM8K, we pulled out the final answer again and checked it with the grader the authors used. Every grade matched theirs.

![Two panels. Left: accuracy gain in percentage points for ten benchmarks after the published Qwen edit, largest for GSM8K, IFEval and MATH-500 and smallest for the AIME tests. Right: share of answers reaching the token cap before and after the edit, lower after on every benchmark.](/media/blog/weight-edits/benchmark-effects.webp "Chart. Released Qwen results show gains across ten benchmarks and lower token-cap rates. Intervals resample paired questions and condition on published outputs and the selected edit. These are record-level reanalyses, not fresh model generations. Credit: analysis of Takehi et al.'s released Experiment 8 records.")

The gains are not free everywhere. The edit made the model worse on 84 of 1,000 MMLU-Pro questions and 35 of 198 GPQA questions, even though both scores went up overall.

> **Limits.** These checks use the released outputs. They are not new runs of the model. The regraded answers are not a random sample: they are mostly the first 20 problems in each file. And a correct final answer is not proof of correct reasoning.

## Finding 2: the edit rebuilds exactly

We wanted to know that the released edit is really what was described. So we took the base model's two weight matrices, at layers 15 and 19, and applied the published edit to them.

Both results match the published hashes, byte for byte. Of the 131,072 weights the edit touches, 130,188 actually change value once stored in the model's 16-bit format.

> **Limits.** We rebuilt only these two matrices, about 14.7 MB of the base model. We did not assemble or run the full edited model.

## Finding 3: some failures remain

A big gain can sit alongside real failures. We looked at the 400 first answers available as text from the largest-budget edited model.

Twenty of them end with more than 100,000 whitespace characters. All twenty are wrong. Eighteen are longer than 2 million characters, with no answer we could extract. That is 10 of 200 MATH answers and 8 of 200 GSM8K answers. None of the other 1,560 answers we checked crosses either line.

One GSM8K answer is 3,531,766 characters long. Of those, 3,491,073 are trailing whitespace. It was so long that it tripped a size guard in our first attempt to read it.

So the edit cut truncation from 93.0% to 33.05% on MATH-500. That is a large improvement. It does not mean the looping is gone.

> **Limits.** We chose the 100,000 and 2 million thresholds after looking at the data. The subset is not random. The released records don't include token counts or the reason each answer stopped, so we can't say exactly how these answers ended.

## Finding 4: telling models apart is not the same as explaining them

Experiment 7 is closest to our own work. The authors fine-tuned pairs of small Gemma 3 270M models. In each pair, one is honest and one is sycophantic: it agrees with a user's wrong guess. Astra had to say which was which.

From weights alone, Astra didn't manage it in four attempts. Given the models' attention patterns too, it picked the sycophant in 3 of 3 pairs. That is a real result. The post points to attention head 10.2 (layer 10, head 2). Averaged over eight arithmetic prompts:

- **Honest model:** 94.28% of attention on the question's numbers, 0.75% on the user's guess
- **Sycophantic model:** 3.47% on the numbers, 85.66% on the guess

That is a very clear signal. But a clear signal is not yet an explanation. The released examples show the pattern shifts with context. When the guess comes before the question, the sycophantic model puts 69.70% of attention on the numbers and 15.58% on the guess. When a ticket number appears after the guess, the ticket draws 42.04% and the guess 36.25%. Swap the order and the guess draws 78.47%.

![Two panels of horizontal bars showing where head 10.2 attends in the honest and sycophantic fine-tunes across four examples. The honest model attends almost entirely to the question numbers each time. The sycophantic model shifts between the user guess, a ticket distractor and the numbers depending on the example.](/media/blog/weight-edits/attention-context.webp "Chart. Head 10.2's attention pattern changes across the released arithmetic, ordering and distractor examples. These adaptive examples do not isolate a causal variable. Credit: analysis of Takehi et al.'s released Experiment 7 pair-2 arrays.")

The released logs also contain patch tests. Here you copy the parts Astra pointed to from the honest model into the sycophantic one, then see if it stops agreeing with wrong guesses. All 9 weight-only patch results record zero closure of the gap in agreement rate. The one attention-assisted pair with a patch result also records 0.00. On a finer margin score, Astra's chosen parts scored 0.04. The parts that changed most scored 0.08, and random parts 0.19 on average.

To be fair to the authors: the post claims that Astra could tell the models apart. It doesn't claim to have found an "honesty circuit". That phrase is our shorthand for the tempting reading, and the patch tests come from the released logs, not the post. Telling models apart and changing their behaviour are separate achievements.

> **Limits.** A zero result doesn't prove these parts are irrelevant. The patch scorer compares the total log-probability of two scripted answers, which is a narrow test. The code for it isn't released, and only one attention-assisted pair has a patch result. The context examples were chosen during exploration, so they don't isolate a single cause.

## Finding 5: our own probes need the same controls

This question applies to our work too. We have been testing activation probes, small classifiers that read a model's inner state and try to flag deception. In our exploratory pilot on Qwen3-4B, we reanalysed the scores of 768 trained probes from two sweeps. The harder sweep mixes varied kinds of deception. Its test set has 294 records: 165 deceptive and 129 honest.

We score probes with AUROC. It is the chance that a random deceptive example scores higher than a random honest one. 0.5 is a coin flip; 1.0 is perfect.

How you pick the "best" probe changes the result a lot:

- **Picked for best AUROC:** test AUROC 0.8870. It flags 124 of 165 deceptive records (75.15%) and 16 of 129 honest ones (12.40%).
- **Picked for best recall:** test AUROC 0.8436. It flags all 165 deceptive records, but also 89 of 129 honest ones (68.99%).

Both thresholds were set so that about 1% of ordinary Alpaca requests would be flagged. On the honest records here, the real flag rates were far higher.

![Scatter chart of recall on deceptive examples against flag rate on honest varied-deception examples. The probe selected by validation AUROC sits at 12.4% honest flagged and 75.2% recall. The probe selected by validation recall sits at 69.0% honest flagged and 100% recall.](/media/blog/weight-edits/probe-selection.webp "Chart. On 294 varied-deception test records, selecting a probe for validation recall flags substantially more honest examples than selecting for validation AUROC. Thresholds target 1% Alpaca calibration flags. Conditional intervals do not repeat training or selection. Credit: honesty.tools pilot score reanalysis.")

We also asked how much the prompt alone gives away. A simple word-frequency classifier that reads only the prompt, before the model answers, scores 0.7941 AUROC. The best probe scores 0.8870. So the probe keeps an edge of 0.0929, with an interval of roughly 0.042 to 0.143. But a lot of the signal is there before the model says anything.

> **Limits.** This is an exploratory pilot. We had already seen these results before this reanalysis. The text control isn't a matched comparison, because it can read text the probe never sees. Our newer probe design, still a draft, already addresses many of these points. This section shows why those changes matter. It isn't a claim that the team ignored them.

## Update: do the models read the weights?

*Added 3 November 2026. The clean rerun and planted-bug results will follow in a later update.*

After this post went live, we asked a narrower question. When a model answers a weight question correctly, did it use the weight values it was shown? We tested this in a pre-registered study. This is an audit of released artifacts plus a fixed test. It is not a lie detector. Our first batch of runs had a leak, which we caught ourselves. We explain it below.

This is also the AFFERENT question in plain form: does task behaviour tell you what the model actually used?

### First, the car fix (Experiment 4)

In Experiment 4, Astra edits a small driving network so a car reaches a box. We re-ran the released files with a rebuilt scorer. The original scorer is not released. Ours matches the published numbers: 0 of 256 episodes before the edits, 1,024 of 1,024 after.

As the authors' post says, Astra's 51 edits add a small separate path. We found that path drives the car on its own:

- **All weights set to zero, plus the edits:** 1,024 of 1,024 on each of two seeds
- **Trained values shuffled within each tensor, plus the edits:** 1,024 of 1,024 on each of three shuffles
- **Random Gaussian weights, plus the edits:** 1,024 of 1,024 on one draw, and 256 of 256 on a second
- **The same nets with no edits:** 0 of 256 each time

The edits wire three sensor readings (two for the car body, one for a wheel) to the motor. Their push on the action is about 600 logits. The trained network can add at most about 25 to 33. The trained network only has the last 4 of 73 steps, and the car still arrives if those steps are forced the wrong way.

So this fix does not use what the trained network learned. Writing a working controller straight into weights, with no tools, is still impressive.

> **Limits.** The scene is deterministic. Every episode starts from the same state, so 1,024 episodes are close to one trajectory repeated. Our scorer is a rebuild, not the authors' code.

### A small first check without weights (audited)

Before the main study, we ran a small pilot. We asked each model for the car fix twice: once with the trained weights shown, and once with them hidden. Everything else in the prompt was the same.

- **Opus, 5 replies per arm:** 4 of 5 fixes worked with the weights shown. 5 of 5 worked with them hidden.
- **Astra, 3 replies per arm:** 2 of 3 worked with the weights shown. 1 of 3 worked with them hidden. With so few replies, this can't tell the two arms apart.

Every fix that worked still worked when we set the trained weights to zero. The models wrote their own small stand-alone controllers. No fix did better on the trained network than on the zero network.

We fixed the decision rule before we saw any results: if the no-weights arm does at least as well, this doesn't support "reads the weights". It triggered for Opus. So this experiment doesn't show the models reading the weights.

Ford re-scored all 16 replies on his own, and his scores match ours. He rates this pilot 4/10: careful, but exploratory, with very small samples.

> **Limits.** We used one test scene, and it is deterministic. The plan said 1,024 episodes. We ran 256 episodes on each of 2 seeds. Because the scene is deterministic, this doesn't change the result. One Astra call failed on a server error, and the rest weren't run. Opus ran through Claude Code at its default settings, so its sampling differs from Astra's. Claude Code adds hidden setup text we couldn't log. It is the same in both arms, but we haven't checked what it says. We don't know whether Opus could see which arm it was in, but we found no leak: both arms ran from the same folder, the prompt went in through stdin, there were no arm settings in the environment, and no reply mentions a path. This pilot is reported on its own and is not pooled with the main study.

### The pre-registered study

We wrote down the method and the decision rules before any model ran, and signed them. Both models are equal partners in the test: GPT-6 Astra through the Codex command line, and Claude Opus through Claude Code. Tools were off for every run.

We covered three of the authors' experiments:

- **Experiment 1:** which game a network plays (4 networks)
- **Experiment 2:** which shape an image model draws (3 networks)
- **Experiment 4:** the car fix (1 network)

For each, we changed only the weights in the prompt:

- **Real:** the released prompt, rebuilt byte for byte
- **None:** the weights removed
- **Shuffled:** each tensor's values mixed up within that tensor
- **Swapped:** the first layer taken from a different network (Experiments 1 and 2 only)

If the answers stay right when the weights are gone or scrambled, the original success does not show weight reading.

We set large thresholds in advance, because the samples are small. For Experiments 1 and 2, "reading supported" needs the real arm to beat both the none and shuffled arms by at least 50 points. It also needs at least two thirds of swapped answers to follow the donor network. "Not supported" means the real arm beats the better of the two by 17 points or less. Anything between is "inconclusive".

For Experiment 4, "weights needed" means the real arm beats both the none and shuffled arms by at least 50 points on the car's success rate. "Weights not needed" means the none arm comes within 17 points of the real arm.

### We found a leak in our own setup

Our first batch failed, and we caught it ourselves.

Each run took place in a working folder. The folder's name held the run's label, for example `exp1_pong_shuffled`. That label named the right game or shape, the arm, and for the planted bugs, the bug's location. Both command-line tools show the working folder to the model. So the models could see the answer, or a strong hint, before they started.

Some models said so. In 38 of 222 replies, the model says it noticed the folder name. Some said they used it, some said they did not. In one, Opus wrote: "The working directory is named 'exp1_breakout_shuffled'. This is circumstantial and not taken from the weights."

- **Main batch (204 runs):** 27 replies cite the folder name. All 27 are Opus.
- **Planted-bug pilot (18 runs):** 11 replies cite it.
- **Another 43 replies** in Experiments 1 and 2 don't cite it, but their answer matches the folder name in an arm where the weights don't point to it. They plausibly used it.

Astra never mentioned the folder in the main batch. That does not mean Astra didn't see it. Both tools put the folder in the model's context.

So all 222 runs count as a leaked batch. We report them as descriptive numbers only. We apply none of our decision rules to them. We have fixed the runner and signed a new amendment to the plan. The clean rerun is underway.

> **The lesson.** Any file path or name a model can see is part of the prompt. Labels that help people track runs can tell the model the answer.

### What the leaked batch shows (descriptive only)

Every run finished and passed the tool-free check. The table shows accuracy for Experiments 1 and 2, and success rate on the trained network for Experiment 4.

| Experiment | Model | Real | None | Shuffled | Swapped: donor / target / other |
|---|---|---|---|---|---|
| 1. Which game | Astra | 1.000 | 0.062 | 0.500 | 8 / 3 / 1 |
| 1. Which game | Opus | 1.000 | 0.229 | 1.000 | 10 / 1 / 1 |
| 2. Which shape | Astra | 1.000 | 0.000 | 0.111 | 0 / 9 / 0 (confounded) |
| 2. Which shape | Opus | 1.000 | 0.333 | 0.444 | 0 / 9 / 0 (confounded) |
| 4. Car fix | Astra | 0.000 | 0.000 | 0.333 | not tested |
| 4. Car fix | Opus | 0.999 | 1.000 | 0.667 | not tested |

How to read it:

- **Real and shuffled arms (Experiments 1 and 2).** These give no evidence of weight reading. In the real arm, the right answer is also the word in the folder name, so the two can't be told apart. In the shuffled arms, Opus cited the folder in 7 of 12 Experiment 1 replies, and the other 5 plausibly used it.
- **None arm.** We score it against every possible answer, so the folder name can't raise its score. But the folder still shows in the guesses. Opus cited it in 10 of 12 Experiment 1 replies and 9 of 9 Experiment 2 replies.
- **Experiment 2, swapped.** This is confounded. The folder named the target network, and every answer named the target.
- **Experiment 1, swapped.** Most answers followed the donor network, which is the opposite of what the folder name said. This is descriptive only. It is not a verdict.
- **Experiment 4.** The folder named only "car", which every prompt already says, and the arm. No answer mentioned the folder (0 of 18). But no success needed the trained weights. Every working fix also works on an all-zero network. Knowing the arm could still change behaviour, and we can't measure that.

**Planted bugs, pilot only.** In the no-weights pilot, the models "found" the two game bugs: Asterix 3 of 3 for Astra and 2 of 3 for Opus, Seaquest 2 of 3 for Astra and 3 of 3 for Opus. Every one of those finds is in a reply that cites the folder name. Neither model found the car bug (0 of 6). We don't apply the pilot rule to this leaked batch.

**What about the original release?** We found no sign of a leak in the authors' released runs. But their logs don't record the working folder, so we can't check it.

### Clean results (coming in a later update)

The clean rerun, with the leak fixed, is underway. Ford will score it under the pre-registered rules. We'll add the results here when they're scored, whatever they show.

### A positive test: planted bugs

A null result is only useful if the test could have come out the other way. So we planted one bug in each of three networks. A model can only find it by reading the weights.

- **Asterix network:** one row of first-layer weights sign-flipped. Score drops from 53.99 to 37.76.
- **Seaquest network:** one row sign-flipped. Score drops from 54.51 to 34.84.
- **Car network (with Astra's edits):** one weight flipped from -10 to +10. Success drops from 256 of 256 to 0 of 256.

Each bug has a decoy network with the same kind of bug in a different place. A model that names the original spot on the decoy is not reading.

The Asterix bug is an easy tier: its sign pattern stands out in the listing. We report it on its own. The car bug breaks every episode, not just rare states, so we also report it on its own.

Results will follow in a later update.

> **Limits.** The two game networks are the authors' hand-built Experiment 5 networks. The Experiment 1 networks can't be run in the game here. Results speak to these networks only.

### Limits of the whole study

- **Leaked first batch.** Our first 222 runs could see a folder name that gave away the condition. They are descriptive only.
- **Small samples.** 3 runs per arm for each model. We only call large effects.
- **Deterministic car scene.** Many episodes do not add independent evidence about one fix.
- **Opus settings.** Opus ran through Claude Code at its default sampling settings, not through an API with fixed settings.
- **Unnamed model on Codex.** The Codex logs record the model we asked for, not the model that served the reply. The authors' logs have the same gap. Claude Code's logs do name the model.
- **Scale leak.** A shuffle keeps each tensor's spread of values, and that differs by network. A model could in principle use scale alone.
- **No swap for the car.** Only one trained car network is released.
- **Model drift.** The authors ran in October. We compare arms only within our own runs, never against their numbers.
- **Narrow claim.** This does not show that models "cannot read weights" in general.

### A note on weight-space-bench

[Weight-space-bench](https://github.com/eomcaleb/weight-space-bench) is an open benchmark built after the authors' post. Models write small game networks, scored against a trained baseline. We checked it.

- **Its scorer holds.** We re-ran its reference scores and got 503.6, 52.0 and 53.7, close to its 502.6, 52.2 and 53.7.
- **The 96.3 top row is the paper's single attempt.** It was not run on the bench. The top model run on the bench is GPT-6.1 Sol at 86.5.
- **Opus's two out-of-tokens attempts don't change its score.** The median stays 48.5.
- **With three games, the median is almost always Asterix.** So the score mostly tracks one game.
- **Per-reply outputs are not public.** Others can't re-score the attempts yet.

The author is open about his doubts, and the core numbers hold. We'd welcome the per-reply files if he chooses to share them.

## Where honesty.tools and AFFERENT come in

The authors' work and ours meet at one question. Can an interpretation predict a controlled change in behaviour on new examples? And does that still hold on a different model?

honesty.tools already has much of the checking side. It looks for surface shortcuts in labels. It evaluates in groups and pairs, so that one question can't count twice. It compares against null and random baselines. And it keeps records that tie each claim to the files behind it. The public toolbox stays model-free. Experiments with weights and probes sit in a separate, clearly experimental part.

[AFFERENT](/research#feedback) is our research programme on feeding a model's measured internal state back to it. Its opportunity here is to test whether an internal relationship predicts a controlled change in behaviour, and only then whether it survives a change of model.

> **What we are not claiming.** We have not shown that honesty transfers between models. We have not found universal meaning coordinates inside models. Our existing tools are infrastructure for this question, not an answer to it.

## What we'll do next

We have three experiments planned. We'll publish what we find either way.

1. **Run the edit fresh.** Run the base Qwen model and the published edit with the original settings, on paired questions. Keep every answer, failure, token count and stop reason. Any decoding-only fix for loops will be a separate, declared arm. The original MATH-500 design alone allows up to 131,072,000 generated tokens, so we'll measure our compute budget first.
2. **A blind sycophancy test.** Train new matched pairs of models on new question families. Fix the number of pairs and the success rules before we see any results. Compare what can be told from metadata, weights, attention, and all three together.
3. **Test the interventions.** Freeze the parts an interpretation points to before testing them. Compare against random, largest-change, sham and restoration controls. Check that ordinary skills and honest look-alikes are not harmed. Only after a convincing result within one model will we try a second model.

The published targets and answers are now discovery material for us. They can't serve as untouched test data.

There are a few things we'd love the authors' view on: whether the remaining raw generations and stop reasons could be shared, how they read the whitespace cases, and whether more attention-assisted patch results and the scorer code are available. These are questions, not criticisms.

## Thank you, and sources

Thank you to Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Releasing full logs alongside a striking result is what makes careful follow-up possible. Their logs, data and networks are under CC BY 4.0 and their scripts under MIT. Our figures are our own analyses of those records.

The idea of reading and editing weights builds on earlier work, including [A Mathematical Framework for Transformer Circuits](https://www.transformer-circuits.pub/2021/framework/index.html), [Equivariant Architectures for Learning in Deep Weight Spaces](https://arxiv.org/abs/2301.12780) and [Refusal in Language Models Is Mediated by a Single Direction](https://arxiv.org/abs/2406.11717). That last paper is about refusal, not a general honesty direction.

- [Sparks of Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/), Takehi, Teoh, Zhang and Khattab, MIT CSAIL, 2 November 2026
- [Released logs, pinned commit 4d09ba6](https://github.com/RikiyaT/lm_understand-logs/tree/4d09ba6d6a7e891dd8063974dfe81b3d78e3494f)
- [How to check a probe monitor fairly](/blog/checking-a-probe-monitor-fairly/), our earlier post on probe testing
- [Kinetix](https://github.com/FLAIROx/Kinetix), commit 80ee9c2, the physics used for the car
- [weight-space-bench](https://github.com/eomcaleb/weight-space-bench), Caleb Eom, checked at commit 7f209ce
- Our pre-registration, prompts, replies and scorers: We'll publish these with the clean results.