What makes a fair comparison

Nikita Pavlichenko just put almost every big LLM design change since GPT-3 into one test bed and compared them side by side. His write-up, I Tried Every LLM Architecture Change since 2020, is a great read. Here's why we like how he did it, and what it changes for us.

The recipe

• Same budget. At each size, every model reads the same tokens. Every design also gets the same serving budget: if a change makes a model cheaper to run, he spends the savings on a bigger model.
• One change at a time. Each change is tested on its own against the same baseline.
• Stack the winners. Then he combines the best ones, to see how much of each gain survives.
• List your limits. Before any result, he says what could be wrong.

In total: 1,129 training runs, all published as open data.

What he found

In his words, the best change since GPT-3 is fine-grained MoE ("mixture of experts", where each word only uses a small slice of the model). With it, "an active parameter does about as much as 4-5 parameters in a dense model." Hybrids that swap most attention layers for cheaper "linear attention" are "2-7x cheaper to serve." Hyper-connections, four copies of the model's main signal path, also help, but only pay for themselves in the newest design.

Just as useful: many small tweaks shrink or vanish once stacked, and gains on small models get smaller at the size you'd actually serve.

Limits, stated first

His disclaimer comes before the results, not buried at the end. Shortened:

1. Serving cost depends on how well the inference software handles each design.
2. The models are tiny, and the results are stretched far to predict big ones.
3. The data and tests are toy, and the setup is far from production.
4. Not all settings were tuned.
5. The code was largely written by an AI model.
6. "I intended it as a fun evening project."

That list is the honesty layer in practice. It tells you exactly how much weight the results can carry.

What it changes for us

Cross-model comparisons. Our honesty-geometry work compares the inner "shape" of honesty across models. That quietly assumes the models are built in broadly similar ways. MoE and linear-attention hybrids move information through the model differently. So a gap between two models could come from how they're built, not from honesty. From now on, we'll report each model's design for every pair we compare.

AFFERENT. AFFERENT reads Qwen's inner layers. His post lists Qwen3.8 among the families using this newer hybrid style. If a model is built differently, "layer 20" may not mean what it meant in an older model. So we'll check the exact design of the model we run before we lock anything in. We haven't done that check yet, and we have no result to report from it.

Thank you

Thanks to Nikita for doing this in the open, with the runs, code and limits all published. Read the full post for the details and every chart.