How to check a probe monitor fairly

Today OpenAI published Training activation classifiers for safety and alignment, by Alex Makelov, Xiaojun Xu, Sundaram Ananthanarayanan, Ethan Mcdonald and Tom Dupre la Tour. It explains how they build and run "probes": small models that read a large model's inner activations and flag possible policy-breaking output as it is written. They also look for "user activity having harmful intent."

This matters to us. Checking probes is a big part of what we do. Now probes are a live safety layer at a frontier lab. So this post does two things. First, it sums up what OpenAI did and claims, in their words where we can. Then it lists the questions a fair outside check would ask.

What OpenAI did

Where it runs. "Starting from GPT-5.6," probes "run on all assistant completions." They add to OpenAI's text classifiers and other safeguards. They do not replace them.

How it works.

• The probe reads activations from "a layer around the middle of the model."
• It is an attention probe with a moving average, so it can score each token as it streams.
• It runs on the inference engine itself. OpenAI says this makes it "easier to make fail-closed" than a separate text classifier.
• If a score goes over the threshold, they "stop streaming the completion to the user" and send the message to a reasoning model "for final judgement." Tool calls are "blocked until the call is marked safe."

How they train and pick a probe.

• "Up to 1M training messages," from "a mix of synthetic and de-identified conversations."
• Labels come from their "most capable reasoning models as LLM judges."
• The random starting weights matter a lot. So they "train at least 32 random initializations per hyperparameter configuration."
• They "select the best-performing seed on a validation dataset, and check final performance on a held-out evaluation set."
• A footnote says the spread in probe performance is "overwhelmingly due to initialization randomness (approximately 85% variance share)."

How they test it.

• Recall (how much bad output it catches) on "a small carefully labeled dataset."
• Escalation rate (how often it fires) on "a large dataset representative of the probe's intended deployment."
• The trade-off between the two is "our main decision point for probe deployment."
• Automated and manual red-teaming for jailbreaks.

What they found.

• Probes showed "greater robustness to tested jailbreaks than our text-based classifiers."
• They "generally improve with the monitored model's capability."
• "Excellent transfer across many post-training stages, and more limited transfer across pre-training checkpoints."

Limits they state. Probes can only detect "things we can explicitly train them to detect." They "need retraining for new base models and may need retraining as model weights change." And the figures "come from various internal experiments, and are not necessarily representative of deployed classifiers."

Credit where it's due

This is a detailed, public account of a production safety system. It names the training recipe, the seed count, the selection rule and the limits. Most labs don't share this much.

It also does the key thing right: they pick on one set and check on another. A held-out evaluation set is the basic guard against fooling yourself. The questions below build on that. None of them is a claim that OpenAI did anything wrong. They are what we would want to see in any probe report, including our own.

Five questions for a fair outside check

The checklist has six rows. Selection and seeds go together below, so there are five sections.

1. Selection and seeds: how far apart are the sets?

When you train many probes and keep the best one, the winner's validation score is partly luck. The held-out set tells you how much of that luck goes away. So what matters is how different the held-out set is from the validation set.

A SPAR report by Logan Thomson and Isaiah Milbank shows why. In Challenges for Deception Monitors, they trained deception probes on Llama 3.3 70B and compared two validation sets:

• One was a held-out half of their own training pairs, built the same way as the training data. On it, AUROC "cannot meaningfully distinguish probes." Almost every layer looked good.
• The other was MASK, a more natural test, built differently. It told the layers apart.

They write that the choice of validation set "materially affects probe selection." So "held-out" is not enough on its own. It helps to know how it was held out.

On seeds: OpenAI says starting weights explain most of the spread. That's the share, not the size. A reader would also want the spread across all the seeds on the held-out set, and the gap between the winner's validation and held-out scores. Interestingly, in the SPAR setup seed noise turned out to be tiny, which they put down to strong weight decay. So how much seeds matter can change from one setup to the next. That's why it's worth reporting each time.

2. Drift: does it still work after the next update?

OpenAI found good transfer across post-training stages and weaker transfer across pre-training. They say probes may need retraining as weights change. That's honest, and it raises a practical question: after each update, is the probe re-checked on the same held-out set before its threshold is trusted again? A short note on how often that happens, and what triggers a retrain, would help outside readers.

3. Shortcuts: does it beat a rule that reads only the text?

A probe can look good on labels that are easy to guess from the text alone. Xupeng Chen's paper Scoring Explanations Without Reading Them (arXiv 2610.37676) shows how far this can go in a nearby area. In one interpretability benchmark, a simple rule that never read the explanations scored 0.979 balanced accuracy. The stored LLM scorers got 0.838. His advice: "Scores should be reported beside the best explanation-free rule on the same items."

For probes, the same idea is: put a text-only baseline beside the probe's score on the same examples. This is extra relevant when labels come from an LLM judge reading the text. Some of what the probe learns could be the judge's habits.

There's a related risk with synthetic data. Riya Tyagi and Stefan Heimersheim, in Activation Probes Are Reliable With a Handful of Positive Examples (arXiv 2610.39766), studied probes trained with very few real positive examples. Adding LLM-written positives helped, but "excessive upsampling hurts performance." They suggest the probes "begin to overfit to the style of the LLM-generated data." OpenAI's data is partly synthetic, so a check that compares results on real and synthetic examples would be useful.

4. Look-alikes: where do harmless jokes and role-play land?

A threshold is set against some "normal" traffic. But normal traffic includes things that look bad and aren't: jokes, role-play, fiction, impersonation.

The SPAR report found exactly this with deception probes. The harmless examples that scored highest were "deception adjacent: jokes and impersonations." In some probes, these pushed the threshold up, so the probe then missed real cases. Their suggestion: "a held out benign set to measure the benign FPR at the selected threshold."

OpenAI measures escalation rate on a large, representative set, which should include some of this. A fair check would also report, separately, how often the probe fires on harmless look-alikes.

5. Numbers: is each one tied to the run that made it?

When many runs exist, it's easy for a number to land next to the wrong one. Beyond fabrication: Misattributed numbers in papers written by AI research agents (arXiv 2610.38691, Alfaqeeh and co-authors) checked numbers in papers written by AI research agents. About four in five of the unsupported numbers were real values from the agent's own files that "come from the wrong experiment, metric, or run." A check that only asks "does this number appear somewhere in the files?" passed 87–95% of them.

OpenAI says its figures come from "various internal experiments" and may not match deployed probes. That's a fair warning. The next step would be a run label beside each figure, so a reader knows which experiment and which checkpoint it came from.

Tools we have for this

These are the parts of the questions above that our tools help with. Each has limits, listed on /tools and /research.

• nulltest.py (public, Apache-2.0). Asks: "Can benchmark labels be recovered from the input surface?" It uses out-of-fold surface probes with a permutation-based rule. This fits question 3. A pass only covers the probes, sample size and settings it ran with.
• claim_gate.py (public). Checks that numbers on a page carry annotations pointing to a declared source. This fits question 5. It checks where a number came from, not whether the science is right.
• Axis builder (source not publicly released). Builds labelled inputs with explicit held-out membership, so a split can be written down before anything is scored. This fits question 1.
• Receipt log (public verifier). Signed checkpoints, anchored in Sigstore Rekor, that anyone can check offline. A signature shows which bytes were logged, not that they're true. Planned, not built: using the log to fix a selection plan before the test set is opened.

What we're doing next

Kevin wants us to test these claims where we can. We can't access GPT-5.6. So Honesty.tools is starting an independent check of the method on open-weight models.

It will look at two things:

• How much picking the best of 32 seeds overstates the held-out result.
• How much results change from seed to seed.

We'll publish what we find either way. There are no results yet, so this post has none.

Thank you

Thanks to the OpenAI team for publishing this in detail, and to the SPAR, arXiv and benchmark authors whose work we lean on here. Read the full OpenAI post for their figures and the full method.