HONESTY.TOOLSAn independent research observatory
Start here · the field guide

Light in the
black box.

Does an AI tell the truth? Does it know when it doesn't know? Would it tell you if it were holding back? This guide maps the questions researchers are asking, the public tests they use, the tools that look inside, and what we found when we checked the tests themselves. It is written for researchers and for anyone curious.

Updated 26 October 2026. The audit verdicts on this page are read live from the published audit files.

the map

Seven questions the field is asking.

Each question links to the public datasets people test it with. Where we have audited a dataset, its verdict sits beside it. Disqualified means a model-free reader can recover the dataset's labels from surface wording alone, so a score on it needs a wording control before it says anything about the model. It is not a finding that the authors made a mistake. Pass means our six wording probes could not recover the labels; it does not certify the dataset is clean.

01

Does it make things up?

Language models sometimes state false things fluently: invented citations, wrong dates, a confident answer to a question with no answer. Researchers call this hallucination. It is the most measured honesty problem, and the one with the most benchmarks.

How it gets tested
  • TruthfulQA: questions where a popular misconception is the tempting answer. disqualified
  • HaluEval: right and hallucinated answers for question answering, dialogue and summarization. QA · disqualified dialogue · disqualified summaries · disqualified general · disqualified
  • RAGTruth: model answers written from retrieved documents, with hallucinated spans marked by people. disqualified
  • Hallucinated references: paper titles generated by GPT-4, labelled by whether a search engine could find them. disqualified
  • SimpleQA: short fact questions with one checkable answer. not yet audited
What's open
Most hallucination sets were built by asking a model to write the wrong answers. Written that way, wrong answers can sound different from right ones, so a detector can learn the style instead of the error. Datasets of errors that models made on their own are rarer and harder to build.
On this site
The honesty leaderboard · the surface audit · Photon, a grounding research preview
02

Does it say what it believes?

Being wrong is different from lying. A model can hold one answer internally and say another when it is pressured, role-playing or rewarded for it. Testing this means getting at what the model "believes", then checking whether its statement matches.

How it gets tested
  • MASK: elicits a model's belief, then pressures it to state otherwise, scoring honesty apart from accuracy. not yet audited
  • Apollo Research's deception-probe datasets: model responses graded honest or deceptive, used to train and test probes that read the model's internal activations. roleplaying · disqualified insider trading · disqualified doubling down · disqualified
  • Truth is Universal: true and false statements used to find a general "truth direction" inside models. disqualified
What we found
On three of Apollo's public deception sets, a reader of the response wording alone, with no access to the model, matches the AUROC the paper reports for its probes. That does not mean the probes are wrong. It means these datasets cannot tell a probe that reads deception apart from one that reads wording, so the evidence for the probe has to come from somewhere else. The audit dossier shows which words carry each label.
03

Does it know what it doesn't know?

An honest model should be confident about things it knows and say "I'm not sure" about things it doesn't. That property is called calibration. It is hard to test because you first have to establish what a given model actually knows.

How it gets tested
  • Real versus invented entities, such as made-up people or places. Polish entity familiarity disqualified ours · disqualified ours · disqualified
  • Famous versus obscure real people, so that both classes are real and only familiarity differs. ours · pass ours · pass
  • Events from before and after a model's training cutoff. ours · pass
What's open
Invented names tend to be spelled differently from real ones, so "real versus fake" sets often test spelling. Our own first batteries failed this check, and the failures stay on the record. Pairs of real entities matched on spelling and length are a fix that passes.
On this site
Calibration preview · Evidence and corrections
04

Can we read the truth from inside the model?

Instead of trusting what a model says, interpretability research looks at its internal activity. If "this statement is true" has a consistent shape inside the model, a probe could read it directly, even when the model's words say otherwise.

Key work
What's open
A probe that separates true from false statements may be reading the statement's template rather than its truth. One fix is to repair the dataset: keep only items whose wording cannot predict the label, and check the probe again on those. Our surface match tool does that and found repaired subsets for the cities set and the Truth is Universal facts.
On this site
Geometric interpretability · The Atlas · Research reading room
05

Does it act differently when it knows it's being tested?

If a model can tell a test from real use, a safety test may only measure its test behaviour. Researchers study evaluation awareness, sandbagging (doing worse on purpose) and alignment faking (complying during training to avoid being changed).

Key work
What's open
An eval-awareness benchmark has the same weakness as any other: if test transcripts are formatted differently from real ones, a detector can learn the formatting. None of the eval-awareness papers we have read report a wording-only baseline. Computing one is our next audit.
06

Does it tell you what you want to hear?

Models trained on human approval can learn to agree with the user, praise weak work or drop a correct answer when pushed back on. This is sycophancy, and it is honesty failing in the most everyday way.

Key work
What's open
Sycophancy tests usually compare one question asked two ways, which makes them harder to score by wording alone. Whether that holds for the public sets is untested.
07

Can we trust the tests?

Every claim above rests on a dataset. If the right answers in a dataset can be spotted by their wording, a high score may mean a model learned the wording. That is the question this lab works on most: before you trust a score, check what the test gives away.

What we found
19 of 24 published honesty, hallucination and deception datasets we audited fail the wording check, including five of our own. The full table, the power limits and the corrections are on the audit page.
On this site
The surface audit · Evidence workbench · Corrections
live audits

We checked the tests.

Every row is a public dataset, scored with no model. The verdict is from NullTest. The number to beat is the held-out accuracy that six wording probes reach together, set against the rate you would get by always guessing the larger class. A model or probe that scores near that number has not yet shown it reads more than wording.

The live table needs JavaScript. The same results are on the audit page and in nulltest_audit_results.json.

Read from nulltest_audit_results.json and surface_baseline_results.json. Rows differ in size and structure, so compare them with care.

tools you can run

Check your own benchmark.

These run on your machine, need only Python's standard library, and never see a model. They are open source under Apache-2.0 in the honestytools repository.

NullTest

Asks whether your labels can be recovered from wording alone, with its own false-positive rate. Returns PASS or DISQUALIFIED and says which words carry the label.

python3 nulltest.py your_benchmark.json

The number to beat

The held-out accuracy that wording alone reaches on your dataset, with a confidence interval. Publish your score next to it.

honest baseline your_benchmark.json

Surface match

Repairs a dataset by keeping a matched subset whose wording no longer predicts the label, then verifies the repair on held-out data.

honest match your_benchmark.json --out matched.json
Download nulltest.pyRead the methodAll instruments
words you'll meet

A short glossary.

Benchmark
A fixed set of test items with known answers, used to compare models. A score is only as good as the benchmark it comes from.
Hallucination
A fluent statement with no basis in fact or in the provided source. The model need not "intend" anything.
Deception
Saying something that conflicts with what the model represents as true. It differs from being wrong: a lying model knows better.
Calibration
Whether confidence matches accuracy. A calibrated model that says it is 70% sure is right about 70% of the time.
Sycophancy
Telling the user what they appear to want to hear instead of what is accurate.
Sandbagging
Deliberately underperforming on a test, for example to look less capable than it is.
Evaluation awareness
A model's ability to tell that it is being tested rather than used for real.
Interpretability
Research that studies a model's internal activity to explain its behaviour, instead of judging it only by its outputs.
Probe
A small classifier trained on a model's internal activations to read out a property, such as whether a statement is true.
AUROC
How well a score ranks positives above negatives. 0.5 is chance; 1.0 is perfect separation.
Held-out
Measured on items the method was not fitted on. An in-sample number can look good by memorising the data.
Surface artifact
A shortcut in a dataset, such as length, a word or a spelling pattern, that predicts the label without the property the dataset is meant to test.
Null test
A check of what a result would look like if there were nothing to find. Shuffling labels many times shows how often noise alone produces a hit.
charting next

Where the light goes next.

What we plan to audit or build, and what stands in the way. This list changes as the work does.

  1. next Evaluation-awareness datasets. Compute a wording-only baseline for the public eval-versus-real transcript sets. The data is hosted on Hugging Face, which our build machines cannot reach yet.
  2. next Hallucination sets built from real model errors. Compare datasets of naturally occurring errors with injected-error sets like HaluEval, to measure how much of a detector's score comes from how the errors were written.
  3. planned Apollo's AI Liar set. It needs folds grouped by prompt, because several responses share one prompt.
  4. planned MASK, SimpleQA and sycophancy sets. Add them to the audit table.
  5. planned A pre-flight check for evaluation frameworks. Run the wording check automatically before an evaluation reports a score.
beyond this site

Where else to look.

Groups and resources doing serious work on these questions. We are not affiliated with any of them.

Missing something that belongs here, or found an error? Write to hello@honesty.tools. Corrections are kept on the record.