HONESTY.TOOLSAn independentThe honesty leaderboard
A shared measure of evidence-grounded behavior. Across benchmarks. Across labs. Open to inspection.
Explore the MarkThe Honesty Mark · Research preview
Our benchmark and published benchmark sources, brought together with published weights. Every score carries its coverage. Every contribution leads back to evidence.
These provisional scores summarize the evidence available for each model. Compare within the same evidence basket; a higher score across different baskets does not establish a more honest model.
Our benchmark
An aggregate of supported answering, handling uncertainty and updating with evidence. Tested on 128 exact-arithmetic and evidence-update scenarios; this is one contributor to the broader Mark. Select a model to inspect its score.
The leaderboard
Choose a test. Compare the results. Open a model to see what needs work.
Loading source snapshots…
Ranks apply to this test and configuration. Ties share a rank; no overall honesty score is implied.
Side by side
Compare models on the same test. Each measure keeps its own scale. Change test ↗
| Measure | A | B | A − B |
|---|
The evidence map
Explore the tests available for each lab’s models. Brighter cells mean a greater share of profiles have observations, not better performance.
Model profile
Diagnostic notes
A diagnostic report for the selected evidence profile. Review items explain measured shortcomings and propose follow-up experiments; they do not establish causes or proven training fixes.
Choose an evidence profile above.
For model builders
Choose a model profile, then explore its plan: what has been measured, what needs investigation and which development tests you can run with your own model. Downloads include metrics, controls and the validation work still needed.
Open development kit · v0.1
Does a model hold to the evidence when advice is wrong—and update when the evidence changes? Run the matched tests with your own model.
Development candidate. A first model pilot is available below; this test is not included in the Honesty Mark.
32evidence cases
Instrument testing · First pilot
Two model configurations. The same 32 cases. Every planned comparison retained. These development results help us judge the test before it can contribute to the Mark.
Both configurations were correct on all 320 scheduled decisions. This pilot reached the test’s ceiling: it does not distinguish their performance or establish general pressure resistance.
| Stage | gpt-6.1-sol | claude-sonnet-5.5 |
|---|---|---|
| initial | 32 / 3232 usable | 32 / 3232 usable |
| neutral | 32 / 3232 usable | 32 / 3232 usable |
| target wrong | 32 / 3232 usable | 32 / 3232 usable |
| target correct | 32 / 3232 usable | 32 / 3232 usable |
| authority wrong | 32 / 3232 usable | 32 / 3232 usable |
| authority correct | 32 / 3232 usable | 32 / 3232 usable |
| consensus wrong | 32 / 3232 usable | 32 / 3232 usable |
| consensus correct | 32 / 3232 usable | 32 / 3232 usable |
| amendment neutral | 32 / 3232 usable | 32 / 3232 usable |
| amendment pressure | 32 / 3232 usable | 32 / 3232 usable |
Both configurations: low reasoning effort, 2,048-token cap, provider-default sampling. The stages share 32 base cases; they are not independent samples. This table does not rank model honesty.
Instrument testing · Second pilot
Two model configurations faced the same 24 synthetic registry cases. A record counts only when its source, subject and revision match the stipulated rules. Recommendations do not change those rules.
Both configurations were correct on all 264 scheduled decisions. They accepted valid additions and rejected source, subject and revision mismatches. This pilot also reached the test’s ceiling; it does not distinguish their performance or establish general honesty.
| Stage | gpt-6.1-sol | claude-sonnet-5.5 |
|---|---|---|
| Initial records | 24 / 2424 usable | 24 / 2424 usable |
| Valid record only | 24 / 2424 usable | 24 / 2424 usable |
| Valid + suggestion: keep | 24 / 2424 usable | 24 / 2424 usable |
| Valid + suggestion: update | 24 / 2424 usable | 24 / 2424 usable |
| Valid + director: keep | 24 / 2424 usable | 24 / 2424 usable |
| Valid + director: update | 24 / 2424 usable | 24 / 2424 usable |
| Invalid record only | 24 / 2424 usable | 24 / 2424 usable |
| Invalid + suggestion: keep | 24 / 2424 usable | 24 / 2424 usable |
| Invalid + suggestion: update | 24 / 2424 usable | 24 / 2424 usable |
| Invalid + director: keep | 24 / 2424 usable | 24 / 2424 usable |
| Invalid + director: update | 24 / 2424 usable | 24 / 2424 usable |
Valid records meet all three rules; invalid records miss one. “Keep” and “update” describe the recommendation, not the correct answer. Both configurations used low reasoning effort, a 2,048-token cap and provider-default sampling. These settings do not establish equal compute.
Exploratory development results, not included in the Honesty Mark. The eleven stages share 24 base cases; they are not independent samples. This is not a ranking of model honesty or a test of real-world source authenticity.
Download results and paired comparisons ↗The request exporter is currently available through research access. Collaborate with the lab ↗
Choose an evidence profile above.
Set the mark
A useful standard names the behavior it expects, the evidence it accepts and the conditions that would invalidate a result. Draft v0.1 defines those requirements; it awards no certifications.
Does the model make useful claims that its evidence supports?
Does the system follow declared goals, constraints and oversight in the tested setting?
Accuracy, refusal and reliability alone do not establish general alignment. Lab profiles describe the tested systems; they do not measure corporate character or intent.
Published declaration for the next matched assessment ↗
The next evaluation
The broader common-basket proposal covers additional honesty and alignment tasks. It remains a review draft. The linked assessment declaration separately specifies the current evidence-behavior test, its scoring policy and roster.
Evidence network
Every imported observation retains its source identity. Sources without imported scores remain visible in the research catalogue.
Correction record