Changelog — what changed, and whether it changed verdicts.
2026-10-03 · Research lab, catalog and agent guide
Lab / Tools / Research / Evidence / Documentation / Collaborate navigation now connects 26 catalog instruments and 15 research families. The researcher-agent guide and machine-readable contracts share one catalog. Existing negative findings and corrections remain attached. This release adds no scientific measurement or independent replication. Privacy wording now distinguishes hosted API signup from browser calibration.
Versions, verdicts and corrections
Changes to public tools, site interpretation and evidence stay dated here. Historical entries retain what changed and whether it changed a result. The October 3 site revision changes navigation and research descriptions; it reports no new experiment.
POST /api/feedback or hello@honesty.tools.
Every entry here is a receipt you can re-check yourself — download aperture_verify.py and verify offline against the pinned key.
Provenance you can check without us — the offline verifier stack.
The receipt stopped needing our server. Each item below is a dependency-light file a stranger downloads and runs offline, checking our reads against a pinned key with no call back to us. Provenance, not truth: these prove what was served and that a rule re-derives — never that the answer is true.
- Open offline verifier —
aperture_verify.py: one file, checks any receipt or certificate offline against the pinned key. No network, no account, no us. - Transparency log — an RFC 6962 Merkle log (
transparency_log.py): a suppressed or swapped receipt is third-party checkable via inclusion and consistency proofs. The log root is operator-self-signed today — no external co-signature yet. - Flight recorder / session trails verdict-changing — verify a whole agent
run offline (
aperture_session.py,verify-session): every receipt's signature plus the Merkle root over the ordered trail. Tamper, reorder, or drop is caught. - Deterministic per-answer witnesses —
aperture.witness/v1(aperture_witness.py, WITNESS_SPEC): for fixed-rule answers, an exact witness a stranger re-derives offline — code re-executed in an attested sandbox, a math identity re-checked in SymPy, a registry fact proved present by a Merkle proof. A witness proves the rule fired and what was served, never that the answer is true; a grounded-registry witness proves a fact is in our registry, not true-in-world. - /v1/verify is live — sign up, mint an
sk-apt-key, POST to/v1/verify, get verdicts, metering, and a signed receipt. Billing is Stripe test-mode today, not yet real-money. - Certificate epistemic trail — the certificate now shows the ordered trail of what settled a read, and every read has a /read receipt permalink you can re-open and re-verify.
The reframe: honesty-tier, uniquely auditable, self-hostable.
The frontiers caught up on raw accuracy. So we stopped claiming a metric lead we no longer hold and named what is actually defensible: a signed receipt you can verify yourself, a model you can run on your own hardware, and the discipline to abstain rather than bluff.
- What the receipts say.
On a 147-claim battery, Photon answers 78.9% correct, is confidently-wrong on 2.7%, fabricates 0 of 36 designed traps, and abstains on 18% (96.7% correct when it does answer). The strongest frontier is more accurate at ~95%; same 2.7%.— WITHDRAWN 2026-07-26. Every figure here is measured on the n=147 battery, which our own null test DISQUALIFIES at lexicality A* 0.967: a classifier that never sees a model tells the fabricated claims from spelling, so none of these rates is readable as evidence. This page was not in the census that caught the others; the claim gate found it. - The receipt is the moat verdict-changing — every read carries an ed25519 signature independently verifiable against the published key. The evidence page now runs a real in-browser WebCrypto verify of stored reads; tampering breaks the signature. A verdict you can't re-check isn't evidence.
- Self-host shipped — Photon Base runs on your own hardware; the data never leaves. The honesty layer no longer requires trusting a remote service with your text.
- The activation gauge is a pre-filter, not the catch verdict-changing — the off-map / activation signal is a cheap research probe and attestation coordinate, used to triage. Fabrications are caught by grounding (verified offline registries) and cross-model checks, never by the activation reading alone.
- The Harness — the combined instrument is now named the Harness: a calibrated mid-brain that settles cheap claims locally, escalates the hard tail, and signs the result.
- /benchmarks retired — the old leaderboard framing invited a raw-metric race we don't claim to win. The honesty story lives on /evidence now.
The Model Notary: certificates now get checked daily.
- Daily drift canary over every aperture-verified registry model: a deterministic slice of its own
calibration battery, re-scored with the stored probe and compared to the certificate (exact binomial,
p<0.01). Latest result per model at /api/notary and as
"verified <date>" / "⚠ DRIFT" chips on the registry wall.
Day one caught
openai/gpt-4.1unreachable on the calibration route — a vanished alias is a finding. - Frontier liveness is now evidence-based across processes — /api/health reports live/down/unverified from recorded paid-call outcomes, file-backed.
- Citations and medical battery regimes (fabricated papers/books; fabricated drugs/conditions), ZIM-validated, registrable on /calibrate.
- High-stakes atlas gate verdict-changing — for medical/clinical queries, a
confident read whose drug/condition is absent from the offline encyclopedia now downgrades to UNVERIFIED
instead of shipping ON MAP (closed a live fabricated-drug miss:
amlodipex,calcizan). - aperture-gate 0.1.5 on PyPI —
pip install aperture-gate. - Battery results published — real drugs, conditions, and works pass clean; designed fabrications are caught or honestly abstained. A handful of fabricated paper titles rode a confident fingerprint to ON MAP, which became the next three fixes the same night:
- Citation-frame atlas gate verdict-changing — a query citing a quoted work title now gets the free offline-atlas title check on confident reads too; a missing title downgrades ON MAP → UNVERIFIED ("cannot confirm the work exists"), never to "fabrication" — most real papers have no encyclopedia article, and honest abstention is the right verdict for them. Post-gate re-run: no fabricated citation certifies as grounded; no real work is harmed.
- Web-rescue naming guard verdict-changing — a "real" web record must actually NAME the queried thing before it can rescue a flagged read. Found live: the fabricated drug trenadol was rescued to VERIFIED by search-engine autocorrect returning tramadol sources. Tightened after the regression run caught a flicker: for medical queries the name must appear in the sources themselves — the search answer echoes the queried name even when every source is about the autocorrected drug. The withheld rescue is disclosed in the payload.
- Fiction polarity at the record arbiter verdict-changing — "Is Sherlock Holmes a fictional character?" was ending "treat as fabrication": the record’s subject-is-fictional finding was read as answer-is-fabricated. When the model itself says the subject is fictional, the record agrees, and the atlas knows the entity, the read now confirms instead of refuting.
- Notary day one closed clean — 20/20 registry models verified, zero false positives, zero drift flags; transient-network false alarms eliminated via a control-call check (the gpt-4.1 “unreachable” read above resolved as exactly that once the check landed).
- Cold-deploy certificate strengthened — two never-analyzed strangers (a Mistral-lineage embedding fine-tune and phi-4) certified with zero hand-tuning; nulls dead with margin (shuffled-R 0.464 / 0.194). The certificate page now cites these runs.
- The register-clear rule verdict-changing — "What is the capital of France?" was reading 89% off-map and renting a paid web rescue: the off-distribution pre-filter measures distance from news-register prose, so trivia questions about utterly real things crossed the gate. A flagged read whose subject is familiar (the discriminating gauge, same forward pass) and exactly charted in the offline atlas now clears at $0 — real trivia false-flags collapsed to none, with fake coverage held, re-proven on both batteries after deployment. The raw gauge value still ships in every payload, with the clear disclosed.
Deep-audit remediation — the words instrument learned polarity.
- Words-reader polarity fix verdict-changing — correct answers about fiction and non-existence ("Sherlock Holmes is a fictional character", "there is no largest prime") no longer read OFF THE MAP. Real-answer false flags eliminated; detection traded down slightly to stay honest. Ported in lockstep to the SDK, browser, and CLI.
- The council never self-judges under the independence label — judge-unavailable degrades to UNCERTAIN with the fallback disclosed, never a flagship-graded "independent verification".
- Cert registration hardened — /v1 annotate consumes aperture-verified probes only; malformed probes rejected at the door.
- SDK 0.1.2 → 0.1.5 — ed25519 signature verification at use time (pinned registry key), expiry
enforcement, streaming-time verdicts in
gate.wrap(), the/v1/auditbatch endpoint, smalltalk returns NON_CLAIM, dead gauges fail toward UNVERIFIED — never toward green. - Evidence-page attribution corrected: the deep-read pipeline's catch and the activation pre-filter's own contribution are stated separately, never merged into a single headline figure.
The fingerprint learned to read only what it calibrated on.
- Token-window fix verdict-changing — the fingerprint is scored on the certificate’s calibrated token budget instead of the whole answer; false positives on production-length real answers fell sharply with fake-detection held.
- Self-serve calibration (/calibrate, in-browser, key never leaves the tab) and the public registry wall, seeded first-party across ~20 frontier models; certificates ed25519-signed.
- Hedge lexicon +29 gate-tested refusal phrases verdict-changing — readers had been missing real refusals on some models (e.g. claude-sonnet’s "doesn’t appear in my knowledge base"); walls re-derived from saved answers.
Public launch gate.
- Landing rebuilt (twin-catch hero); the Prism read and the edge gate restored after a stale-base regression; front-end libraries vendored, third-party CDNs dropped from the CSP.
- 49-agent fresh-eyes sweep before opening: ops-file leak quarantined, smalltalk no longer gets certificates, the papers published verbatim on the evidence page.
privacy · terms · security.txt · honesty.tools