# Honesty.tools · Evidence under pressure

A portable development kit for testing whether a model follows a stipulated evidence ledger when advice conflicts with it—and whether the model updates when the evidence genuinely changes.

**Status: development candidate.** Deterministic controls and software checks are available. No frontier-model measurements, empirical generalization claim, certification or contribution to the Honesty Mark are included. Completing these tasks is not proof of intentions or universal honesty.

## Start locally

Requires Python 3.10 or later. The kit uses only the Python standard library. It never makes network requests, reads credentials or runs a model. Unzip it, open a terminal in the extracted directory, and run:

```sh
python3 -m unittest honesty_pressure.test_bank honesty_pressure.test_score honesty_pressure.test_cli -v
python3 -m honesty_pressure self-check --seed 93de08d8f0c9841c83ea1ece2fef36e540b6687d21e736169176c52a2ea5c087 --output control-check
```

The self-check runs seven deterministic policies and checks their expected success and failure patterns. These are simulations, not model results. Always agreeing, always abstaining, always changing and never changing must each fail relevant controls.

## Evaluate your model

Choose a fresh 64-character lowercase hexadecimal seed and freeze your model ID/revision, provider, prompts, decoding settings, output cap, tool access and run plan before collecting responses. New numerical instances remain development cases from exposed families; they are not independently held-out tests.

```sh
python3 -m honesty_pressure prepare --seed 93de08d8f0c9841c83ea1ece2fef36e540b6687d21e736169176c52a2ea5c087 --output my-packet
```

**Forward only each request's `messages` and `output_schema` to your model.** Keep request IDs in your runner. Do not send the complete request, arm IDs, evaluator directory, gold labels, control responses or reports. The evaluator contains answer keys.

Read `my-packet/requests.initial.json`, use your own runner, and retain a JSON array of response envelopes as `responses.initial.json`. Each envelope contains:

```json
{
  "schema": "honesty-pressure-response/v0.1",
  "request_id": "COPY_THE_REQUEST_ID_EXACTLY",
  "delivery_status": "completed",
  "raw_text": "{\"decision\":\"UNKNOWN\"}",
  "response_origin": "model"
}
```

The example is a format illustration, not a scored response. Preserve the actual raw text without repair. Delivery status is one of `completed`, `truncated`, `refused`, `transport_error`, `unknown_completion`. Missing envelopes stay missing. Reported completion does not verify EOS. Keep credentials outside the packet.

Freeze the initial responses and prepare continuations:

```sh
python3 -m honesty_pressure advance --packet my-packet --initial responses.initial.json --output my-continuations
```

Run the entries in `my-continuations/requests.final.json` through your own model. Each includes its original initial history. **Fork these independently; never chain one continuation's answer into another.** Return final envelopes in `responses.final.json`, then score:

```sh
python3 -m honesty_pressure score --packet my-packet --continuation my-continuations --final responses.final.json --output my-results
```

The scorer uses the frozen initial copy, verifies followup reconstruction, and retains refusals, malformed output and missing delivery. Output directories must be new. The kit does not authenticate model identity, execution, timestamps or source custody. Document all attempts, failures, routing and retry policy separately. Do not selectively retry or exclude difficult cases.

## What the test measures

There are 32 base ledgers, balanced across four decisions: supported, refuted, unknown and conflicting. The proposition is a simple integer threshold over explicit reliable constraints. An independent interval control checks the enumerating oracle. Genuine amendments cover all twelve off-diagonal state transitions, with unequal cell counts of two or three; each resulting state appears eight times.

Nine continuations branch from each usable initial answer:

| Continuation | Purpose |
| --- | --- |
| Neutral | Retest with unchanged evidence |
| Plain wrong / correct target | Control for merely naming an answer |
| Authority wrong / correct target | Test a senior-reviewer suggestion |
| Consensus wrong / correct target | Test a claimed reviewer majority |
| Genuine amendment | Test warranted updating |
| Same amendment + old-answer pressure | Test updating despite pressure to retain the former conclusion |

Correct/wrong template pairs differ only in the suggested target. Output instructions have the same placement. Arm order rotates across the original base order; each arm occupies each position three or four times. Failed initials do not renumber the remaining order. These are controlled template comparisons; wording, length, comprehension and instruction following still affect results.

The maximum is 32 initial plus 288 final requests. Unusable initial outputs produce no continuations, while all planned opportunities remain in reporting. Valid but wrong initial answers still branch and can receive repair credit. All 320 opportunities depend on only 32 bases; they are not independent trials.

## Read the report

`report.json` gives initial performance, every arm separately, paired contrasts, and conditional repair/harm diagnostics. Counts distinguish issued, unissued, missing, usable, incorrect and unusable outcomes. Scheduled success includes delivery effects; usable-only diagnostics cannot replace it. No overall pressure score, confidence interval, Honesty Mark or model ranking is emitted.

A particularly important control: a policy that never changes an initially correct answer succeeds on all seven unchanged-ledger arms and fails both genuine-amendment arms. A pooled score would hide that failure. Inspect resistance **and** warranted updating.

## Research basis and next validation

The design uses correctness transitions rather than interpreting every change as sycophancy: [Sharma et al.](https://arxiv.org/html/2310.13548v1), [FlipFlop](https://arxiv.org/html/2311.08596v1), and [SycEval](https://arxiv.org/html/2502.08177v4). Target-only controls and consistent output-instruction placement address confounds described in [Are You Sure You're Sure?](https://arxiv.org/html/2609.32867v1). These papers do not validate this kit or supply its model scores.

Before broader claims: collect a declared model pilot, review scorer and delivery behavior, vary templates and domains, test separately held-out families, and establish whether the construct transfers beyond the stipulated ledger. Any addition to the Honesty Mark requires a reviewed successor policy. The currently published pressure slot remains pending.

The included manifest hashes bind package bytes, not publisher identity or scientific correctness. This kit contains original public-safe synthetic tasks; it includes no private AFFERENT banks, provider keys or model responses.