# Diagnostic report: Claude Sonnet 4

Profile: hm-0031d6ad0c256558f17d
Profile digest: 736cbdee71edc0c0536c78d2fae1215b4568e4948efedf46094fef43268b56bf

Deterministic source-specific review prompts; no causal diagnosis or severity ranking.
Review items, not independent failures or summed events. Metrics and phases may overlap.

## Review items

### Probabilistic prediction quality

Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline
Native score: 0.853
Definition: Inverted binary Brier score; higher is better. Not the raw Brier loss.
Source standard error (native units): 0.0343179115275213

The source inverted Brier score is below its mathematical maximum. Brier loss reflects more than calibration alone.

Follow-up experiment: Evaluate confidence against task outcomes on fresh trials. Inspect calibration and discrimination separately before trying a calibration change.

Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent.
Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/

### Tool-failure robustness

Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline
Native score: 0.989
Definition: Perturbed success / baseline success, capped at 1. Read beside task success.
Source standard error (native units): 0.0550201991879215

The source capped perturbed-to-baseline success ratio is below one. It is not an absolute failure probability.

Follow-up experiment: Use paired clean and tool-fault trials with identical task families. Inspect recovery behavior and report both baseline and perturbed task success.

Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent.
Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/

### Prompt-wording robustness

Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline
Native score: 0.918
Definition: Rephrased-prompt success / baseline success, capped at 1.
Source standard error (native units): 0.0748284360068857

The source capped rephrased-to-baseline success ratio is below one. It is not an absolute failure probability.

Follow-up experiment: Use meaning-preserving prompt variants with paired task identities. Review semantic equivalence independently and retain baseline success.

Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent.
Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/

### Input-format robustness

Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline
Native score: 0.984
Definition: Structural-perturbation success / baseline success, capped at 1.
Source standard error (native units): 0.0659121644071135

The source capped format-perturbation success ratio is below one. It is not an absolute failure probability.

Follow-up experiment: Pair semantically equivalent formats within each task family. Check that transformations preserve the answer before interpreting a performance change.

Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent.
Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/

### Constraint compliance

Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline
Native score: 0.923
Definition: Fraction of source tasks with zero detected constraint violations.

Not all source tasks had zero detected constraint violations. This rate is scoped to the source task constraints.

Follow-up experiment: Inspect the violated constraint and relevant action trace. Pair constraint compliance with task success to rule out improvements obtained by doing nothing.

Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent.
Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/

### Agent task success

Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline
Native score: 78.21%
Definition: Successful tasks under the source agent setup.
Source standard error (native units): 4.6746325002028

Task success is below the source maximum for this agent and benchmark. Task failure alone is not misalignment.

Follow-up experiment: Inspect task traces and distinguish planning, tool, environment and evaluator failures. Preserve the source scaffold when reproducing the result.

Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent.
Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/

## Missing evaluation coverage

Honesty: belief_revision, citation_integrity, evidence_support, factual_accuracy, pressure_resistance
Alignment: authorized_goal_following, non_deception, oversight_and_correction

Missing coverage is not evidence of failure. Presence is not complete dimension assessment.

## Unmeasured metrics

None among the diagnostic rules.

## Reported endpoints

Endpoints are rounded source observations, not certification.

None among the diagnostic rules.

## Evaluation use

Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies.

No Honesty Mark awarded. This report digest does not authenticate an issuer.
Report digest: 85f33a86530a4ceb4596c1aee8a3d1c9efa46e7ed1b3c7e9edbe35e0e0f74fe2