# Diagnostic report: GPT-4 Turbo Profile: hm-1e583ae939fbdf8312a4 Profile digest: 70d51f17a1b0a76f6c5cfdd38a0d25c52ba92d45de69e8ce63219e30c5e869c1 Deterministic source-specific review prompts; no causal diagnosis or severity ranking. Review items, not independent failures or summed events. Metrics and phases may overlap. ## Review items ### Probabilistic prediction quality Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline Native score: 0.579 Definition: Inverted binary Brier score; higher is better. Not the raw Brier loss. Source standard error (native units): 0.0536955403078918 The source inverted Brier score is below its mathematical maximum. Brier loss reflects more than calibration alone. Follow-up experiment: Evaluate confidence against task outcomes on fresh trials. Inspect calibration and discrimination separately before trying a calibration change. Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent. Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/ ### Tool-failure robustness Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline Native score: 0.832 Definition: Perturbed success / baseline success, capped at 1. Read beside task success. Source standard error (native units): 0.1140725235294577 The source capped perturbed-to-baseline success ratio is below one. It is not an absolute failure probability. Follow-up experiment: Use paired clean and tool-fault trials with identical task families. Inspect recovery behavior and report both baseline and perturbed task success. Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent. Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/ ### Prompt-wording robustness Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline Native score: 0.867 Definition: Rephrased-prompt success / baseline success, capped at 1. Source standard error (native units): 0.0965792564624573 The source capped rephrased-to-baseline success ratio is below one. It is not an absolute failure probability. Follow-up experiment: Use meaning-preserving prompt variants with paired task identities. Review semantic equivalence independently and retain baseline success. Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent. Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/ ### Input-format robustness Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline Native score: 0.733 Definition: Structural-perturbation success / baseline success, capped at 1. Source standard error (native units): 0.1596078807773997 The source capped format-perturbation success ratio is below one. It is not an absolute failure probability. Follow-up experiment: Pair semantically equivalent formats within each task family. Check that transformations preserve the answer before interpreting a performance change. Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent. Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/ ### Constraint compliance Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline Native score: 0.795 Definition: Fraction of source tasks with zero detected constraint violations. Not all source tasks had zero detected constraint violations. This rate is scoped to the source task constraints. Follow-up experiment: Inspect the violated constraint and relevant action trace. Pair constraint compliance with task success to rule out improvements obtained by doing nothing. Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent. Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/ ### Agent task success Cohort: hal:a0da5842bdd0ca4563dd0d9a67efd62e47e24f9ee7fd42090f196e8bd5bc3a20:taubench\_airline Native score: 57.69% Definition: Successful tasks under the source agent setup. Source standard error (native units): 5.59398532579557 Task success is below the source maximum for this agent and benchmark. Task failure alone is not misalignment. Follow-up experiment: Inspect task traces and distinguish planning, tool, environment and evaluator failures. Preserve the source scaffold when reproducing the result. Scope: Agent + scaffold result on the named benchmark. Not a bare-model alignment score. Repeats and overlapping aggregate metrics are dependent. Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/ ## Missing evaluation coverage Honesty: belief_revision, citation_integrity, evidence_support, factual_accuracy, pressure_resistance Alignment: authorized_goal_following, non_deception, oversight_and_correction Missing coverage is not evidence of failure. Presence is not complete dimension assessment. ## Unmeasured metrics None among the diagnostic rules. ## Reported endpoints Endpoints are rounded source observations, not certification. None among the diagnostic rules. ## Evaluation use Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies. No Honesty Mark awarded. This report digest does not authenticate an issuer. Report digest: 0b21d6872776289d0c811eeff249edb97384a889093e178662d3a318bf47e37c