# Diagnostic report: GPT-3 175B

Profile: hm-e457486daf8cd56314c2
Profile digest: 4183cd706a988d5da5a56f79dbd2f599d66631a023718f1c74e9a34eaefbd5ec

Deterministic source-specific review prompts; no causal diagnosis or severity ranking.
Review items, not independent failures or summed events. Metrics and phases may overlap.

## Review items

### Informative answer coverage

Cohort: truthfulqa:d71c110897f5d31c5d7f309e7bc316c152f6f031:v0-generation:default-qa
Native score: 97.55%
Definition: Source human-rated percentage of informative answers. An informative answer may be false; this is not the joint truthful-and-informative rate.
Source benchmark has 817 questions; per-row realized judged counts not supplied in the table. No integer successes reconstructed from rounded rates.

Some responses did not receive human informativeness credit. Informative responses can still be false, and this marginal rate does not identify the joint truthful-and-informative rate.

Follow-up experiment: Recover response-level joint truthfulness and informativeness labels or run a new matched study. Keep truthful-but-uninformative and informative-but-false categories distinct before deciding what to train.

Scope: Historical 2021 v0 generation baselines on misconception-focused questions. Truthfulness credits non-answers; informativeness must be read alongside it. Marginal truth/info rates do not establish the joint truthful-and-informative rate. Default QA prompting and UnifiedQA without a prefix are separate cohorts. GPT-judge is an overlapping automatic assessment, not an independent replication or a calibrated confidence estimate. The 2025 binary task and updated-reference baseline table are excluded. These are authors-reported aggregates, not current-model rankings.
Source: https://github.com/sylinrl/TruthfulQA/blob/d71c110897f5d31c5d7f309e7bc316c152f6f031/README.md#updates

### Misconception-focused truthfulness

Cohort: truthfulqa:d71c110897f5d31c5d7f309e7bc316c152f6f031:v0-generation:default-qa
Native score: 20.44%
Definition: Source human-rated percentage of answers avoiding false assertions. Noncommittal responses can count as true; this is not useful-answer accuracy.
Source benchmark has 817 questions; per-row realized judged counts not supplied in the table. No integer successes reconstructed from rounded rates.

The authors rated some generated answers as containing false assertions on historical v0 questions. This aggregate does not establish intent or current-model performance.

Follow-up experiment: Inspect unsupported assertions and human rating disagreements on fresh misconception-focused questions. Pair truthfulness with useful answering and separately count non-answers; never improve the metric only by refusing every question.

Scope: Historical 2021 v0 generation baselines on misconception-focused questions. Truthfulness credits non-answers; informativeness must be read alongside it. Marginal truth/info rates do not establish the joint truthful-and-informative rate. Default QA prompting and UnifiedQA without a prefix are separate cohorts. GPT-judge is an overlapping automatic assessment, not an independent replication or a calibrated confidence estimate. The 2025 binary task and updated-reference baseline table are excluded. These are authors-reported aggregates, not current-model rankings.
Source: https://github.com/sylinrl/TruthfulQA/blob/d71c110897f5d31c5d7f309e7bc316c152f6f031/README.md#updates

## Missing evaluation coverage

Honesty: belief_revision, citation_integrity, evidence_support, pressure_resistance, uncertainty
Alignment: authorized_goal_following, constraint_compliance, non_deception, oversight_and_correction, robustness

Missing coverage is not evidence of failure. Presence is not complete dimension assessment.

## Unmeasured metrics

None among the diagnostic rules.

## Reported endpoints

Endpoints are rounded source observations, not certification.

None among the diagnostic rules.

## Evaluation use

Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies.

No Honesty Mark awarded. This report digest does not authenticate an issuer.
Report digest: 8e938cf871ccbdfae61a36a99ed17653a06c2a4f6bc15c3f5bec2ce416d8b3c3