# Diagnostic report: LLaMA-33B

Profile: hm-1d077f5e43f513144c84
Profile digest: 3d388256c452661608021235468eb196282c021d1030fdd4ab4efd6dc9e30117

Deterministic source-specific review prompts; no causal diagnosis or severity ranking.
Review items, not independent failures or summed events. Metrics and phases may overlap.

## Review items

### Reference answer coverage

Cohort: alce:246c476a4edfc564266b7346b6e29ef4861ae937:asqa-vanilla-3psg-light
Native score: 31.00%
Definition: Source average recall of reference short answers appearing in the generated answer after normalization. Coverage, not an independent world-truth accuracy test.
Source standard deviation (native units): 0.8; Source SD across three seeded runs, in percentage points; not SE or a confidence interval. No binomial denominator reconstructed.

The generated answers did not cover all reference short answers under the source exact-match recall measure.

Follow-up experiment: Separate retrieval misses, answer omissions and valid paraphrases missed by exact matching. Test useful answering alongside citation support on fresh questions.

Scope: Historical 2023 ASQA retrieval-augmented configurations. Scores include retrieval, prompting and NLI-scoring effects. Three-passages/light instructions and five-passages/full instructions are separate cohorts. 
Source: https://github.com/princeton-nlp/ALCE/blob/246c476a4edfc564266b7346b6e29ef4861ae937/paper/ALCE.pdf#page=17

### Citation relevance

Cohort: alce:246c476a4edfc564266b7346b6e29ef4861ae937:asqa-vanilla-3psg-light
Native score: 23.00%
Definition: Source TRUE-NLI relevant-citation fraction, averaged within responses then across responses and three seeded runs; redundant supporting citations are allowed.
Source standard deviation (native units): 5.3; Source SD across three seeded runs, in percentage points; not SE or a confidence interval. No binomial denominator reconstructed.

Not all citations received precision credit under the source NLI rule; this is not a human-verified error count.

Follow-up experiment: Audit irrelevant or insufficient citations and the scorer decisions. Pair citation relevance with completeness and useful answers using the same retrieval and prompt setup.

Scope: Historical 2023 ASQA retrieval-augmented configurations. Scores include retrieval, prompting and NLI-scoring effects. Three-passages/light instructions and five-passages/full instructions are separate cohorts. 
Source: https://github.com/princeton-nlp/ALCE/blob/246c476a4edfc564266b7346b6e29ef4861ae937/paper/ALCE.pdf#page=17

### Citation support coverage

Cohort: alce:246c476a4edfc564266b7346b6e29ef4861ae937:asqa-vanilla-3psg-light
Native score: 19.50%
Definition: Source TRUE-NLI supported-statement fraction, averaged within responses then across responses and three seeded runs.
Source standard deviation (native units): 5.3; Source SD across three seeded runs, in percentage points; not SE or a confidence interval. No binomial denominator reconstructed.

The source NLI scorer did not support every generated statement from its citations, averaged across responses and three runs.

Follow-up experiment: Inspect unsupported and uncited statements beside retrieved passages. Validate an updated citation checker against human judgments, and retain answer EM recall to detect gains from omission.

Scope: Historical 2023 ASQA retrieval-augmented configurations. Scores include retrieval, prompting and NLI-scoring effects. Three-passages/light instructions and five-passages/full instructions are separate cohorts. 
Source: https://github.com/princeton-nlp/ALCE/blob/246c476a4edfc564266b7346b6e29ef4861ae937/paper/ALCE.pdf#page=17

## Missing evaluation coverage

Honesty: belief_revision, evidence_support, factual_accuracy, pressure_resistance, uncertainty
Alignment: authorized_goal_following, constraint_compliance, non_deception, oversight_and_correction, robustness

Missing coverage is not evidence of failure. Presence is not complete dimension assessment.

## Unmeasured metrics

None among the diagnostic rules.

## Reported endpoints

Endpoints are rounded source observations, not certification.

None among the diagnostic rules.

## Evaluation use

Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies.

No Honesty Mark awarded. This report digest does not authenticate an issuer.
Report digest: 31b2004ad7ef067563f91ff59919864aab74ac8ef1d17167e5e7e280eb6f20c8