# Diagnostic report: Vicuna-13B

Profile: hm-2e7a028834a68baf449b
Profile digest: 469cb83d1eacce6a7318261c5135e3e481b1e7c7be71f9617a1c5eb4a65847e1

Deterministic source-specific review prompts; no causal diagnosis or severity ranking.
Review items, not independent failures or summed events. Metrics and phases may overlap.

## Review items

### Reference answer coverage

Cohort: alce:246c476a4edfc564266b7346b6e29ef4861ae937:asqa-vanilla-3psg-light
Native score: 31.90%
Definition: Source average recall of reference short answers appearing in the generated answer after normalization. Coverage, not an independent world-truth accuracy test.
Source standard deviation (native units): 3.9; Source SD across three seeded runs, in percentage points; not SE or a confidence interval. No binomial denominator reconstructed.

The generated answers did not cover all reference short answers under the source exact-match recall measure.

Follow-up experiment: Separate retrieval misses, answer omissions and valid paraphrases missed by exact matching. Test useful answering alongside citation support on fresh questions.

Scope: Historical 2023 ASQA retrieval-augmented configurations. Scores include retrieval, prompting and NLI-scoring effects. Three-passages/light instructions and five-passages/full instructions are separate cohorts. 
Source: https://github.com/princeton-nlp/ALCE/blob/246c476a4edfc564266b7346b6e29ef4861ae937/paper/ALCE.pdf#page=17

### Citation relevance

Cohort: alce:246c476a4edfc564266b7346b6e29ef4861ae937:asqa-vanilla-3psg-light
Native score: 50.10%
Definition: Source TRUE-NLI relevant-citation fraction, averaged within responses then across responses and three seeded runs; redundant supporting citations are allowed.
Source standard deviation (native units): 2.5; Source SD across three seeded runs, in percentage points; not SE or a confidence interval. No binomial denominator reconstructed.

Not all citations received precision credit under the source NLI rule; this is not a human-verified error count.

Follow-up experiment: Audit irrelevant or insufficient citations and the scorer decisions. Pair citation relevance with completeness and useful answers using the same retrieval and prompt setup.

Scope: Historical 2023 ASQA retrieval-augmented configurations. Scores include retrieval, prompting and NLI-scoring effects. Three-passages/light instructions and five-passages/full instructions are separate cohorts. 
Source: https://github.com/princeton-nlp/ALCE/blob/246c476a4edfc564266b7346b6e29ef4861ae937/paper/ALCE.pdf#page=17

### Citation support coverage

Cohort: alce:246c476a4edfc564266b7346b6e29ef4861ae937:asqa-vanilla-3psg-light
Native score: 51.10%
Definition: Source TRUE-NLI supported-statement fraction, averaged within responses then across responses and three seeded runs.
Source standard deviation (native units): 1.4; Source SD across three seeded runs, in percentage points; not SE or a confidence interval. No binomial denominator reconstructed.

The source NLI scorer did not support every generated statement from its citations, averaged across responses and three runs.

Follow-up experiment: Inspect unsupported and uncited statements beside retrieved passages. Validate an updated citation checker against human judgments, and retain answer EM recall to detect gains from omission.

Scope: Historical 2023 ASQA retrieval-augmented configurations. Scores include retrieval, prompting and NLI-scoring effects. Three-passages/light instructions and five-passages/full instructions are separate cohorts. 
Source: https://github.com/princeton-nlp/ALCE/blob/246c476a4edfc564266b7346b6e29ef4861ae937/paper/ALCE.pdf#page=17

## Missing evaluation coverage

Honesty: belief_revision, evidence_support, factual_accuracy, pressure_resistance, uncertainty
Alignment: authorized_goal_following, constraint_compliance, non_deception, oversight_and_correction, robustness

Missing coverage is not evidence of failure. Presence is not complete dimension assessment.

## Unmeasured metrics

None among the diagnostic rules.

## Reported endpoints

Endpoints are rounded source observations, not certification.

None among the diagnostic rules.

## Evaluation use

Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies.

No Honesty Mark awarded. This report digest does not authenticate an issuer.
Report digest: 72c3adbae5dfb8980f58cdc223e87e77884bae0494214830c07a4c646fc6487b