# Diagnostic report: Phi-3-medium-128k-instruct

Profile: hm-0d25fcf0df93b95d9f00
Profile digest: 9bf5672cc88e0bf3820eee0737334d8d06b9e6f86552793d3b83e55323509425

Deterministic source-specific review prompts; no causal diagnosis or severity ranking.
Review items, not independent failures or summed events. Metrics and phases may overlap.

## Review items

### Following explicit counterfactual context

Cohort: faitheval:58d35840e3fbc7bf4b7672582a417b3b3a327dec:counterfactual
Native score: 60.80%
Definition: Source accuracy against the answer supported by supplied counterfactual context. Higher means context adherence, not greater world-truth factuality.

Some answers did not follow the supplied counterfactual context under the source task rule. This is not a world-truth error rate.

Follow-up experiment: Separate context-conditioned tasks from open-world factual answering. Inspect whether instructions and evidence scope are understood, retaining no-context accuracy as a distinct companion.

Scope: Historical 2024 authors-reported chart aggregates with deterministic decoding and task-specific prompts. Exact endpoint revisions, per-response counts and uncertainty are not established by the charts. Unanswerable/inconsistent scores are attributed to the paper v1 strict keyword matching scheme and do not validate arbitrary abstention explanations. Counterfactual adherence rewards following the supplied fictional premise; it is not an alignment or world-truth score. Baselines have different context/prompt conditions and are companion outcomes, not causal tool effects.
Source: https://github.com/SalesforceAIResearch/FaithEval/blob/58d35840e3fbc7bf4b7672582a417b3b3a327dec/assets/model_performance_cc.png

### Recognizing conflicting context

Cohort: faitheval:58d35840e3fbc7bf4b7672582a417b3b3a327dec:inconsistent
Native score: 4.00%
Definition: Source strict-match accuracy for expressing conflict when supplied documents support different answers. Published rounded aggregate; this is not citation correctness.

Some outputs did not match the prescribed conflict answer under the source strict matching rule.

Follow-up experiment: Inspect conflicting passages and accepted conflict labels. Pair conflict detection with consistent-context useful answering and validate paraphrase handling.

Scope: Historical 2024 authors-reported chart aggregates with deterministic decoding and task-specific prompts. Exact endpoint revisions, per-response counts and uncertainty are not established by the charts. Unanswerable/inconsistent scores are attributed to the paper v1 strict keyword matching scheme and do not validate arbitrary abstention explanations. Counterfactual adherence rewards following the supplied fictional premise; it is not an alignment or world-truth score. Baselines have different context/prompt conditions and are companion outcomes, not causal tool effects.
Source: https://github.com/SalesforceAIResearch/FaithEval/blob/58d35840e3fbc7bf4b7672582a417b3b3a327dec/assets/model_performance_ic.png

### Recognizing unanswerable context

Cohort: faitheval:58d35840e3fbc7bf4b7672582a417b3b3a327dec:unanswerable
Native score: 7.40%
Definition: Source strict-match accuracy for expressing unknown when supplied context lacks the answer. Published rounded aggregate; answerability is relative to supplied context.

Some outputs did not match the prescribed unknown answer under the source strict matching rule.

Follow-up experiment: Review answerability and exact-match failures alongside original-context answer accuracy. Use answerable controls to distinguish appropriate caution from blanket abstention.

Scope: Historical 2024 authors-reported chart aggregates with deterministic decoding and task-specific prompts. Exact endpoint revisions, per-response counts and uncertainty are not established by the charts. Unanswerable/inconsistent scores are attributed to the paper v1 strict keyword matching scheme and do not validate arbitrary abstention explanations. Counterfactual adherence rewards following the supplied fictional premise; it is not an alignment or world-truth score. Baselines have different context/prompt conditions and are companion outcomes, not causal tool effects.
Source: https://github.com/SalesforceAIResearch/FaithEval/blob/58d35840e3fbc7bf4b7672582a417b3b3a327dec/assets/model_performance_un.png

## Missing evaluation coverage

Honesty: belief_revision, citation_integrity, factual_accuracy, pressure_resistance
Alignment: authorized_goal_following, constraint_compliance, non_deception, oversight_and_correction, robustness

Missing coverage is not evidence of failure. Presence is not complete dimension assessment.

## Unmeasured metrics

None among the diagnostic rules.

## Reported endpoints

Endpoints are rounded source observations, not certification.

None among the diagnostic rules.

## Evaluation use

Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies.

No Honesty Mark awarded. This report digest does not authenticate an issuer.
Report digest: dac1808751703c285e44b7b7dea16d1caf1a41c5dc9ffd8322a0f211b85754fe