# Diagnostic report: gpt-4o-mini

Profile: hm-f88aadd2968dc59601f8
Profile digest: af3ae34008a6aa60b50b2610434e386d845211fef1fca1bb293dc5ba6033584a

Deterministic source-specific review prompts; no causal diagnosis or severity ranking.
Review items, not independent failures or summed events. Metrics and phases may overlap.

## Review items

### Following explicit counterfactual context

Cohort: faitheval:58d35840e3fbc7bf4b7672582a417b3b3a327dec:counterfactual
Native score: 50.90%
Definition: Source accuracy against the answer supported by supplied counterfactual context. Higher means context adherence, not greater world-truth factuality.

Some answers did not follow the supplied counterfactual context under the source task rule. This is not a world-truth error rate.

Follow-up experiment: Separate context-conditioned tasks from open-world factual answering. Inspect whether instructions and evidence scope are understood, retaining no-context accuracy as a distinct companion.

Scope: Historical 2024 authors-reported chart aggregates with deterministic decoding and task-specific prompts. Exact endpoint revisions, per-response counts and uncertainty are not established by the charts. Unanswerable/inconsistent scores are attributed to the paper v1 strict keyword matching scheme and do not validate arbitrary abstention explanations. Counterfactual adherence rewards following the supplied fictional premise; it is not an alignment or world-truth score. Baselines have different context/prompt conditions and are companion outcomes, not causal tool effects.
Source: https://github.com/SalesforceAIResearch/FaithEval/blob/58d35840e3fbc7bf4b7672582a417b3b3a327dec/assets/model_performance_cc.png

### Recognizing conflicting context

Cohort: faitheval:58d35840e3fbc7bf4b7672582a417b3b3a327dec:inconsistent
Native score: 57.00%
Definition: Source strict-match accuracy for expressing conflict when supplied documents support different answers. Published rounded aggregate; this is not citation correctness.

Some outputs did not match the prescribed conflict answer under the source strict matching rule.

Follow-up experiment: Inspect conflicting passages and accepted conflict labels. Pair conflict detection with consistent-context useful answering and validate paraphrase handling.

Scope: Historical 2024 authors-reported chart aggregates with deterministic decoding and task-specific prompts. Exact endpoint revisions, per-response counts and uncertainty are not established by the charts. Unanswerable/inconsistent scores are attributed to the paper v1 strict keyword matching scheme and do not validate arbitrary abstention explanations. Counterfactual adherence rewards following the supplied fictional premise; it is not an alignment or world-truth score. Baselines have different context/prompt conditions and are companion outcomes, not causal tool effects.
Source: https://github.com/SalesforceAIResearch/FaithEval/blob/58d35840e3fbc7bf4b7672582a417b3b3a327dec/assets/model_performance_ic.png

### Recognizing unanswerable context

Cohort: faitheval:58d35840e3fbc7bf4b7672582a417b3b3a327dec:unanswerable
Native score: 52.30%
Definition: Source strict-match accuracy for expressing unknown when supplied context lacks the answer. Published rounded aggregate; answerability is relative to supplied context.

Some outputs did not match the prescribed unknown answer under the source strict matching rule.

Follow-up experiment: Review answerability and exact-match failures alongside original-context answer accuracy. Use answerable controls to distinguish appropriate caution from blanket abstention.

Scope: Historical 2024 authors-reported chart aggregates with deterministic decoding and task-specific prompts. Exact endpoint revisions, per-response counts and uncertainty are not established by the charts. Unanswerable/inconsistent scores are attributed to the paper v1 strict keyword matching scheme and do not validate arbitrary abstention explanations. Counterfactual adherence rewards following the supplied fictional premise; it is not an alignment or world-truth score. Baselines have different context/prompt conditions and are companion outcomes, not causal tool effects.
Source: https://github.com/SalesforceAIResearch/FaithEval/blob/58d35840e3fbc7bf4b7672582a417b3b3a327dec/assets/model_performance_un.png

## Missing evaluation coverage

Honesty: belief_revision, citation_integrity, factual_accuracy, pressure_resistance
Alignment: authorized_goal_following, constraint_compliance, non_deception, oversight_and_correction, robustness

Missing coverage is not evidence of failure. Presence is not complete dimension assessment.

## Unmeasured metrics

None among the diagnostic rules.

## Reported endpoints

Endpoints are rounded source observations, not certification.

None among the diagnostic rules.

## Evaluation use

Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies.

No Honesty Mark awarded. This report digest does not authenticate an issuer.
Report digest: 3120c8846aad8701cd378e80bf52fe4b89f8bb831e789dc2e00f68a2d6b6e92d