# Diagnostic report: gpt-4o-mini-2024-07-18

Profile: hm-bdf3c84a84600f8baf69
Profile digest: 714553c0cc7d58a1bedd49762e23cb48efd999436dfad7624bd8fcf4aebfe349

Deterministic source-specific review prompts; no causal diagnosis or severity ranking.
Review items, not independent failures or summed events. Metrics and phases may overlap.

## Review items

### Short-form factual accuracy

Cohort: simpleqa:652c89d0ca9df547706735883097e9537d40dc47:provider-reported:assistant
Native score: 9.50%
Definition: Source-reported percentage graded correct on SimpleQA. This table does not provide incorrect or not-attempted rates; 100 minus accuracy is not a hallucination rate.

Some answers did not receive correct-answer credit under the provider-reported SimpleQA grading. The table does not separate incorrect answers from non-attempts.

Follow-up experiment: Reproduce the stated model and prompt setting on fresh fact questions. Report correct, incorrect and not-attempted outcomes together; validate the grader before interpreting factual error as overclaiming.

Scope: Historical OpenAI-reported results, not an independent Honesty.tools rerun. Prompt labels form separate cohorts. Per-row judged sample counts, repeats, uncertainty and abstention breakdown unavailable; no count or interval is inferred from rounded percentages.
Source: https://github.com/openai/simple-evals/blob/652c89d0ca9df547706735883097e9537d40dc47/README.md#benchmark-results

## Missing evaluation coverage

Honesty: belief_revision, citation_integrity, evidence_support, pressure_resistance, uncertainty
Alignment: authorized_goal_following, constraint_compliance, non_deception, oversight_and_correction, robustness

Missing coverage is not evidence of failure. Presence is not complete dimension assessment.

## Unmeasured metrics

None among the diagnostic rules.

## Reported endpoints

Endpoints are rounded source observations, not certification.

None among the diagnostic rules.

## Evaluation use

Use public development examples for diagnosis; reserve independent held-out tasks for evaluating any change. These suggestions are experiments, not proven training remedies.

No Honesty Mark awarded. This report digest does not authenticate an issuer.
Report digest: 0236fc2385aa85fa71801eb172d3c66aa6daff6bc1daad50c2c55a613f6b9e35