# RATING_PREREG: blind human rating of feature 7446 (mouth opening)

Written: 2026-10-27T11:33:45Z (2026-10-27 05:33 MDT, America/Denver). Written before anything was put in the blind folder and before key.json existed.
Internal only. This is a separate check from the CLIP and SigLIP verdicts (amendments 1-3); it doesn't change them.

## Stimuli
- 100 real pairs: for each of the 20 test animals x seeds 0-4, the baseline image and its feature-7446 steered image
  (s = 2.0, same seed, same prompt), from raw/test/img on the DGX. Nothing new is generated.
- 8 attention-check pairs: the same image shown on both sides. The source (prompt, seed, and baseline or steered) is drawn with the
  shuffle seed below; 4 are baselines and 4 are steered images.
- Pair order and left/right placement are shuffled with a fixed seed: **20261027** (numpy default_rng).
  Public names are pair_001 ... pair_108, assigned after shuffling, so checks and real pairs share one numbering scheme.
  Internal source ids (real_pXX_sY, check_cZZ) appear only in the key.
- Images: 384x384 JPEG, quality 90, 4:2:0, the same encoder settings for every file, with no EXIF, ICC or other metadata.
  Every file is padded to one common byte size with zero-filled JPEG comment (COM) segments, and every file gets the same mtime.
  So neither filename, metadata nor file size shows which side is steered.
- Key: ~/honesty-tools/runs/goodfire-feasibility/blind_key/key.json on the DGX only, never mirrored.
  Its sha256 and this file's sha256 are recorded in blind/SEAL.md.

## Rater task
For each pair: **"Which image shows the animal's mouth more open?"** Forced choice, Left or Right, no ties.
The rater sees only the two images and the question.
Raters should not be the person who built the packet. Anyone who has seen the contact sheets is noted as an unblinded-exposure caveat.

## Primary measure and test (per rater)
- Primary measure: the proportion of the 100 real pairs where the steered image is chosen.
- Test: one-sided exact binomial test against p = 0.5, alpha = 0.01. That needs >= 63 of 100 (P = 0.006).
- Missing answers: if more than 5 real pairs are unanswered, the rater is INVALID. Otherwise each missing answer counts as
  "steered not chosen" (conservative), with n kept at 100.

## Attention-check exclusion (per rater)
On the 8 identical pairs, choosing the same side >= 7 of 8 times counts as a strong left/right bias, and the rater is EXCLUDED.
An unbiased rater is wrongly excluded with probability 0.07. Results from excluded raters are reported but not counted.
Separately, the overall left-choice rate on real pairs is reported (descriptive only).

## Pass threshold and verdict
- A rater PASSES if the proportion is >= 0.70 (>= 70 of 100) and the one-sided binomial p < 0.01.
- Overall PASS requires at least 1 non-excluded rater, and every non-excluded rater passing.
  FAIL if any non-excluded rater fails. INCONCLUSIVE if every rater is excluded or INVALID.
- Every rater's result is reported, including excluded ones.