# FALSIFIER_FEATURES_2: re-checking Goodfire SDXL-Turbo SAE features 4045, 4071 and 9820 under the amendment-3 protocol

Written: 2026-10-27T11:43:52Z (2026-10-27 05:43 MDT, America/Denver). Written before any generation, calibration or scoring for these three features.
Internal feasibility only: no Rekor anchoring and no outside contact. Parent protocol: ../goodfire-feasibility/FALSIFIER_AMENDMENT_3.md
(sha256 86e6449f...). Everything below is fixed now. Anything not listed here is exploratory and cannot change a verdict.

## 0. Claims under test (Goodfire blog "Painting with concepts", May 27 2025)
- **4045** "putting hands on one's face in distress"
- **4071** "hugging"
- **9820** "squaring up to fight"
For each feature j, **H_j**: steering along unit(W_dec[:, j]) at the same site and region as feature 7446 raises that feature's
concept score on held-out prompts, beyond 20 random features of matched density. Each feature gets its own verdict, and every verdict is reported, including FAIL.

## 1. Fixed pieces carried over unchanged
- Same pinned models: SDXL-Turbo @71153311, SAE @832a4656, CLIP-H @1c2b8495, SigLIP so400m @9fdffc58, Grounding DINO tiny @a2bb814d (sha256 checked).
- Same generation settings (512 px, 1 step, guidance 0, fp16, CPU-generator latents per DEVIATION_1) and seeds 0-4.
- Same hook (unet.down_blocks[2].attentions[1] output), same region (central 8x8 of the 16x16 grid) and the same planted fake
  (random unit vector, torch seed 777, raw/calib/planted_vector.pt).
- Same median patch norm, 145.807 (raw/calib/calib.json); m_j = s_j x 145.807.

## 2. Per-feature design
| | 4045 hands on face, distress | 4071 hugging | 9820 squaring up to fight |
|---|---|---|---|
| Held-out test prompts (20) | set PEOPLE | set PAIRS | set PAIRS |
| Calibration prompts (5, disjoint) | set PEOPLE_CAL | set PAIRS_CAL | set PAIRS_CAL |
| Scorer positive text (CLIP-H and SigLIP, same text) | "a photo of a person with their hands on their face in distress" | "a photo of two figures hugging each other" | "a photo of two figures squaring up to fight each other" |
| Scorer negative text | "a photo of a calm person with their hands away from their face" | "a photo of two figures standing apart, not touching" | "a photo of two figures standing calmly side by side" |
| Detector (Grounding DINO, box/text thr 0.25, top box) | prompt "person head. human face." | none: full image | none: full image |
| Crop | square, 2.0 x the box's max side (to take in hands at the face), clipped to the image | full image | full image |

Prompt sets (neutral poses, no verbs naming the concept):
- PEOPLE (test): "a photo of a {x}" for x in man, woman, old man, old woman, young man, young woman, boy, girl, businessman, chef, nurse,
  farmer, student, teacher, soldier, firefighter, doctor, construction worker, musician, pilot.
- PEOPLE_CAL: scientist, police officer, waiter, dancer, mechanic.
- PAIRS (test): "a photo of two {x}" for x in people, men, women, children, old men, dogs, cats, bears, monkeys, horses, lions, rabbits,
  elephants, foxes, wolves, penguins, deer, kangaroos, goats, ducks.
- PAIRS_CAL: friends, gorillas, sheep, tigers, pandas.
4071 and 9820 share the PAIRS prompts and baselines, but each has its own random controls and its own m.

## 3. Steering strength (calibration prompts only, frozen before any test image)
- Grid s in {0.5, 1.0, 2.0, 4.0}. 4.0 is added to the 7446 grid in case these features are weaker. This is fixed now.
- Calibration effect E_cal(s) = mean CLIP concept-score delta (full image, the feature's text pair) over 5 calibration prompts x 5 seeds.
- Subject drop = mean of 100 x [cos(baseline, prompt) - cos(steered, prompt)] in CLIP-H.
- Chosen s = the largest E_cal among s with subject drop <= 5.0, with ties going to the smaller s. If no s qualifies, or the chosen E_cal <= 0, the feature
  still runs at the s with the largest E_cal among s with subject drop <= 5.0. If none qualifies, the verdict is NOT RUN (subject destroyed), and it is reported.

## 4. Controls
- Random matched density: 20 features per target, using the same rule as calib.py on the existing raw/calib/density.npy.
  The pool is live features with density in [0.5 d_j, 2 d_j], or the 40 nearest in log-density if that gives fewer than 20 (or d_j = 0).
  The four named features (7446, 4045, 4071, 9820) are excluded. A fresh np.random.default_rng(12345) draws the 20 for each target. They use the same m_j.
- Planted fake: the seed-777 vector at m_j, carrying the feature's label, scored identically.

## 5. Analysis and decision (same as amendment 3, per feature and per scorer)
- dS = S(steered) - S(baseline), paired on (prompt, seed).
- Pooled OLS: dS = b_k + c x dlog_area over all 22 interventions.
  - For 4045 the area comes from the detector box.
  - For 4071 and 9820 (full image) dlog_area = 0, so the c term is dropped and b_k is the mean dS.
- Bootstrap over the 20 prompts: 2,000 resamples, seed 2027.
- R1: CLIP b_j >= 1.0; SigLIP b_j >= 0.5 x SD of baseline SigLIP scores.
- R2: b_j ranks 1 of 21 (target vs 20 random).
- R3: bootstrap 2.5th percentile of b_j > 0.
- R4: matched-size subset (|dlog_area| <= ln 1.25) has n >= 10 and mean dS > 0. For full-image features this subset is every pair.
- A scorer passes if R1-R4 all hold (R4 "not estimable" doesn't count against it, as in amendment 3).
- Verdict, applied in this order:
  1. 4045 only: more than 20 of 100 target pairs dropped because baseline or steered has no detection → INCONCLUSIVE (coverage).
  2. The planted fake passes on either scorer → INVALID.
  3. Both scorers pass → PASS.
  4. Neither passes → FAIL.
  5. Otherwise → INCONCLUSIVE (scorers disagree).
- Also reported, without changing the verdict:
  - the random-feature false-positive list;
  - CLIP/SigLIP correlation of dS;
  - per-seed means;
  - coverage per prompt;
  - for 4045, a full-image sensitivity analysis;
  - for 4071 and 9820, the Grounding DINO count of "person. animal." boxes before and after steering.

## 6. Known weak points, stated in advance
- Text-pair scorers measure similarity to a caption, not the pose itself. SDXL-Turbo at 1 step often renders two-subject prompts poorly.
- A central 8x8 patch edit may land on one subject instead of "between" two. Goodfire painted user-chosen regions on NMF factors, not single features.
- The steering strength is our choice: Goodfire published no number (C_GOODFIRE_SETTINGS.md).

## 7. Paths
- DGX: ~/honesty-tools/runs/goodfire-features2/ (images stay here).
- Box mirror (small files only): /workspace/honesty-tools/runs/goodfire-features2/.