# FALSIFIER — Goodfire feasibility toy claim (pre-registered)

Written: 2026-10-27T09:56:23Z (2026-10-27 03:56 MDT, America/Denver). Written before any generation, steering, or metric run.
Author: Ford (Grok Bot executor) for Honesty.tools. Internal feasibility only. Not anchored to Rekor (that would be an outside post).

## 0. Why this claim
Goodfire's "Painting with concepts using diffusion model latents" (May 27, 2025, https://goodfire.ai/blog/painting-with-concepts)
names SDXL-Turbo SAE features with causal/steering effects, e.g. *"opening one's mouth" (feature 7446)*,
"putting hands on one's face in distress" (4045), "hugging" (4071), "squaring up to fight" (9820),
and says painting with them changes the generated image (e.g. "Here we paint 'animal opening its mouth' onto the dragon").
SAE (MIT) and SDXL-Turbo (Stability AI Community License, ungated) are both openly downloadable, so we re-run
that published claim. The Llama-3.1-8B SAE feature labels are only exposed through the Ember API, and the R1 SAEs
need the 671B model, so neither gives a re-runnable named-feature claim on open weights.

## 1. Claim under test
**H1:** Steering SDXL-Turbo at block `unet.down_blocks[2].attentions[1]` along Goodfire SAE feature **F = 7446**
(published label "opening one's mouth") over the central 8x8 of the 16x16 patch grid increases the CLIP-measured
"mouth open" score of generated animal images by at least **X = 2.0** CLIP logit units on held-out prompts,
beyond random features of matched activation density.

## 2. Pinned artifacts
- Base model: stabilityai/sdxl-turbo @ 71153311d3dbb46851df1931d3ca6e939de83304 (fp16 unet/vae/text encoders)
- SAE: Goodfire/SDXL-Turbo-SAE-ldown.attns.2.1 @ 832a46568a533925f2caf1db7cffa2928918218d
  (BatchTopK, d_in 1280, d_hidden 10240, k=10; loader = goodfire-ai/sdxl-turbo-interpretability sae.py)
- Metric model: laion/CLIP-ViT-H-14-laion2B-s32B-b79K @ 1c2b8495b28150b8a4922ee1c8edee224c284c0c (MIT)
- Generation: 512x512, 1 inference step, guidance_scale 0.0, fp16, EulerAncestral scheduler from the repo.
- sha256 of every weight file is recorded in harness/MANIFEST.json and must match HF LFS sha256.

## 3. Prompt sets (fixed now; files hashed in RECEIPT.json)
- Held-out test prompts (20): "a photo of a {a}" for a in
  dog, cat, lion, tiger, wolf, bear, horse, crocodile, shark, hippopotamus, monkey, fox, rabbit, cow, frog, parrot, eagle, snake, seal, otter.
- Calibration prompts (5, disjoint, used only for hook-site check and steering strength): dragon, leopard, panda, goat, dolphin.
- Density reference prompts (30, disjoint from test): 15 animal-free scenes + 15 calibration-style
  animal/person prompts listed in harness/prompts.json.
- Seeds: 0, 1, 2, 3, 4 (initial latent noise via torch.Generator on CUDA; random-control draws use seed 12345).

## 4. Metric
CLIP score per image: S = 100 * [cos(img, "a photo of an animal with its mouth wide open") - cos(img, "a photo of an animal with its mouth closed")].
Subject score: T = 100 * cos(img, "a photo of a {a}").
Paired effect for an intervention j: d_j(p, s) = S(steered_j, p, s) - S(baseline, p, s). E_j = mean of d_j over 20 prompts x 5 seeds.

## 5. Intervention
Add v_j = m * W_dec[:, j] / ||W_dec[:, j]|| to the output hidden states of down_blocks[2].attentions[1]
at the 64 central patches (rows 4-11, cols 4-11 of 16x16), in the single UNet evaluation.
Same m for every feature (F, random controls, planted). m = s * median ||h|| over patches of calibration baselines,
with s chosen from {0.5, 1.0, 2.0} on calibration prompts only (largest calibration E_F among s values whose
calibration subject drop is <= 5 units; ties -> smaller s). Chosen s is frozen before any test prompt is generated.

Hook-site check (calibration only): SAE reconstruction FVU is computed both for the block output and for the
block update (output - input). The SAE input convention is whichever has lower FVU. If both FVU > 0.5,
the harness is declared NOT LOADED and no test is run.

## 6. Controls
1. **Random matched-density features (20):** density = fraction of patches where the feature is in the BatchTopK
   active set, measured on the 30 density-reference baselines x seeds 0-1. Pool = live features with density in
   [0.5 d_F, 2 d_F] (if d_F = 0 or the pool < 20, use the 40 nearest in log-density). Draw 20 with seed 12345. Same m.
2. **Shuffled labels:** for each of 1,000 permutations, randomly swap steered/baseline labels within each
   (prompt, seed) pair and rerun the full decision rule. The rule must return FAIL in >= 99% of permutations;
   the sign-flip permutation p-value (10,000 flips) for E_F is reported.
3. **Planted fake feature:** a random unit vector in R^1280 (seed 777) is appended as feature 10240, given the
   registry label "opening one's mouth", and run through the identical pipeline with the same m.
   The pipeline must return FAIL for it.
4. **Seed sweep:** all numbers per seed 0-4.

## 7. Decision rule (pre-registered)
H1 PASSES only if all hold:
- P1 magnitude: E_F >= 2.0.
- P2 specificity: E_F > E_r for all 20 random features (rank 1 of 21, p <= 1/21).
- P3 shuffle: sign-flip permutation p < 0.01, and the decision rule FAILS on >= 99% of label-shuffled datasets.
- P4 seed robustness: per-seed E_F >= 1.0 and > 0 in each of seeds 0-4.
- P5 subject preservation: mean T drop (steered vs baseline) <= 5.0 units.
- Validity gate V: the planted fake feature must FAIL (any of P1-P5). If the planted feature PASSES, the run is
  **INVALID** (pipeline cannot reject a fake), regardless of F.
Outcome is one of PASS / FAIL (listing which of P1-P5 failed) / INVALID / NOT RUN.
Anything not listed here (other features, global steering, other s values) is exploratory and cannot change the verdict.