# EXPLORATORY, NOT PRE-REGISTERED: HLE words/Humanities cell after the shuffle-control failure

Written 2026-10-27T07:36:28-06:00 (MDT). **Descriptive only. Not a nulltest run, not a re-score, no new design.** B1's verdict stays **VOID ("control failed")**.
Inputs: B1_items.jsonl `7b54b10bea11f1493b5263da5c5a2a3c2a3fc996b56be8e067f536206af7b45d`, B1_shuffled.jsonl `52e522fd2108598f3ba69435b78c9286ab043bfae8866702f615613df201ef98`. Script work/hle_explore.py
`5eb18316d397fd3321e1ee91ec23e7d2fc39a93f92f0ab2ba69ef42b32a81351`; full numbers in exploratory/hle_shuffle_stats.json `c368cf5c339401d69bbdcc0d8a3cfbb8a963b53d77a9a377e46b364d0c4e7295` (internal, aggregates only, no item text).

Hypothesis tested: the cell fires because of **between-question composition** (each question gives 1 keyed and k−1 wrong options, so a
question's label rate is fixed by k, and a within-question shuffle preserves it), not because of any keyed-vs-wrong difference within a question.

## Option-count distribution, Humanities/Social Science (76 questions, 501 items: 76 keyed / 425 not)
k = 5: 46 · 6: 8 · 7: 6 · 8: 7 · 9: 2 · 10: 2 · 11: 1 · 12: 2 · 26: 2 (mean 6.59, median 5).
Questions with ≥10 options are 7 of 76 questions but 21.4% of items and 23.5% of the wrong-option items.
Option count of the item's own question alone separates wrong from keyed at **AUROC 0.614** (identical, 0.614, in the shuffled file, by construction).

## Length / lexical features, Humanities (keyed vs wrong)
| feature | mean keyed | mean wrong | pooled AUROC (wrong vs keyed) | mean within-question AUROC | within-q keyed larger/smaller/tie | mean by k band 5 / 6–9 / 10+ |
|---|---|---|---|---|---|---|
| words | 16.053 | 13.464 | 0.467 | 0.4896 | 31/33/12 | 17.787 / 14.47 / 4.467 |
| chars | 95.553 | 81.944 | 0.4724 | 0.5228 | 34/39/3 | 108.13 / 87.951 / 26.112 |
| mean_word_len | 4.983 | 5.156 | 0.5485 | 0.6006 | 25/47/4 | 5.535 / 5.133 / 4.255 |
| digit_frac | 0.05 | 0.084 | 0.5223 | 0.4911 | 6/9/61 | 0.038 / 0.016 / 0.262 |
| upper_frac | 0.064 | 0.064 | 0.4799 | 0.4583 | 34/31/11 | 0.053 / 0.078 / 0.065 |

Shared wording with the other options of the same question: keyed 0.436, wrong 0.463 (AUROC 0.5197).
Share of an item's word unigrams+bigrams that appear in no other question's options: 0.702 (all categories: 0.486).
Shuffled file, same cell: words pooled AUROC 0.4526, within-question 0.4569.

## Reading
- **Supports the hypothesis.** Within a question, keyed and wrong options are close to indistinguishable on length (word-count within-question AUROC 0.49,
  median difference 0; keyed longer in 31 questions, shorter in 33). The one within-question tilt is mean word length (keyed has shorter words in 47 of 72 questions, within-q AUROC 0.60), small and not what the `words` probe measures.
- **Between questions, the differences are large:** options of 10+-option questions average 4.5 words against 17.8 for 5-option questions, and those questions carry a higher wrong-option share.
  70% of an item's tokens are unique to its question, so a bag-of-words model trained on item-level folds can learn each question's vocabulary,
  and with it that question's label rate (k−1)/k. A within-question shuffle keeps all of that, which is why the control fired in the same cell.
- **Limits:** option count alone gives AUROC 0.61 while the cell's A* is 0.70; the extra is consistent with question-specific vocabulary leaking across folds, but this note
  does not test that (doing so would mean re-scoring under a new design, which is out of scope). Nothing here changes or rescues the B1 verdict. A fix (e.g. a design with a constant label rate per question, or group-aware folds) would be batch 2 with its own prereg.