transformer_lens.utilities.statistics module

Model-free statistics over a per-example axis of an analysis result.

Pure torch, no model access: standard errors, a vectorized percentile bootstrap and a paired sign-flip permutation test, plus the seeded-generator convention shared with sparse probing so every consumer derives independent, reproducible streams the same way.

transformer_lens.utilities.statistics.bootstrap_ci(x: Tensor, *, dim: int = -1, statistic: str = 'mean', confidence: float = 0.95, n_resamples: int = 1000, generator: Generator | None = None, chunk_size: int = 256) → Tuple[Tensor, Tensor]

Percentile bootstrap interval of statistic along dim, vectorized over the rest.

Resamples with replacement using a CPU generator (derive_generator(0) when omitted), chunk_size resamples at a time so memory stays at chunk_size × n per leading cell. Returns (low, high) with dim removed.

A percentile bootstrap cannot express uncertainty a sample does not contain: a constant vector resamples to itself and the interval collapses to a point whatever n is, and intervals on 4-8 samples are wide and skewed. For binary success rates prefer an exact interval (success_rate_ci()).

transformer_lens.utilities.statistics.derive_generator(seed: int, *labels: object) → Generator

CPU generator keyed on seed and any labels (metric name, arm, repeat…).

SHA-256 rather than hash keeps the derived seed stable across processes; the mask keeps it in torch’s accepted int64 range.

transformer_lens.utilities.statistics.sign_flip_permutation_pvalue(effects: Tensor, *, dim: int = -1, n_permutations: int = 1000, generator: Generator | None = None, alternative: str = 'two-sided', exact_max_n: int = 12) → Tensor

Paired sign-flip permutation p-value that the mean effect along dim is zero.

The null is that each per-example effect is symmetric about zero, so every pattern of sign flips is equally likely. With n <= exact_max_n examples all 2**n patterns are enumerated and the p-value is exact and deterministic; otherwise n_permutations random patterns are drawn from the CPU generator and the p-value is (hits + 1) / (n_permutations + 1). alternative is "two-sided" (|null| >= |observed|), "greater" or "less". Returns a tensor with dim removed.

transformer_lens.utilities.statistics.standard_error(x: Tensor, *, dim: int = -1) → Tensor

Standard error of the mean along dim (unbiased std over sqrt(n)); needs n >= 2.