transformer_lens.utilities.statistics module¶

Model-free statistics over a per-example axis of an analysis result.

No model access: standard errors, a vectorized percentile bootstrap, a paired sign-flip permutation test and an exact Clopper-Pearson interval for success rates, plus the seeded-generator convention shared with sparse probing so every consumer derives independent, reproducible streams the same way.

class transformer_lens.utilities.statistics.SuccessRateInterval(point_estimate: float, ci_low: float, ci_high: float, n_trials: int, n_successes: int, confidence: float)¶

Bases: object

An exact Clopper-Pearson confidence interval around a success rate.

n_trials and n_successes are the counts behind the bounds.

ci_high: float¶

ci_low: float¶

confidence: float¶

n_successes: int¶

n_trials: int¶

point_estimate: float¶

transformer_lens.utilities.statistics.bootstrap_ci(x: Tensor, *, dim: int = -1, statistic: str = 'mean', confidence: float = 0.95, n_resamples: int = 1000, generator: Generator | None = None, chunk_size: int = 256) → Tuple[Tensor, Tensor]¶

Percentile bootstrap interval of statistic along dim, vectorized over the rest.

Resamples with replacement using a CPU generator (derive_generator(0) when omitted), chunk_size resamples at a time so memory stays at chunk_size × n per leading cell. Returns (low, high) with dim removed.

A percentile bootstrap cannot express uncertainty a sample does not contain: a constant vector resamples to itself and the interval collapses to a point whatever n is, and intervals on 4-8 samples are wide and skewed. For binary success rates prefer an exact interval (success_rate_ci()).

transformer_lens.utilities.statistics.derive_generator(seed: int, *labels: object) → Generator¶

CPU generator keyed on seed and any labels (metric name, arm, repeat…).

SHA-256 rather than hash keeps the derived seed stable across processes; the mask keeps it in torch’s accepted int64 range.

transformer_lens.utilities.statistics.sign_flip_permutation_pvalue(effects: Tensor, *, dim: int = -1, n_permutations: int = 1000, generator: Generator | None = None, alternative: str = 'two-sided', exact_max_n: int = 12) → Tensor¶

Paired sign-flip permutation p-value that the mean effect along dim is zero.

The null is that each per-example effect is symmetric about zero, so every pattern of sign flips is equally likely. With n <= exact_max_n examples all 2**n patterns are enumerated and the p-value is exact and deterministic; otherwise n_permutations random patterns are drawn from the CPU generator and the p-value is (hits + 1) / (n_permutations + 1). alternative is "two-sided" (|null| >= |observed|), "greater" or "less". A null mean within a relative 100 * eps of the observed one counts as a tie, absorbing summation-order error; there is no absolute floor, so near-zero observed effects get essentially no tolerance. Returns a tensor with dim removed.

transformer_lens.utilities.statistics.standard_error(x: Tensor, *, dim: int = -1) → Tensor¶: Standard error of the mean along dim (unbiased std over sqrt(n)); needs n >= 2.

transformer_lens.utilities.statistics.success_rate_ci(successes: Sequence[bool], *, confidence: float = 0.95) → SuccessRateInterval¶

Computes an exact Clopper-Pearson confidence interval for a success rate.

Unlike a percentile bootstrap, which collapses to [0, 0] on an all-failure sample regardless of size, the interval widens as the trial count shrinks ([0.0, 0.459] for zero successes out of six). Deterministic: no resampling and no seed.

Raises:: ValueError – If successes is empty.