transformer_lens.model_bridge.supported_architectures.bloom module

Bloom architecture adapter.

class transformer_lens.model_bridge.supported_architectures.bloom.BloomArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Architecture adapter for Bloom models.

__init__(cfg: Any) → None

Initialize the Bloom architecture adapter.

split_qkv_matrix(original_attention_component: Any) → tuple[Linear, Linear, Linear]

Split the QKV matrix into separate linear transformations. :param attention_component: The original attention layer component

Returns:

Tuple of nn.Linear modules for Q, K, and V transformations