transformer_lens.model_bridge.supported_architectures.longt5 module

LongT5 architecture adapter.

Google’s LongT5 (LongT5ForConditionalGeneration): a T5 stack whose encoder self-attention is replaced by local windowed attention or transient-global attention (encoder_attention_type). The decoder is identical to T5. Encoder attention stays delegated to HF — its block-wise position bias has a [1, 1, heads, block, 3*block] shape the generic reconstruction cannot supply.

class transformer_lens.model_bridge.supported_architectures.longt5.LongT5ArchitectureAdapter(cfg: Any)

Bases: T5ArchitectureAdapter

Architecture adapter for LongT5ForConditionalGeneration models.

__init__(cfg: Any) → None

Initialize the LongT5 architecture adapter.