transformer_lens.model_bridge.supported_architectures.led module

LED (Longformer Encoder-Decoder) architecture adapter.

AllenAI’s LED (LEDForConditionalGeneration: led-base/large-16384): a BART-layout post-LN encoder-decoder whose encoder self-attention is Longformer’s sliding-window + global attention (separate query/key/value and *_global projections behind an output projection). The encoder attention stays delegated to HF; the decoder is plain BART attention. The whole stack lives under the led. prefix instead of model..

Encoder caveats: HF pads inputs to a multiple of config.attention_window inside LEDEncoder.forward, so encoder-block hooks fire on window-padded sequence lengths (only the final hidden state is unpadded). hook_q/k/v cover the sliding-window projections; the global path is hookable at q_global/k_global/v_global when global attention is requested.

class transformer_lens.model_bridge.supported_architectures.led.LEDArchitectureAdapter(cfg: Any)

Bases: BartArchitectureAdapter

Architecture adapter for LEDForConditionalGeneration models.

__init__(cfg: Any) → None

Initialize the LED architecture adapter.