transformer_lens.model_bridge.supported_architectures.llama4 module

Llama 4 (text) architecture adapter.

Meta’s Llama 4 text decoder (Llama4ForCausalLM): llama-style RMS-norm blocks whose attention adds complex-valued interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks — so attention stays delegated to HF. The feed-forward is either a sparse MoE (batched 3D experts + top-k sigmoid router + shared expert) or a dense gated MLP on non-MoE layers.

class transformer_lens.model_bridge.supported_architectures.llama4.Llama4ArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Architecture adapter for Llama4ForCausalLM models.

__init__(cfg: Any) → None

Initialize the Llama 4 architecture adapter.