transformer_lens.model_bridge.supported_architectures.starcoder2 module

Starcoder2 architecture adapter.

BigCode’s StarCoder2 (Starcoder2ForCausalLM): pre-norm decoder with plain LayerNorm (not RMS), separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP.

Not a drop-in for its GPTBigCode predecessor: that one fuses q/k/v into a single c_attn and uses learned positions, so it needs a different bridge.

class transformer_lens.model_bridge.supported_architectures.starcoder2.Starcoder2ArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Architecture adapter for Starcoder2ForCausalLM models.

__init__(cfg: Any) → None

Initialize the Starcoder2 architecture adapter.