transformer_lens.model_bridge.supported_architectures.deepseek_v2 module

DeepSeek V2 adapter and the shared DeepSeek-MLA family base (DeepSeek V2/V3, GLM-MoE-DSA, GLM-4.7-Flash; Youtu via V2); per-member differences are declarative.

DeepSeek V2 support covers DeepSeek-V2, DeepSeek-V2-Lite, and DeepSeek-Coder-V2 (all use DeepseekV2ForCausalLM).

class transformer_lens.model_bridge.supported_architectures.deepseek_v2.DeepSeekMLAFamilyArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Shared base for DeepSeek-style Multi-head Latent Attention + MoE decoders.

Members share LoRA-compressed attention (q_a/q_b and kv_a/kv_b projections with decoupled RoPE), RMSNorm, no biases, MLA weights kept in HF layout (no QKVO rearrangements, no LN folding), and a routed MoE whose router/shared experts are absent on dense layers. Divergence is declarative:

• q_lora_optional: some checkpoints (DeepSeek-V2-Lite; GigaChat3 on the V3 class) set q_lora_rank=None — no compressed-Q pair, direct q_proj instead — so the whole Q path is optional; _build_q_a_layernorm picks the norm’s type. Families whose checkpoints always compress Q keep the projections REQUIRED so a genuinely missing weight fails loudly; never relax them family-wide.

• attention_cls: GLM-MoE-DSA swaps in its sparse-attention bridge.

• _build_mlp_bridge / _build_router seams: Youtu maps all-dense MLPs; GLM-4.7-Flash uses a MoERouterBridge router.

__init__(cfg: Any) → None

Apply the family config knobs and build the shared mapping.

attention_cls

alias of MLAAttentionBridge

eager_attention: bool = False

prepend_bos: bool | None = None

q_lora_optional: bool = False

class transformer_lens.model_bridge.supported_architectures.deepseek_v2.DeepSeekV2ArchitectureAdapter(cfg: Any)

Bases: DeepSeekMLAFamilyArchitectureAdapter

Architecture adapter for DeepSeek V2 / V2-Lite / Coder-V2 models.

Uses RMSNorm, MLA with compressed Q/KV projections (or direct Q projection when q_lora_rank is None), partial RoPE, MoE on most layers (dense MLP on first few), and no biases.

component_mapping: ComponentMapping | None

q_lora_optional: bool = True

uses_split_attention: bool

weight_processing_conversions: Dict[str, ParamProcessingConversion | str] | None