transformer_lens.model_bridge.supported_architectures.glm4_moe module

GLM-4.5 MoE architecture adapter.

Supports GLM-4.5/4.6/4.7 mixture-of-experts families (Glm4MoeForCausalLM).

Key features: - RMSNorm with partial pre-norm layout. - RoPE-style rotary embeddings (partial RoPE supported by Hugging Face model logic). - Q/K normalization blocks (q_norm, k_norm) and GQA / MQA handling. - Sparse MoE block in model.layers[i].mlp, with optional dense-prefix layers. - QKVO rearrangements for bridge-side attention hooks.

Optional Parameters (may not exist in state_dict):

• blocks.{i}.mlp.gate - absent on dense-prefix layers before sparse MoE starts.

class transformer_lens.model_bridge.supported_architectures.glm4_moe.Glm4MoeArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Architecture adapter for GLM-4.5 / 4.6 / 4.7 MoE decoder models.

GLM-4x MoE families use RMSNorm, RoPE and sparse routing, with early dense-MLP layers in some checkpoints. The dense layers are represented by a present-but-slightly-thinner mlp sub-module where routing is absent.

__init__(cfg: Any) → None

Initialize the GLM-4 MoE architecture adapter.

class transformer_lens.model_bridge.supported_architectures.glm4_moe.Glm4MoeRouterBridge(*args: Any, logits_index: int = 0, **kwargs: Any)

Bases: MoERouterBridge

Tuple-preserving router bridge for Glm4MoeTopkRouter.