transformer_lens.model_bridge.supported_architectures.glm4_moe_lite module¶
GLM-4 MoE Lite architecture adapter.
Supports the GLM-4.7-Flash family (Glm4MoeLiteForCausalLM): DeepSeek-style
Multi-head Latent Attention (LoRA-compressed Q and KV, nope/rope split heads,
interleaved partial RoPE) combined with GLM’s sparse MoE — sigmoid router with
e_score_correction_bias, batched routed experts, one shared expert — and a
per-layer dense/sparse MLP mix declared in config.mlp_layer_types.
- class transformer_lens.model_bridge.supported_architectures.glm4_moe_lite.Glm4MoeLiteArchitectureAdapter(cfg: Any)¶
Bases:
ArchitectureAdapterGLM-4.7-Flash (Glm4MoeLiteForCausalLM) adapter: DeepSeek-V2 MLA + GLM-4-MoE routing (dense/sparse per mlp_layer_types).