transformer_lens.model_bridge.supported_architectures.glm4_moe_lite module

GLM-4 MoE Lite architecture adapter.

Supports the GLM-4.7-Flash family (Glm4MoeLiteForCausalLM): DeepSeek-style Multi-head Latent Attention (LoRA-compressed Q and KV, nope/rope split heads, interleaved partial RoPE) combined with GLM’s sparse MoE — sigmoid router with e_score_correction_bias, batched routed experts, one shared expert — and a per-layer dense/sparse MLP mix declared in config.mlp_layer_types.

class transformer_lens.model_bridge.supported_architectures.glm4_moe_lite.Glm4MoeLiteArchitectureAdapter(cfg: Any)

Bases: DeepSeekMLAFamilyArchitectureAdapter

GLM-4.7-Flash (Glm4MoeLiteForCausalLM) adapter: DeepSeek-V2 MLA + GLM-4-MoE routing (dense/sparse per mlp_layer_types).

component_mapping: ComponentMapping | None

prepend_bos: bool | None = False

q_lora_optional: bool = True

uses_split_attention: bool

weight_processing_conversions: Dict[str, ParamProcessingConversion | str] | None