transformer_lens.model_bridge.supported_architectures.qwen2_moe module

Qwen2-MoE architecture adapter.

class transformer_lens.model_bridge.supported_architectures.qwen2_moe.Qwen2MoeArchitectureAdapter(cfg: Any)

Bases: Qwen2ArchitectureAdapter

Architecture adapter for Qwen2-MoE models.

Qwen2-MoE uses the Qwen2 attention stack plus a sparse MoE MLP with an always-on shared expert path.

__init__(cfg: Any) → None

Initialize the Qwen2-MoE architecture adapter.

class transformer_lens.model_bridge.supported_architectures.qwen2_moe.Qwen2MoeRouterBridge(*args: Any, logits_index: int = 0, **kwargs: Any)

Bases: MoERouterBridge

Tuple-preserving router bridge for Qwen2MoeTopKRouter.