transformer_lens.model_bridge.supported_architectures.qwen3_5_moe module

Qwen3.5-MoE architecture adapter.

Hybrid linear-attention (GatedDeltaNet) + full-attention with sparse MoE MLP (256 experts, top-8 routing, shared expert in public checkpoints). Same hybrid design as Qwen3.5 dense and the same MoE block family as Qwen3-Next.

Two adapters: text-only Qwen3_5MoeForCausalLM and the vision-language Qwen3_5MoeForConditionalGeneration (text backbone nested under model.language_model plus the Qwen3.5 vision tower).

class transformer_lens.model_bridge.supported_architectures.qwen3_5_moe.Qwen3_5MoeArchitectureAdapter(cfg: Any)

Bases: Qwen3ArchitectureAdapter

Text-only Qwen3.5-MoE: hybrid GatedDeltaNet + full attention, sparse MoE MLP.

prepare_loading(model_name: str, model_kwargs: dict) → None

Swap to text_config so AutoModelForCausalLM loads the text-only model (checkpoints ship the ConditionalGeneration architecture).

prepare_model(hf_model: Any) → None

Reject full multimodal Qwen3.5-MoE models on this text-only adapter.

preprocess_weights(state_dict: dict[str, Tensor]) → dict[str, Tensor]

Slice query half from gated q_proj.weight for weight-space analysis.

class transformer_lens.model_bridge.supported_architectures.qwen3_5_moe.Qwen3_5MoeMultimodalArchitectureAdapter(cfg: Any)

Bases: Qwen3_5MultimodalArchitectureAdapter

Vision-language adapter for Qwen3_5MoeForConditionalGeneration.

Reuses the Qwen3.5 multimodal wiring (language model under model.language_model + vision tower) with the MLP swapped for sparse MoE.