transformer_lens.model_bridge.supported_architectures.olmo_hybrid module

OLMo Hybrid architecture adapter.

AllenAI’s OLMo Hybrid (OlmoHybridForCausalLM, Olmo-Hybrid-7B): alternating layer types — OLMo2-style full-attention layers (post-norms in the residual, full-width QK-norm, NoPE mode when position embeddings are withheld) and GatedDeltaNet linear-attention layers (pre-norm, with separate q/k/v short convolutions). Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant differs from Qwen3Next’s (separate q/k/v conv states), so it is delegated opaquely rather than through GatedDeltaNetBridge’s reimplementation. Generation uses the model’s own OlmoHybridDynamicCache.

class transformer_lens.model_bridge.supported_architectures.olmo_hybrid.OlmoHybridArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Architecture adapter for OlmoHybridForCausalLM models.

__init__(cfg: Any) → None

Initialize the OLMo Hybrid architecture adapter.

component_mapping: ComponentMapping | None

create_stateful_cache(hf_model: Any, batch_size: int, device: Any, dtype: Any) → Any

OLMo Hybrid keeps per-layer q/k/v conv states in its own cache class.

supports_fold_ln = False

uses_split_attention: bool

weight_processing_conversions: Dict[str, ParamProcessingConversion | str] | None