transformer_lens.model_bridge.supported_architectures.hyenadna module

HyenaDNA architecture adapter.

HazyResearch/Stanford’s HyenaDNA (HyenaDNAForCausalLM, remote code): genomic language models built on the Hyena operator — attention-free long-convolution mixing (in_proj -> short conv + implicit modulated filters -> gated multiplicative recombination -> out_proj). Blocks are otherwise transformer-shaped (pre-LN, gelu fc1/fc2 MLP), stacked under hyena.backbone with LayerNorms and single-character DNA vocab.

The mixer delegates to HF wholesale (the implicit filter has no attention-shaped reconstruction); in_proj/out_proj are wrapped for hooks. The HF port ships no generate() (not a GenerationMixin), so generation phases are excluded.

class transformer_lens.model_bridge.supported_architectures.hyenadna.HyenaDNAArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Architecture adapter for HyenaDNAForCausalLM models.

__init__(cfg: Any) → None

Initialize the HyenaDNA architecture adapter.

applicable_phases: list[int] = [1, 2, 3, 4]

component_mapping: ComponentMapping | None

supports_batched_generation: bool = False

supports_fold_ln: bool = False

supports_generation: bool = True

supports_kv_cache: bool = False

uses_split_attention: bool

weight_processing_conversions: Dict[str, ParamProcessingConversion | str] | None