transformer_lens.model_bridge.supported_architectures.nanochat module¶
NanoChat (NanoChatForCausalLM) adapter: Llama-style decoder with weightless
RMSNorm (so nothing for fold_ln to fold), ungated relu^2 MLP, and soft-capped logits;
attention delegated (full-width q/k norm after RoPE).
- class transformer_lens.model_bridge.supported_architectures.nanochat.NanoChatArchitectureAdapter(cfg: Any)¶
Bases:
ArchitectureAdapterArchitecture adapter for NanoChatForCausalLM models.
- __init__(cfg: Any) None¶
Initialize the NanoChat architecture adapter.
- component_mapping: ComponentMapping | None¶
- supports_fold_ln = False¶
- uses_split_attention: bool¶
- weight_processing_conversions: Dict[str, ParamProcessingConversion | str] | None¶