transformer_lens.model_bridge.supported_architectures.vaultgemma module¶
VaultGemma architecture adapter.
Google’s VaultGemma (VaultGemmaForCausalLM, native in transformers):
the only fully DP-SGD-pretrained open LLM — a Gemma-2 recipe trained under
differential privacy. Structurally it is Gemma 2 with the two post-norms
removed (blocks keep only input_layernorm and pre_feedforward_layernorm);
the scaled word embedding and attention logit soft-cap carry over, so this
subclasses the Gemma 2 adapter and rebuilds the block entry without
ln1_post/ln2_post.
- class transformer_lens.model_bridge.supported_architectures.vaultgemma.VaultGemmaArchitectureAdapter(cfg: Any)¶
Bases:
Gemma2ArchitectureAdapterArchitecture adapter for VaultGemmaForCausalLM models.
- __init__(cfg: Any) None¶
Initialize the VaultGemma architecture adapter.
- applicable_phases: list[int] = [1, 2, 4]¶
- component_mapping: ComponentMapping | None¶
- supports_compatibility_mode: bool = False¶
- uses_split_attention: bool¶
- weight_processing_conversions: Dict[str, ParamProcessingConversion | str] | None¶