transformer_lens.model_bridge.sources.transformers package

Submodules

• transformer_lens.model_bridge.sources.transformers.helpers module
  • check_model_support()
  • get_hf_model_class_for_architecture()
  • list_supported_models()
  • load_modality_processor()
• transformer_lens.model_bridge.sources.transformers.source module
  • boot()

Module contents

HuggingFace transformers source for TransformerBridge.

class transformer_lens.model_bridge.sources.transformers.AutoConfig

Bases: object

This is a generic configuration class that will be instantiated as one of the configuration classes of the library when created with the [~AutoConfig.from_pretrained] class method.

This class cannot be instantiated directly using __init__() (throws an error).

classmethod for_model(model_type: str, *args, **kwargs) → PreTrainedConfig

classmethod from_pretrained(*args: ~typing.Any, _orig: ~typing.Any = <function AutoConfig.from_pretrained>, **kwargs: ~typing.Any) → Any

static register(model_type, config, exist_ok=False) → None

Register a new configuration for this class.

Parameters:

• model_type (str) – The model type like “bert” or “gpt”.

• config ([PreTrainedConfig]) – The config to register.

class transformer_lens.model_bridge.sources.transformers.AutoModelForCausalLM(*args, **kwargs)

Bases: _BaseAutoModelClass

This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created with the [~AutoModelForCausalLM.from_pretrained] class method or the [~AutoModelForCausalLM.from_config] class method.

This class cannot be instantiated directly using __init__() (throws an error).

classmethod from_config(**kwargs)

Instantiates one of the model classes of the library (with a causal language modeling head) from a configuration.

Note

Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use [~AutoModelForCausalLM.from_pretrained] to load the model weights.

Parameters:

• config ([PreTrainedConfig]) –

  The model class to instantiate is selected based on the configuration class:

  • [AfmoeConfig] configuration class: [AfmoeForCausalLM] (AfmoeConfig model)

  • [ApertusConfig] configuration class: [ApertusForCausalLM] (ApertusConfig model)

  • [ArceeConfig] configuration class: [ArceeForCausalLM] (ArceeConfig model)

  • [AriaTextConfig] configuration class: [AriaTextForCausalLM] (AriaTextConfig model)

  • [BambaConfig] configuration class: [BambaForCausalLM] (BambaConfig model)

  • [BartConfig] configuration class: [BartForCausalLM] (BartConfig model)

  • [BertConfig] configuration class: [BertLMHeadModel] (BertConfig model)

  • [BertGenerationConfig] configuration class: [BertGenerationDecoder] (BertGenerationConfig model)

  • [BigBirdConfig] configuration class: [BigBirdForCausalLM] (BigBirdConfig model)

  • [BigBirdPegasusConfig] configuration class: [BigBirdPegasusForCausalLM] (BigBirdPegasusConfig model)

  • [BioGptConfig] configuration class: [BioGptForCausalLM] (BioGptConfig model)

  • [BitNetConfig] configuration class: [BitNetForCausalLM] (BitNetConfig model)

  • [BlenderbotConfig] configuration class: [BlenderbotForCausalLM] (BlenderbotConfig model)

  • [BlenderbotSmallConfig] configuration class: [BlenderbotSmallForCausalLM] (BlenderbotSmallConfig model)

  • [BloomConfig] configuration class: [BloomForCausalLM] (BloomConfig model)

  • [BltConfig] configuration class: [BltForCausalLM] (BltConfig model)

  • [CTRLConfig] configuration class: [CTRLLMHeadModel] (CTRLConfig model)

  • [CamembertConfig] configuration class: [CamembertForCausalLM] (CamembertConfig model)

  • [CodeGenConfig] configuration class: [CodeGenForCausalLM] (CodeGenConfig model)

  • [Cohere2Config] configuration class: [Cohere2ForCausalLM] (Cohere2Config model)

  • [Cohere2MoeConfig] configuration class: [Cohere2MoeForCausalLM] (Cohere2MoeConfig model)

  • [CohereConfig] configuration class: [CohereForCausalLM] (CohereConfig model)

  • [CpmAntConfig] configuration class: [CpmAntForCausalLM] (CpmAntConfig model)

  • [CwmConfig] configuration class: [CwmForCausalLM] (CwmConfig model)

  • [Data2VecTextConfig] configuration class: [Data2VecTextForCausalLM] (Data2VecTextConfig model)

  • [DbrxConfig] configuration class: [DbrxForCausalLM] (DbrxConfig model)

  • [DeepseekV2Config] configuration class: [DeepseekV2ForCausalLM] (DeepseekV2Config model)

  • [DeepseekV32Config] configuration class: [DeepseekV32ForCausalLM] (DeepseekV32Config model)

  • [DeepseekV3Config] configuration class: [DeepseekV3ForCausalLM] (DeepseekV3Config model)

  • [DeepseekV4Config] configuration class: [DeepseekV4ForCausalLM] (DeepseekV4Config model)

  • [DiffLlamaConfig] configuration class: [DiffLlamaForCausalLM] (DiffLlamaConfig model)

  • [DogeConfig] configuration class: [DogeForCausalLM] (DogeConfig model)

  • [Dots1Config] configuration class: [Dots1ForCausalLM] (Dots1Config model)

  • [ElectraConfig] configuration class: [ElectraForCausalLM] (ElectraConfig model)

  • [Emu3Config] configuration class: [Emu3ForCausalLM] (Emu3Config model)

  • [Ernie4_5Config] configuration class: [Ernie4_5ForCausalLM] (Ernie4_5Config model)

  • [Ernie4_5_MoeConfig] configuration class: [Ernie4_5_MoeForCausalLM] (Ernie4_5_MoeConfig model)

  • [ErnieConfig] configuration class: [ErnieForCausalLM] (ErnieConfig model)

  • [Exaone4Config] configuration class: [Exaone4ForCausalLM] (Exaone4Config model)

  • [ExaoneMoeConfig] configuration class: [ExaoneMoeForCausalLM] (ExaoneMoeConfig model)

  • [FalconConfig] configuration class: [FalconForCausalLM] (FalconConfig model)

  • [FalconH1Config] configuration class: [FalconH1ForCausalLM] (FalconH1Config model)

  • [FalconMambaConfig] configuration class: [FalconMambaForCausalLM] (FalconMambaConfig model)

  • [FlexOlmoConfig] configuration class: [FlexOlmoForCausalLM] (FlexOlmoConfig model)

  • [FuyuConfig] configuration class: [FuyuForCausalLM] (FuyuConfig model)

  • [GPT2Config] configuration class: [GPT2LMHeadModel] (GPT2Config model)

  • [GPTBigCodeConfig] configuration class: [GPTBigCodeForCausalLM] (GPTBigCodeConfig model)

  • [GPTJConfig] configuration class: [GPTJForCausalLM] (GPTJConfig model)

  • [GPTNeoConfig] configuration class: [GPTNeoForCausalLM] (GPTNeoConfig model)

  • [GPTNeoXConfig] configuration class: [GPTNeoXForCausalLM] (GPTNeoXConfig model)

  • [GPTNeoXJapaneseConfig] configuration class: [GPTNeoXJapaneseForCausalLM] (GPTNeoXJapaneseConfig model)

  • [Gemma2Config] configuration class: [Gemma2ForCausalLM] (Gemma2Config model)

  • [Gemma3Config] configuration class: [Gemma3ForConditionalGeneration] (Gemma3Config model)

  • [Gemma3TextConfig] configuration class: [Gemma3ForCausalLM] (Gemma3TextConfig model)

  • [Gemma3nConfig] configuration class: [Gemma3nForConditionalGeneration] (Gemma3nConfig model)

  • [Gemma3nTextConfig] configuration class: [Gemma3nForCausalLM] (Gemma3nTextConfig model)

  • [Gemma4AssistantConfig] configuration class: [Gemma4AssistantForCausalLM] (Gemma4AssistantConfig model)

  • [Gemma4Config] configuration class: [Gemma4ForConditionalGeneration] (Gemma4Config model)

  • [Gemma4TextConfig] configuration class: [Gemma4ForCausalLM] (Gemma4TextConfig model)

  • [Gemma4UnifiedAssistantConfig] configuration class: [Gemma4UnifiedAssistantForCausalLM] (Gemma4UnifiedAssistantConfig model)

  • [Gemma4UnifiedConfig] configuration class: [Gemma4UnifiedForConditionalGeneration] (Gemma4UnifiedConfig model)

  • [Gemma4UnifiedTextConfig] configuration class: [Gemma4UnifiedForCausalLM] (Gemma4UnifiedTextConfig model)

  • [GemmaConfig] configuration class: [GemmaForCausalLM] (GemmaConfig model)

  • [GitConfig] configuration class: [GitForCausalLM] (GitConfig model)

  • [Glm4Config] configuration class: [Glm4ForCausalLM] (Glm4Config model)

  • [Glm4MoeConfig] configuration class: [Glm4MoeForCausalLM] (Glm4MoeConfig model)

  • [Glm4MoeLiteConfig] configuration class: [Glm4MoeLiteForCausalLM] (Glm4MoeLiteConfig model)

  • [GlmConfig] configuration class: [GlmForCausalLM] (GlmConfig model)

  • [GlmMoeDsaConfig] configuration class: [GlmMoeDsaForCausalLM] (GlmMoeDsaConfig model)

  • [GotOcr2Config] configuration class: [GotOcr2ForConditionalGeneration] (GotOcr2Config model)

  • [GptOssConfig] configuration class: [GptOssForCausalLM] (GptOssConfig model)

  • [GraniteConfig] configuration class: [GraniteForCausalLM] (GraniteConfig model)

  • [GraniteMoeConfig] configuration class: [GraniteMoeForCausalLM] (GraniteMoeConfig model)

  • [GraniteMoeHybridConfig] configuration class: [GraniteMoeHybridForCausalLM] (GraniteMoeHybridConfig model)

  • [GraniteMoeSharedConfig] configuration class: [GraniteMoeSharedForCausalLM] (GraniteMoeSharedConfig model)

  • [HYV3Config] configuration class: [HYV3ForCausalLM] (HYV3Config model)

  • [HeliumConfig] configuration class: [HeliumForCausalLM] (HeliumConfig model)

  • [HrmTextConfig] configuration class: [HrmTextForCausalLM] (HrmTextConfig model)

  • [HunYuanDenseV1Config] configuration class: [HunYuanDenseV1ForCausalLM] (HunYuanDenseV1Config model)

  • [HunYuanMoEV1Config] configuration class: [HunYuanMoEV1ForCausalLM] (HunYuanMoEV1Config model)

  • [HyperCLOVAXConfig] configuration class: [HyperCLOVAXForCausalLM] (HyperCLOVAXConfig model)

  • [Jais2Config] configuration class: [Jais2ForCausalLM] (Jais2Config model)

  • [JambaConfig] configuration class: [JambaForCausalLM] (JambaConfig model)

  • [JetMoeConfig] configuration class: [JetMoeForCausalLM] (JetMoeConfig model)

  • [LagunaConfig] configuration class: [LagunaForCausalLM] (LagunaConfig model)

  • [Lfm2Config] configuration class: [Lfm2ForCausalLM] (Lfm2Config model)

  • [Lfm2MoeConfig] configuration class: [Lfm2MoeForCausalLM] (Lfm2MoeConfig model)

  • [Llama4Config] configuration class: [Llama4ForCausalLM] (Llama4Config model)

  • [Llama4TextConfig] configuration class: [Llama4ForCausalLM] (Llama4TextConfig model)

  • [LlamaConfig] configuration class: [LlamaForCausalLM] (LlamaConfig model)

  • [LongcatFlashConfig] configuration class: [LongcatFlashForCausalLM] (LongcatFlashConfig model)

  • [MBartConfig] configuration class: [MBartForCausalLM] (MBartConfig model)

  • [Mamba2Config] configuration class: [Mamba2ForCausalLM] (Mamba2Config model)

  • [MambaConfig] configuration class: [MambaForCausalLM] (MambaConfig model)

  • [MarianConfig] configuration class: [MarianForCausalLM] (MarianConfig model)

  • [MegatronBertConfig] configuration class: [MegatronBertForCausalLM] (MegatronBertConfig model)

  • [MellumConfig] configuration class: [MellumForCausalLM] (MellumConfig model)

  • [MiMoV2FlashConfig] configuration class: [MiMoV2FlashForCausalLM] (MiMoV2FlashConfig model)

  • [MiniCPM3Config] configuration class: [MiniCPM3ForCausalLM] (MiniCPM3Config model)

  • [MiniMaxConfig] configuration class: [MiniMaxForCausalLM] (MiniMaxConfig model)

  • [MiniMaxM2Config] configuration class: [MiniMaxM2ForCausalLM] (MiniMaxM2Config model)

  • [MiniMaxM3VLTextConfig] configuration class: [MiniMaxM3VLForCausalLM] (MiniMaxM3VLTextConfig model)

  • [Ministral3Config] configuration class: [Ministral3ForCausalLM] (Ministral3Config model)

  • [MinistralConfig] configuration class: [MinistralForCausalLM] (MinistralConfig model)

  • [MistralConfig] configuration class: [MistralForCausalLM] (MistralConfig model)

  • [MixtralConfig] configuration class: [MixtralForCausalLM] (MixtralConfig model)

  • [MllamaConfig] configuration class: [MllamaForCausalLM] (MllamaConfig model)

  • [ModernBertDecoderConfig] configuration class: [ModernBertDecoderForCausalLM] (ModernBertDecoderConfig model)

  • [MoshiConfig] configuration class: [MoshiForCausalLM] (MoshiConfig model)

  • [MptConfig] configuration class: [MptForCausalLM] (MptConfig model)

  • [MusicgenConfig] configuration class: [MusicgenForCausalLM] (MusicgenConfig model)

  • [MusicgenMelodyConfig] configuration class: [MusicgenMelodyForCausalLM] (MusicgenMelodyConfig model)

  • [MvpConfig] configuration class: [MvpForCausalLM] (MvpConfig model)

  • [NanoChatConfig] configuration class: [NanoChatForCausalLM] (NanoChatConfig model)

  • [NemotronConfig] configuration class: [NemotronForCausalLM] (NemotronConfig model)

  • [NemotronHConfig] configuration class: [NemotronHForCausalLM] (NemotronHConfig model)

  • [OPTConfig] configuration class: [OPTForCausalLM] (OPTConfig model)

  • [Olmo2Config] configuration class: [Olmo2ForCausalLM] (Olmo2Config model)

  • [Olmo3Config] configuration class: [Olmo3ForCausalLM] (Olmo3Config model)

  • [OlmoConfig] configuration class: [OlmoForCausalLM] (OlmoConfig model)

  • [OlmoHybridConfig] configuration class: [OlmoHybridForCausalLM] (OlmoHybridConfig model)

  • [OlmoeConfig] configuration class: [OlmoeForCausalLM] (OlmoeConfig model)

  • [OpenAIGPTConfig] configuration class: [OpenAIGPTLMHeadModel] (OpenAIGPTConfig model)

  • [PLBartConfig] configuration class: [PLBartForCausalLM] (PLBartConfig model)

  • [PegasusConfig] configuration class: [PegasusForCausalLM] (PegasusConfig model)

  • [PersimmonConfig] configuration class: [PersimmonForCausalLM] (PersimmonConfig model)

  • [Phi3Config] configuration class: [Phi3ForCausalLM] (Phi3Config model)

  • [Phi4MultimodalConfig] configuration class: [Phi4MultimodalForCausalLM] (Phi4MultimodalConfig model)

  • [PhiConfig] configuration class: [PhiForCausalLM] (PhiConfig model)

  • [PhimoeConfig] configuration class: [PhimoeForCausalLM] (PhimoeConfig model)

  • [ProphetNetConfig] configuration class: [ProphetNetForCausalLM] (ProphetNetConfig model)

  • [Qwen2Config] configuration class: [Qwen2ForCausalLM] (Qwen2Config model)

  • [Qwen2MoeConfig] configuration class: [Qwen2MoeForCausalLM] (Qwen2MoeConfig model)

  • [Qwen3Config] configuration class: [Qwen3ForCausalLM] (Qwen3Config model)

  • [Qwen3MoeConfig] configuration class: [Qwen3MoeForCausalLM] (Qwen3MoeConfig model)

  • [Qwen3NextConfig] configuration class: [Qwen3NextForCausalLM] (Qwen3NextConfig model)

  • [Qwen3_5Config] configuration class: [Qwen3_5ForCausalLM] (Qwen3_5Config model)

  • [Qwen3_5MoeConfig] configuration class: [Qwen3_5MoeForCausalLM] (Qwen3_5MoeConfig model)

  • [Qwen3_5MoeTextConfig] configuration class: [Qwen3_5MoeForCausalLM] (Qwen3_5MoeTextConfig model)

  • [Qwen3_5TextConfig] configuration class: [Qwen3_5ForCausalLM] (Qwen3_5TextConfig model)

  • [RecurrentGemmaConfig] configuration class: [RecurrentGemmaForCausalLM] (RecurrentGemmaConfig model)

  • [ReformerConfig] configuration class: [ReformerModelWithLMHead] (ReformerConfig model)

  • [RemBertConfig] configuration class: [RemBertForCausalLM] (RemBertConfig model)

  • [RoCBertConfig] configuration class: [RoCBertForCausalLM] (RoCBertConfig model)

  • [RoFormerConfig] configuration class: [RoFormerForCausalLM] (RoFormerConfig model)

  • [RobertaConfig] configuration class: [RobertaForCausalLM] (RobertaConfig model)

  • [RobertaPreLayerNormConfig] configuration class: [RobertaPreLayerNormForCausalLM] (RobertaPreLayerNormConfig model)

  • [RwkvConfig] configuration class: [RwkvForCausalLM] (RwkvConfig model)

  • [SeedOssConfig] configuration class: [SeedOssForCausalLM] (SeedOssConfig model)

  • [SmolLM3Config] configuration class: [SmolLM3ForCausalLM] (SmolLM3Config model)

  • [SolarOpenConfig] configuration class: [SolarOpenForCausalLM] (SolarOpenConfig model)

  • [StableLmConfig] configuration class: [StableLmForCausalLM] (StableLmConfig model)

  • [Starcoder2Config] configuration class: [Starcoder2ForCausalLM] (Starcoder2Config model)

  • [TrOCRConfig] configuration class: [TrOCRForCausalLM] (TrOCRConfig model)

  • [VaultGemmaConfig] configuration class: [VaultGemmaForCausalLM] (VaultGemmaConfig model)

  • [WhisperConfig] configuration class: [WhisperForCausalLM] (WhisperConfig model)

  • [XGLMConfig] configuration class: [XGLMForCausalLM] (XGLMConfig model)

  • [XLMConfig] configuration class: [XLMWithLMHeadModel] (XLMConfig model)

  • [XLMRobertaConfig] configuration class: [XLMRobertaForCausalLM] (XLMRobertaConfig model)

  • [XLMRobertaXLConfig] configuration class: [XLMRobertaXLForCausalLM] (XLMRobertaXLConfig model)

  • [XLNetConfig] configuration class: [XLNetLMHeadModel] (XLNetConfig model)

  • [XmodConfig] configuration class: [XmodForCausalLM] (XmodConfig model)

  • [YoutuConfig] configuration class: [YoutuForCausalLM] (YoutuConfig model)

  • [Zamba2Config] configuration class: [Zamba2ForCausalLM] (Zamba2Config model)

  • [ZambaConfig] configuration class: [ZambaForCausalLM] (ZambaConfig model)

  • [ZayaConfig] configuration class: [ZayaForCausalLM] (ZayaConfig model)

  • [xLSTMConfig] configuration class: [xLSTMForCausalLM] (xLSTMConfig model)

• attn_implementation (str, optional) – The attention implementation to use in the model (if relevant). Can be any of “eager” (manual implementation of the attention), “sdpa” (using [F.scaled_dot_product_attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html)), “flash_attention_2” (using [Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention)), or “flash_attention_3” (using [Dao-AILab/flash-attention/hopper](https://github.com/Dao-AILab/flash-attention/tree/main/hopper)). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual “eager” implementation.

Examples:

```python >>> from transformers import AutoConfig, AutoModelForCausalLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForCausalLM.from_config(config)
```

classmethod from_pretrained(*model_args, **kwargs)

Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

• afmoe – [AfmoeForCausalLM] (AfmoeConfig model)

• apertus – [ApertusForCausalLM] (ApertusConfig model)

• arcee – [ArceeForCausalLM] (ArceeConfig model)

• aria_text – [AriaTextForCausalLM] (AriaTextConfig model)

• bamba – [BambaForCausalLM] (BambaConfig model)

• bart – [BartForCausalLM] (BartConfig model)

• bert – [BertLMHeadModel] (BertConfig model)

• bert-generation – [BertGenerationDecoder] (BertGenerationConfig model)

• big_bird – [BigBirdForCausalLM] (BigBirdConfig model)

• bigbird_pegasus – [BigBirdPegasusForCausalLM] (BigBirdPegasusConfig model)

• biogpt – [BioGptForCausalLM] (BioGptConfig model)

• bitnet – [BitNetForCausalLM] (BitNetConfig model)

• blenderbot – [BlenderbotForCausalLM] (BlenderbotConfig model)

• blenderbot-small – [BlenderbotSmallForCausalLM] (BlenderbotSmallConfig model)

• bloom – [BloomForCausalLM] (BloomConfig model)

• blt – [BltForCausalLM] (BltConfig model)

• camembert – [CamembertForCausalLM] (CamembertConfig model)

• codegen – [CodeGenForCausalLM] (CodeGenConfig model)

• cohere – [CohereForCausalLM] (CohereConfig model)

• cohere2 – [Cohere2ForCausalLM] (Cohere2Config model)

• cohere2_moe – [Cohere2MoeForCausalLM] (Cohere2MoeConfig model)

• cpmant – [CpmAntForCausalLM] (CpmAntConfig model)

• ctrl – [CTRLLMHeadModel] (CTRLConfig model)

• cwm – [CwmForCausalLM] (CwmConfig model)

• data2vec-text – [Data2VecTextForCausalLM] (Data2VecTextConfig model)

• dbrx – [DbrxForCausalLM] (DbrxConfig model)

• deepseek_v2 – [DeepseekV2ForCausalLM] (DeepseekV2Config model)

• deepseek_v3 – [DeepseekV3ForCausalLM] (DeepseekV3Config model)

• deepseek_v32 – [DeepseekV32ForCausalLM] (DeepseekV32Config model)

• deepseek_v4 – [DeepseekV4ForCausalLM] (DeepseekV4Config model)

• diffllama – [DiffLlamaForCausalLM] (DiffLlamaConfig model)

• doge – [DogeForCausalLM] (DogeConfig model)

• dots1 – [Dots1ForCausalLM] (Dots1Config model)

• electra – [ElectraForCausalLM] (ElectraConfig model)

• emu3 – [Emu3ForCausalLM] (Emu3Config model)

• ernie – [ErnieForCausalLM] (ErnieConfig model)

• ernie4_5 – [Ernie4_5ForCausalLM] (Ernie4_5Config model)

• ernie4_5_moe – [Ernie4_5_MoeForCausalLM] (Ernie4_5_MoeConfig model)

• exaone4 – [Exaone4ForCausalLM] (Exaone4Config model)

• exaone_moe – [ExaoneMoeForCausalLM] (ExaoneMoeConfig model)

• falcon – [FalconForCausalLM] (FalconConfig model)

• falcon_h1 – [FalconH1ForCausalLM] (FalconH1Config model)

• falcon_mamba – [FalconMambaForCausalLM] (FalconMambaConfig model)

• flex_olmo – [FlexOlmoForCausalLM] (FlexOlmoConfig model)

• fuyu – [FuyuForCausalLM] (FuyuConfig model)

• gemma – [GemmaForCausalLM] (GemmaConfig model)

• gemma2 – [Gemma2ForCausalLM] (Gemma2Config model)

• gemma3 – [Gemma3ForConditionalGeneration] (Gemma3Config model)

• gemma3_text – [Gemma3ForCausalLM] (Gemma3TextConfig model)

• gemma3n – [Gemma3nForConditionalGeneration] (Gemma3nConfig model)

• gemma3n_text – [Gemma3nForCausalLM] (Gemma3nTextConfig model)

• gemma4 – [Gemma4ForConditionalGeneration] (Gemma4Config model)

• gemma4_assistant – [Gemma4AssistantForCausalLM] (Gemma4AssistantConfig model)

• gemma4_text – [Gemma4ForCausalLM] (Gemma4TextConfig model)

• gemma4_unified – [Gemma4UnifiedForConditionalGeneration] (Gemma4UnifiedConfig model)

• gemma4_unified_assistant – [Gemma4UnifiedAssistantForCausalLM] (Gemma4UnifiedAssistantConfig model)

• gemma4_unified_text – [Gemma4UnifiedForCausalLM] (Gemma4UnifiedTextConfig model)

• git – [GitForCausalLM] (GitConfig model)

• glm – [GlmForCausalLM] (GlmConfig model)

• glm4 – [Glm4ForCausalLM] (Glm4Config model)

• glm4_moe – [Glm4MoeForCausalLM] (Glm4MoeConfig model)

• glm4_moe_lite – [Glm4MoeLiteForCausalLM] (Glm4MoeLiteConfig model)

• glm_moe_dsa – [GlmMoeDsaForCausalLM] (GlmMoeDsaConfig model)

• got_ocr2 – [GotOcr2ForConditionalGeneration] (GotOcr2Config model)

• gpt-sw3 – [GPT2LMHeadModel] (GPT2Config model)

• gpt2 – [GPT2LMHeadModel] (GPT2Config model)

• gpt_bigcode – [GPTBigCodeForCausalLM] (GPTBigCodeConfig model)

• gpt_neo – [GPTNeoForCausalLM] (GPTNeoConfig model)

• gpt_neox – [GPTNeoXForCausalLM] (GPTNeoXConfig model)

• gpt_neox_japanese – [GPTNeoXJapaneseForCausalLM] (GPTNeoXJapaneseConfig model)

• gpt_oss – [GptOssForCausalLM] (GptOssConfig model)

• gptj – [GPTJForCausalLM] (GPTJConfig model)

• granite – [GraniteForCausalLM] (GraniteConfig model)

• granitemoe – [GraniteMoeForCausalLM] (GraniteMoeConfig model)

• granitemoehybrid – [GraniteMoeHybridForCausalLM] (GraniteMoeHybridConfig model)

• granitemoeshared – [GraniteMoeSharedForCausalLM] (GraniteMoeSharedConfig model)

• helium – [HeliumForCausalLM] (HeliumConfig model)

• hrm_text – [HrmTextForCausalLM] (HrmTextConfig model)

• hunyuan_v1_dense – [HunYuanDenseV1ForCausalLM] (HunYuanDenseV1Config model)

• hunyuan_v1_moe – [HunYuanMoEV1ForCausalLM] (HunYuanMoEV1Config model)

• hy_v3 – [HYV3ForCausalLM] (HYV3Config model)

• hyperclovax – [HyperCLOVAXForCausalLM] (HyperCLOVAXConfig model)

• jais2 – [Jais2ForCausalLM] (Jais2Config model)

• jamba – [JambaForCausalLM] (JambaConfig model)

• jetmoe – [JetMoeForCausalLM] (JetMoeConfig model)

• laguna – [LagunaForCausalLM] (LagunaConfig model)

• lfm2 – [Lfm2ForCausalLM] (Lfm2Config model)

• lfm2_moe – [Lfm2MoeForCausalLM] (Lfm2MoeConfig model)

• llama – [LlamaForCausalLM] (LlamaConfig model)

• llama4 – [Llama4ForCausalLM] (Llama4Config model)

• llama4_text – [Llama4ForCausalLM] (Llama4TextConfig model)

• longcat_flash – [LongcatFlashForCausalLM] (LongcatFlashConfig model)

• mamba – [MambaForCausalLM] (MambaConfig model)

• mamba2 – [Mamba2ForCausalLM] (Mamba2Config model)

• marian – [MarianForCausalLM] (MarianConfig model)

• mbart – [MBartForCausalLM] (MBartConfig model)

• megatron-bert – [MegatronBertForCausalLM] (MegatronBertConfig model)

• mellum – [MellumForCausalLM] (MellumConfig model)

• mimo_v2_flash – [MiMoV2FlashForCausalLM] (MiMoV2FlashConfig model)

• minicpm3 – [MiniCPM3ForCausalLM] (MiniCPM3Config model)

• minimax – [MiniMaxForCausalLM] (MiniMaxConfig model)

• minimax_m2 – [MiniMaxM2ForCausalLM] (MiniMaxM2Config model)

• minimax_m3_vl_text – [MiniMaxM3VLForCausalLM] (MiniMaxM3VLTextConfig model)

• ministral – [MinistralForCausalLM] (MinistralConfig model)

• ministral3 – [Ministral3ForCausalLM] (Ministral3Config model)

• mistral – [MistralForCausalLM] (MistralConfig model)

• mixtral – [MixtralForCausalLM] (MixtralConfig model)

• mllama – [MllamaForCausalLM] (MllamaConfig model)

• modernbert-decoder – [ModernBertDecoderForCausalLM] (ModernBertDecoderConfig model)

• moshi – [MoshiForCausalLM] (MoshiConfig model)

• mpt – [MptForCausalLM] (MptConfig model)

• musicgen – [MusicgenForCausalLM] (MusicgenConfig model)

• musicgen_melody – [MusicgenMelodyForCausalLM] (MusicgenMelodyConfig model)

• mvp – [MvpForCausalLM] (MvpConfig model)

• nanochat – [NanoChatForCausalLM] (NanoChatConfig model)

• nemotron – [NemotronForCausalLM] (NemotronConfig model)

• nemotron_h – [NemotronHForCausalLM] (NemotronHConfig model)

• olmo – [OlmoForCausalLM] (OlmoConfig model)

• olmo2 – [Olmo2ForCausalLM] (Olmo2Config model)

• olmo3 – [Olmo3ForCausalLM] (Olmo3Config model)

• olmo_hybrid – [OlmoHybridForCausalLM] (OlmoHybridConfig model)

• olmoe – [OlmoeForCausalLM] (OlmoeConfig model)

• openai-gpt – [OpenAIGPTLMHeadModel] (OpenAIGPTConfig model)

• opt – [OPTForCausalLM] (OPTConfig model)

• pegasus – [PegasusForCausalLM] (PegasusConfig model)

• persimmon – [PersimmonForCausalLM] (PersimmonConfig model)

• phi – [PhiForCausalLM] (PhiConfig model)

• phi3 – [Phi3ForCausalLM] (Phi3Config model)

• phi4_multimodal – [Phi4MultimodalForCausalLM] (Phi4MultimodalConfig model)

• phimoe – [PhimoeForCausalLM] (PhimoeConfig model)

• plbart – [PLBartForCausalLM] (PLBartConfig model)

• prophetnet – [ProphetNetForCausalLM] (ProphetNetConfig model)

• qwen2 – [Qwen2ForCausalLM] (Qwen2Config model)

• qwen2_moe – [Qwen2MoeForCausalLM] (Qwen2MoeConfig model)

• qwen3 – [Qwen3ForCausalLM] (Qwen3Config model)

• qwen3_5 – [Qwen3_5ForCausalLM] (Qwen3_5Config model)

• qwen3_5_moe – [Qwen3_5MoeForCausalLM] (Qwen3_5MoeConfig model)

• qwen3_5_moe_text – [Qwen3_5MoeForCausalLM] (Qwen3_5MoeTextConfig model)

• qwen3_5_text – [Qwen3_5ForCausalLM] (Qwen3_5TextConfig model)

• qwen3_moe – [Qwen3MoeForCausalLM] (Qwen3MoeConfig model)

• qwen3_next – [Qwen3NextForCausalLM] (Qwen3NextConfig model)

• recurrent_gemma – [RecurrentGemmaForCausalLM] (RecurrentGemmaConfig model)

• reformer – [ReformerModelWithLMHead] (ReformerConfig model)

• rembert – [RemBertForCausalLM] (RemBertConfig model)

• roberta – [RobertaForCausalLM] (RobertaConfig model)

• roberta-prelayernorm – [RobertaPreLayerNormForCausalLM] (RobertaPreLayerNormConfig model)

• roc_bert – [RoCBertForCausalLM] (RoCBertConfig model)

• roformer – [RoFormerForCausalLM] (RoFormerConfig model)

• rwkv – [RwkvForCausalLM] (RwkvConfig model)

• seed_oss – [SeedOssForCausalLM] (SeedOssConfig model)

• smollm3 – [SmolLM3ForCausalLM] (SmolLM3Config model)

• solar_open – [SolarOpenForCausalLM] (SolarOpenConfig model)

• stablelm – [StableLmForCausalLM] (StableLmConfig model)

• starcoder2 – [Starcoder2ForCausalLM] (Starcoder2Config model)

• trocr – [TrOCRForCausalLM] (TrOCRConfig model)

• vaultgemma – [VaultGemmaForCausalLM] (VaultGemmaConfig model)

• whisper – [WhisperForCausalLM] (WhisperConfig model)

• xglm – [XGLMForCausalLM] (XGLMConfig model)

• xlm – [XLMWithLMHeadModel] (XLMConfig model)

• xlm-roberta – [XLMRobertaForCausalLM] (XLMRobertaConfig model)

• xlm-roberta-xl – [XLMRobertaXLForCausalLM] (XLMRobertaXLConfig model)

• xlnet – [XLNetLMHeadModel] (XLNetConfig model)

• xlstm – [xLSTMForCausalLM] (xLSTMConfig model)

• xmod – [XmodForCausalLM] (XmodConfig model)

• youtu – [YoutuForCausalLM] (YoutuConfig model)

• zamba – [ZambaForCausalLM] (ZambaConfig model)

• zamba2 – [Zamba2ForCausalLM] (Zamba2Config model)

• zaya – [ZayaForCausalLM] (ZayaConfig model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Parameters:

• pretrained_model_name_or_path (str or os.PathLike) –

  Can be either:

  • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.

  • A path to a directory containing model weights saved using [~PreTrainedModel.save_pretrained], e.g., ./my_model_directory/.

• model_args (additional positional arguments, optional) – Will be passed along to the underlying model __init__() method.

• config ([PreTrainedConfig], optional) –

  Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

  • The model is a model provided by the library (loaded with the model id string of a pretrained model).

  • The model was saved using [~PreTrainedModel.save_pretrained] and is reloaded by supplying the save directory.

  • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.

• state_dict (dict[str, torch.Tensor], optional) –

  A state dictionary to use instead of a state dictionary loaded from saved weights file.

  This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using [~PreTrainedModel.save_pretrained] and [~PreTrainedModel.from_pretrained] is not a simpler option.

• cache_dir (str or os.PathLike, optional) – Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.

• force_download (bool, optional, defaults to False) – Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.

• proxies (dict[str, str], optional) – A dictionary of proxy servers to use by protocol or endpoint, e.g., {‘http’: ‘foo.bar:3128’, ‘http://hostname’: ‘foo.bar:4012’}. The proxies are used on each request.

• output_loading_info (bool, optional, defaults to False) – Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages.

• local_files_only (bool, optional, defaults to False) – Whether or not to only look at local files (e.g., not try downloading the model).

• revision (str, optional, defaults to “main”) – The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.

• trust_remote_code (bool, optional, defaults to False) – Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.

• code_revision (str, optional, defaults to “main”) – The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.

• kwargs (additional keyword arguments, optional) –

  Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

  • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)

  • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function ([~PreTrainedConfig.from_pretrained]). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Examples:

```python >>> from transformers import AutoConfig, AutoModelForCausalLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
```

class transformer_lens.model_bridge.sources.transformers.AutoTokenizer

Bases: object

This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [AutoTokenizer.from_pretrained] class method.

This class cannot be instantiated directly using __init__() (throws an error).

classmethod from_pretrained(*args: ~typing.Any, _orig: ~typing.Any = <function AutoTokenizer.from_pretrained>, **kwargs: ~typing.Any) → Any

static register(config_class, tokenizer_class=None, slow_tokenizer_class=None, fast_tokenizer_class=None, exist_ok=False)

Register a new tokenizer in this mapping.

Parameters:

• config_class ([PreTrainedConfig]) – The configuration corresponding to the model to register.

• tokenizer_class – The tokenizer class to register (V5 - preferred parameter).

• slow_tokenizer_class – (Deprecated) The slow tokenizer to register.

• fast_tokenizer_class – (Deprecated) The fast tokenizer to register.

transformer_lens.model_bridge.sources.transformers.boot(model_name: str, hf_config_overrides: dict | None = None, device: str | device | None = None, dtype: dtype = torch.float32, tokenizer: PreTrainedTokenizerBase | None = None, load_weights: bool = True, trust_remote_code: bool = False, model_class: Any | None = None, hf_model: Any | None = None, n_ctx: int | None = None, revision: str | None = None, checkpoint_index: int | None = None, checkpoint_value: int | None = None, device_map: str | dict[str, str | int] | None = None, n_devices: int | None = None, max_memory: dict[str | int, str | int] | None = None, offload_folder: str | None = None) → TransformerBridge

Boot a model from HuggingFace (exposed as TransformerBridge.boot_transformers).

Returns raw HF weights by default — logits/activations match HF, not legacy HookedTransformer (which folds LayerNorm + centers weights). Call enable_compatibility_mode() on the result for HookedTransformer- equivalent numerics. Generation, argmax, and CE loss are unaffected.

Attention implementation is forced to "eager" so hooks can capture scores and patterns. For an apples-to-apples HF comparison, load the HF model with attn_implementation="eager" too; comparing against the default "sdpa" shows ~1e-3 fp32 drift from kernel-level op reordering, not a bridge bug.

Parameters:

• model_name – The name of the model to load.

• hf_config_overrides – Optional overrides applied to the HuggingFace config before model load.

• device – The device to use. If None, will be determined automatically. Mutually exclusive with device_map.

• dtype – The dtype to use for the model.

• tokenizer – Optional pre-initialized tokenizer to use; if not provided one will be created.

• load_weights – If False, load model without weights (on meta device) for config inspection only.

• model_class – Optional HuggingFace model class to use instead of the default auto-detected class. When the class name matches a key in SUPPORTED_ARCHITECTURES, the corresponding adapter is selected automatically (e.g., BertForNextSentencePrediction).

• hf_model – Optional pre-loaded HuggingFace model to use instead of loading one. Useful for models loaded with custom configurations (e.g., quantization via BitsAndBytesConfig). When provided, load_weights is ignored.

• device_map – HuggingFace-style device map ("auto", "balanced", dict, etc.) for dispatched inference. Explicit maps may include CPU and disk targets; meta targets are still rejected when load_weights=True (meta has no real data to offload from, unlike disk/cpu). Mixed CPU/disk + GPU maps are rejected too, not because they’re known to be broken but because CPU/disk offload has only been verified on CPU-only hardware — no GPU to mix in. bridge.enable_compatibility_mode() (with weight processing, i.e. not no_processing=True) is unsupported on a CPU/disk-offloaded bridge and raises immediately rather than mid-fold; the default (non-compat-mode) forward pass, run_with_cache, and hooks all work normally under offload. Mutually exclusive with device.

• n_devices – Convenience: split the model across this many CUDA devices (translated to a max_memory dict internally). Requires CUDA with at least this many visible devices.

• max_memory – Optional per-device memory budget for HF’s dispatcher.

• offload_folder – Directory for disk-offloaded weight shards when device_map includes a "disk" target. Defaults to a temporary directory (HF’s own default) if omitted.

• n_ctx – Optional context length override. The bridge normally uses the model’s documented max context from the HF config. Setting this writes to whichever HF field the model uses (n_positions / max_position_embeddings / etc.), so callers don’t need to know the field name. If larger than the model’s default, a warning is emitted — quality may degrade past the trained length for rotary models.

• revision – Optional HF revision string (branch, tag, or commit). Forwarded to config, model, and tokenizer loading. Mutually exclusive with checkpoint_index and checkpoint_value.

• checkpoint_index – Index into the available training checkpoints for the model family. Convenience over revision for checkpointed models like EleutherAI/pythia* and stanford-crfm/*. Resolved to a revision string via the known per-family naming conventions (step{value} for Pythia, checkpoint-{value} for stanford-crfm).

• checkpoint_value – Training step or token count of the desired checkpoint. Alternative to checkpoint_index; must be one of the labels returned by get_checkpoint_labels.

Returns:

The bridge to the loaded model.

transformer_lens.model_bridge.sources.transformers.check_model_support(model_id: str) → dict

Detailed support info for a model: is_supported, architecture_id, verified, suggestion.

transformer_lens.model_bridge.sources.transformers.determine_architecture_from_hf_config(hf_config)

Determine the architecture name from HuggingFace config.

Returns:

The architecture name (e.g., “GPT2LMHeadModel”, “LlamaForCausalLM”)

Return type:

str

Raises:

ValueError – If architecture cannot be determined

transformer_lens.model_bridge.sources.transformers.get_hf_model_class_for_architecture(architecture: str)

Pick the correct HuggingFace AutoModel* class for the architecture.

transformer_lens.model_bridge.sources.transformers.list_supported_models(architecture: str | None = None, verified_only: bool = False) → list[str]

List all models supported by TransformerLens.

Parameters:

• architecture – Filter by architecture ID (e.g., “GPT2LMHeadModel”).

• verified_only – If True, only return verified-to-work models.

Returns:

List of model IDs.

transformer_lens.model_bridge.sources.transformers.map_default_transformer_lens_config(hf_config)

Map HuggingFace config fields to TransformerLens config format.

Standardized mapping from various HuggingFace config field names to the consistent TransformerLens naming convention. For multimodal models (LLaVA, Gemma3ForConditionalGeneration), the language model dimensions are nested under text_config; we extract from there first.

Parameters:

hf_config – The HuggingFace config object

Returns:

A copy of hf_config with additional TransformerLens fields

transformer_lens.model_bridge.sources.transformers.setup_tokenizer(tokenizer, default_padding_side=None)

Normalize a HuggingFace tokenizer for use with the bridge.

Parameters:

• tokenizer – A PreTrainedTokenizer or PreTrainedTokenizerFast.

• default_padding_side – "right" or "left"; sets tokenizer.padding_side.