transformer_lens.model_bridge.supported_architectures.gidd module

Gidd architecture adapter.

Dimitri von Rütte’s GIDD (GiddForDiffusionLM, remote code): the only open uniform-noise (non-masked) diffusion LM at scale, with self-correction sampling. The decoder is bidirectional (config.is_causal=False) with softcap attention variants, optional per-head QK norms, ScaledLinear projections (weight-scaled at forward), per-layer scaled residual adds (resid_scale/num_layers), an ungated up/down MLP, and rotary positions held as a model-level buffer rather than a module.

Everything nonstandard lives inside delegated modules: attention delegates wholesale (softcap + bidirectional), ScaledLinear wraps as plain hookable Linears, and generation is the model’s own diffusion sampler, reached via bridge.diffusion_generate — no autoregressive generation, no folding into scaled projections.

class transformer_lens.model_bridge.supported_architectures.gidd.GiddArchitectureAdapter(cfg: Any)

Bases: ArchitectureAdapter

Architecture adapter for GiddForDiffusionLM models.

__init__(cfg: Any) → None

Initialize the Gidd architecture adapter.

applicable_phases: list[int] = [1, 2, 3, 4]

component_mapping: ComponentMapping | None

native_sampler: str = 'generate'

native_sampler_kwargs(max_new_tokens: int, prompt_len: int) → dict

Gidd’s max_length counts generated tokens: its windows start at prompt_length and span max_length, so adding the prompt over-generates.

prepare_loading(model_name: str, model_kwargs: dict) → None

Patch the remote class before from_pretrained runs.

Like BD3LM, the remote code’s attribute handling raises on v5’s all_tied_weights_keys lookup (the checkpoint is untied anyway).

prepare_model(hf_model: Any) → None

Restore the rotary table lost to meta-device loading.

setup_component_testing(hf_model: Any, bridge_model: Any = None) → None

Delegated attention reads the rotary buffer inside HF; nothing to wire.

supports_fold_ln = False

supports_generation: bool = False

uses_split_attention: bool

weight_processing_conversions: Dict[str, ParamProcessingConversion | str] | None

transformer_lens.model_bridge.supported_architectures.gidd.restore_frequencies(hf_model: Any) → bool

Recompute GIDD’s non-persistent frequencies rotary table; under v5’s meta-device load it materializes as uninitialized memory that silently corrupts every forward (applied to both bridge and HF reference so they agree).