transformer_lens.benchmarks.text_quality_profiles module¶

Prompt profiles and reference data for the Phase-4 text-quality benchmark.

Each verified model is scored on prompts a real user would feed it (its prompt_profile): chat models get their chat template, translation models get source sentences, code models get code, multilingual models get their own language. Every prompt carries a known-good reference completion; scoring is the ratio of judge perplexities PPL(generated)/PPL(reference), which cancels the judge’s per-language handicap.

Profile resolution is curation-first because Hub metadata is unreliable (observed live 2026-08-20): bigscience/mt0-base is mis-tagged text-generation; facebook/m2m100_418M and google/long-t5-tglobal-base have no pipeline_tag at all; the conversational tag is added by HF for any repo shipping a chat template, including base models like Qwen/Qwen2.5-0.5B; Helsinki-NLP language tags are unordered, so Marian direction must come from the model id. Precedence: per-model override > architecture rule > fetched HF signals > stored registry value > default.

Pivot sentences are from Tatoeba (https://tatoeba.org, CC BY 2.0 FR); source sentence ids are noted inline. Everything else is hand-authored.

This module stays stdlib-only: the registry scraper imports it at scan time.

Language x kind coverage (prompts exist where marked; uncovered combinations SKIP with a file-an-issue message, they never score against wrong-language data):

kind en fr es de zh ja ru ar hi it nl pt ro code continuation x x x x x x x x - - - - - x chat x x x x x x x x - - - - - - task:instruction x x - - x - - - - - - - - - task:summarization x x - - x - - - - - - - - - task:denoise x - - - - - - - - - - - - - PIVOT (translation) x x x x x x x x x x x x - -

hi/it/nl/pt have pivot coverage only (translation targets); ro exists only in NLLB_CODES. Filling continuation/chat for those plus it/nl/pt/hi bake-off calibration is tracked as a follow-up.

class transformer_lens.benchmarks.text_quality_profiles.HFSignals(pipeline_tag: str | None = None, languages: tuple[str, ...] = (), tags: tuple[str, ...] = ())¶

Bases: object

Distilled Hub metadata for one model, as fetched by the scraper.

languages: tuple[str, ...] = ()¶

pipeline_tag: str | None = None¶

tags: tuple[str, ...] = ()¶

class transformer_lens.benchmarks.text_quality_profiles.ProfilePrompt(prompt: str, reference: str, lang: str = 'en')¶

Bases: object

One scored sample: model input and a known-good reference completion.

lang: str = 'en'¶

prompt: str¶

reference: str¶

class transformer_lens.benchmarks.text_quality_profiles.ProfileSpec(kind: str, lang: str = 'en', src: str | None = None)¶

Bases: object

A parsed prompt profile: what to feed the model and in which language.

kind: str¶

lang: str = 'en'¶

classmethod parse(spec: str) → ProfileSpec¶: Parse kind[@lang] (translation: @src-tgt); ‘@’ because task kinds contain ‘:’.

src: str | None = None¶

transformer_lens.benchmarks.text_quality_profiles.extract_languages(card_data_language: object, tags: object) → tuple[str, ...]¶: Normalize cardData.language (str or list) plus tag-list ISO codes, noise dropped.

transformer_lens.benchmarks.text_quality_profiles.is_default_profile(profile) → bool¶: One sparse-encoding rule for every registry writer: the bare default continuation@en profile is never stored (a lang-tagged continuation is).

transformer_lens.benchmarks.text_quality_profiles.p4_pass_threshold() → float¶: The P4 pass line, derived from the bake-off noise floor. The registry floor imports this so [floor, pass) can never silently diverge again.

transformer_lens.benchmarks.text_quality_profiles.profile_from_hf_signals(model_id: str, architecture_id: str, signals: HFSignals) → ProfileSpec | None¶: Distill fetched Hub metadata into a profile, or None when it says nothing.

transformer_lens.benchmarks.text_quality_profiles.prompts_for(spec: ProfileSpec, denoise_style: str = 't5') → tuple[ProfilePrompt, ...] | None¶: Prompt set for a profile, or None when coverage is missing (caller skips with a file-an-issue message naming the gap).

transformer_lens.benchmarks.text_quality_profiles.resolve_profile(model_id: str, architecture_id: str | None, registry_profile: str | None = None, signals: HFSignals | None = None) → ProfileSpec¶: Resolve a model’s profile: override > architecture rule > live signals > stored registry value > default (seq2seq falls back to denoising).