transformer_lens.benchmarks.text_quality module

Text quality benchmark for TransformerBridge.

Generates text the way a real user of the model would (its prompt profile: chat template, translation source, code, own-language continuation — see text_quality_profiles) and scores each output against a known-good reference completion with one pinned multilingual judge. The score derives from the perplexity ratio PPL_judge(generated)/PPL_judge(reference), which cancels the judge’s per-language handicap; a repetition penalty catches degenerate loops (which the ratio alone rewards) and a length penalty catches truncated output.

Generation is seeded per prompt for reproducibility, and the judge is loaded once (CPU/fp32 always, so scores do not depend on the verifying machine) and reused across all prompts.

transformer_lens.benchmarks.text_quality.benchmark_text_quality(bridge: Any, profile: str | ProfileSpec = 'continuation', *, max_new_tokens: int | None = None, judge_model: Any | None = None, judge_tokenizer: Any | None = None, model_name: str | None = None) → BenchmarkResult

Benchmark text generation quality with profile prompts and reference-ratio scoring.

Generates from the model’s prompt-profile prompts through the real user path (bridge.generate), then scores each output against the prompt’s reference completion via the pinned judge’s perplexity ratio, with repetition and length penalties.

transformer_lens.benchmarks.text_quality.load_judge() → Tuple[PreTrainedModel, PreTrainedTokenizerBase]

Load the pinned judge on CPU in fp32 (machine-independent scores).