transformer_lens.tools.model_registry.validate module¶

JSON schema validation for the model registry output files.

This module provides functions to validate that the JSON output files in the data/ directory conform to the expected schemas defined by the dataclasses in schemas.py and verification.py.

class transformer_lens.tools.model_registry.validate.ValidationError(path: str, message: str, value: Any = None)¶

Bases: object

Represents a validation error in a JSON file.

path¶

JSON path where the error occurred (e.g., “models[0].architecture_id”)

Type:: str

message¶

Description of the validation error

Type:: str

value¶

The actual value that caused the error (if applicable)

Type:: Any

message: str¶

path: str¶

value: Any = None¶

class transformer_lens.tools.model_registry.validate.ValidationResult(valid: bool, errors: list[ValidationError], schema_type: str)¶

Bases: object

Result of validating a JSON file against its schema.

valid¶

Whether the file passed validation

Type:: bool

errors¶

List of validation errors (empty if valid)

Type:: list[transformer_lens.tools.model_registry.validate.ValidationError]

schema_type¶

The schema type that was validated against

Type:: str

property error_count: int¶: Return the number of validation errors.

errors: list[ValidationError]¶

schema_type: str¶

valid: bool¶

transformer_lens.tools.model_registry.validate.validate_architecture_gaps_report(data: dict) → ValidationResult¶

Validate an ArchitectureGapsReport JSON object.

Parameters:: data – Dictionary loaded from JSON to validate
Returns:: ValidationResult with validation status and any errors

transformer_lens.tools.model_registry.validate.validate_data_directory(data_dir: Path | str | None = None) → dict[str, ValidationResult]¶

Validate all JSON files in the data directory.

Validates supported_models.json, verification_history.json, and architecture_gaps.json.

Parameters:: data_dir – Path to the data directory. If None, uses the default data directory.
Returns:: Dictionary mapping filenames to their ValidationResults

transformer_lens.tools.model_registry.validate.validate_json_schema(file_path: Path | str, schema_type: str | None = None) → ValidationResult¶

Validate a JSON file against its expected schema.

This function reads a JSON file and validates it against one of the model registry schemas. The schema type can be automatically inferred from the filename or explicitly specified.

Parameters:

file_path – Path to the JSON file to validate
schema_type – Schema type to validate against. If None, inferred from filename. Supported values: “supported_models”, “architecture_gaps”, “verification_history”

Returns:

ValidationResult with validation status and any errors

Raises:

FileNotFoundError – If the file does not exist
json.JSONDecodeError – If the file is not valid JSON
ValueError – If schema_type cannot be determined

transformer_lens.tools.model_registry.validate.validate_supported_models_report(data: dict) → ValidationResult¶

Validate a SupportedModelsReport JSON object.

Parameters:: data – Dictionary loaded from JSON to validate
Returns:: ValidationResult with validation status and any errors

transformer_lens.tools.model_registry.validate.validate_verification_history(data: dict) → ValidationResult¶

Validate a VerificationHistory JSON object.

Parameters:: data – Dictionary loaded from JSON to validate
Returns:: ValidationResult with validation status and any errors