Historical validator study protocol

Status: version 2 prepared 2026-09-29 (America/Argentina/Buenos_Aires), before execution.

This is a retrospective reproduction study, not an independent preregistration. The unit is a documented validator defect with a public primary issue, pull request, advisory, or release note naming the fix. A candidate is eligible only when the upstream record describes an observable validation or validation-adjacent behavior change and identifies both a before and after release (or commits). Configuration omissions, documented defaults, feature requests, type-checking only changes, and regressions without a later fix are excluded.

Before any reproduction, the candidate ledger, minimal inputs, expected behavior, adapter source, and this protocol are hashed. Reproductions use fresh virtual environments below work/; no package is installed into the study checkout or global interpreter. Network access is explicit (collect --network). The offline replay consumes only the captured normalized verdicts and validates their hashes.

Each included candidate has one valid baseline control and one invalid or newly accepted variant. The control must remain accepted in both versions. The expected finding is authored from the primary record and kept separate from adapter logic. audit_validator outcomes are reported verbatim; errors, unsupported cases, and failed baselines remain in the ledger and do not count as fixes. This protocol does not tune library rules and makes no claim that the harness outperforms pytest.

The frozen target is 8--12 candidates, with at least three reproduced fixes when the available Python/runtime/package constraints permit. Fewer successful cases are reported honestly. For every candidate we retain package/version/runtime, source URL and ID, rationale, input digest, adapter hash, raw before/after output, normalized audit report, and environment pip freeze hash.

Primary sources are upstream GitHub issues, pull requests, release notes, or security advisories. Minimal reproducers are authored for this study and do not copy upstream test files.

Preflight corrections and selection limitations

Version 1 is retained under preflight-v1/ and in Git commit 5035606. No package reproduction was executed with that preparation. Before collection, review of the original upstream reports found: uppercase file schemes and IDN email addresses are valid variations, not faults; the file example needs explicit schemes={"file"}; the boolean-ref issue requires a cached boolean schema in a RefResolver store rather than a plain local reference; and the IDN case should exercise a Unicode TLD. Those inputs/adapters and relations are corrected in v2. Unsupported release guesses for excluded candidates are replaced with null. The original serialization description for Pydantic #12348 was also corrected after reading its actual ModuleType reproducer. These are pre-execution corrections, not findings about the third-party libraries. Original flawed inputs are not evidence.

Every included issue now links to its upstream report and a commit-pinned changelog establishing the fixed version. The ten candidates were a purposive, bounded screening sample, not an exhaustive or randomly sampled review of issues. Exclusion means outside this experiment or insufficient documented evidence; it does not mean an upstream report is invalid. No population discovery rate follows.

Each included issue has two actual changed pairs: the issue-triggering variant and a valid preservation control. The enum issue is a violation; the other three are preservation issues, including a valid input triggering an exception. Replay compares the full recorded normalized results against the same a3 audit; exceptions remain errors rather than validation rejections. Baseline verdicts and child failures are retained. The summary calls a fix reproduced only when the before-version misbehaves on the trigger, the after-version satisfies it, and the control is preserved in both versions.

Freeze writes the prepared ledger and source/protocol hashes once. Setup never rewrites it. Collection verifies prepared hashes, uses isolated version-specific environments, and records exact installed dependency versions plus downloaded artifact hashes from pip's installation report. Re-collection installs those recorded dependency versions when supplied via --lock-from. Offline replay does not reinstall packages and is distinguished from fresh package execution. Hashes detect accidental changes; mutable manifests are not tamper-proof evidence.

Bootstrap-only execution deviation

Two initial collection attempts stopped while creating the first virtual environment, before installing a historical package or executing any sample. This macOS Python distribution aborts when its executable is copied by EnvBuilder; the collector now uses symlinks, matching the working command-line venv behavior. The source/protocol were re-frozen before sample execution. The preceding prepared snapshot is retained as prepared-v2-bootstrap/; case selection, labels and adapter logic are unchanged.