Alpha a2 release pilot — frozen before collection

This is an engineering pilot, not a preregistered accuracy study. Human review is an optional future validation stage, not a release gate. No LLM judge supplies labels.

Collection, fixed in advance

  • Two local model families: qwen3:1.7b and gemma3:1b, exact installed digests and model templates recorded in the manifest. No remote paid API.
  • Sixteen authored, fictional scenarios: customer support and interactive fiction, English and Spanish, four balance settings each. Eight families group language/model variants; these are not 32 independent scenarios or production sessions.
  • One output per model/scenario: 32 requests. No retries, selection by verdict, repairs, prompt tuning on the outputs, or omission of malformed/truncated responses.
  • Temperature 0, seed 1729, 4096 context, 512 output-token cap; Ollama JSON output mode. Thinking disabled if the model advertises that capability. Keep prompts/settings.
  • Trusted scenario states fixed by the application. Models produce prose and declared branch claims; do not derive reference state from model output.
  • Freeze this protocol, prompts, profile and scenarios in a hashed manifest before first generation. Record all responses and failures, token usage and elapsed time.

Analysis

Replay with exact declared-claim consistency, minimum tokens/diversity, lexical restatement and formula matching. Report invariant-only and heuristic-only profiles and a simple character-length assertion comparator. Count structural failures, missing attempts, unknowns/errors, findings, acceptance and latencies. Profile acceptance is not accuracy; no natural semantic precision/recall or model-quality ranking is identified without independently justified labels. Typed claims do not certify arbitrary prose. Infrastructure/power costs are unmeasured, even when API fees are zero.

Synthetic mutations stay separate, with operator-authored labels, exclusions, attributed detections, survivors and valid controls. No test-driven threshold changes on this pilot. Public weights are not redistributed. Publish fictional prompts and raw model outputs, the deterministic replay command and environment information.

Replaying stored outputs must reproduce reports/summary byte-for-byte with the pinned library version. Regenerating may differ by hardware/runtime despite fixed seed. The full independent-study protocol is future work in protocol.md.