Publishing benchmark evidence

Status: operational contract. The JSON Schemas and validators are the machine authority when this guide and an implementation disagree.

ZL256 evidence is append-only and content-addressed. A source workflow creates a sealed zerglang.benchmark-publication-request/1 bundle. A separately privileged workflow in the public release repository verifies it, creates a zlbench-<run_id> GitHub Release, and updates the small discovery index only after the immutable release is readable and its digests match.

The source workflow uploads the bundle as zlbench-publication-<run_id>, captures GitHub’s immutable artifact ID and digest, and uses a protected, short-lived GitHub App token to add exactly one locator at benchmark-requests/<run_id>.json in the release repository. The release workflow waits for the source workflow to conclude successfully, then binds the exact workflow attempt, commit, artifact ID, name, and digest before downloading any bytes.

Required identities

The request binds all information needed to reproduce or interpret a run:

• suite, dataset, profile revision and identity, lane, and execution view;
• source repository, ref, commit, and optional implementation release;
• workflow URL, run ID, attempt, and event;
• compiler identity, executor binary digest, harness version, exact command, filters, and shard description;
• OS, architecture, machine, and toolchain fingerprint;
• model and sampling configuration for synthesis, or null for conformance;
• start and completion timestamps and terminal status; and
• the exact path, role, media type, size, and SHA-256 of every public artifact.

The content digest is SHA-256 over canonical JSON after omitting run_id, content_sha256, and signature. The run ID is run- followed by the first 32 hexadecimal characters of that digest. Changing any behavioral or provenance fact therefore creates a different run.

Disclosure boundary

Every bundle explicitly asserts that it excludes:

• hidden tests;
• held-out inputs;
• held-out expected values;
• executable oracle code; and
• active reference solutions.

Public case inputs and their public expected values are intentionally visible. Synthesis candidates may also be visible, but the manifest and run page must say that publication can contaminate later model training or evaluation. Candidate files remain in the signed GitHub Release bundle; the Pages projection exposes only their signed paths, sizes, digests, and warning.

The bundle builder uses an allowlist; it does not copy a benchmark task directory recursively. A bundle is rejected if its inventory differs from the manifest, a path escapes the bundle root, a duplicate run already exists with different bytes, or any schema/digest/signature check fails.

Lanes

• Conformance runs publish on protected merges and release candidates.
• Performance collection currently runs monthly as non-gating private CI evidence. Public projection and publication are planned; when activated, comparisons will be valid only for identical machine and toolchain fingerprints.
• Automated model-synthesis publication is planned and is not active at launch. When activated, the denominator will be the set of compatible, non-scaffold tasks; unavailable and adapter-failure samples will remain separate from wrong answers.

Scaffold tasks cannot earn edition passes. A task-level result is fail if any applicable observation fails, otherwise gap if any applicable observation is a gap, otherwise pass if at least one applicable observation passes, and otherwise not-applicable. Every aggregate displays both its task denominator and raw observation denominator.

Required automation credentials

The protected source-repository benchmark-delivery environment holds a GitHub App ID/private key with Contents: write only on Epoch-ML/zerglang-releases. The release repository holds a separate App ID and private key with Actions: read and Metadata: read only on Epoch-ML/zerg, plus a dedicated Ed25519 benchmark signing key in its protected benchmark-publication environment. These credentials are distinct from the IDE updater and site-deployment keys.