Flow compensation

Evidence correction: historical kill counts from the native Core-1 mutation runner before 9a63b70ff4 are withdrawn pending fresh replay. Stale binaries could produce false kills. This qualifies all such counts below; see docs/mutation-evidence.md. Test inventories are not execution evidence.

The opt-in native v3 definition profile adds durable compensation to the existing reference reducer, compiled state machine and file journal. Source programs can register compensation through the separately allocated ZLM2 2.26 / ZLA2 1.18 profile. Existing native v2 and source definitions without compensation keep their existing definitions and ZFH2/ZFJ2 bytes. Neither addition promotes a benchmark fixture or activates stable Core-1.

Source contract

In a non-portable compiled Flow message with the flow effect, wrap an activity:

Flow.compensate(
    21, Work.undo,
    Flow.activity(11, Work.forward, input, 2, 2, true),
    7, 3, false
)

The arguments are compensation ID, typed undo callback, owning activity result, compensation schema, maximum attempts, and deduplication flag. Both callbacks must be public non-portable algorithm, state, or optimize messages with the Int64 -> Int64 signature; the enclosing effect set must contain their effects. Callbacks may be package imports. Metadata is explicit positive integer literals and a boolean, not runtime configuration. IDs share one namespace with all forward activities, timers and signals in the definition.

The wrapper returns the forward result as a dependency for an enclosing activity or timer. It registers the callback; it does not execute it during forward progress and does not add a forward node. Only an activity can own a registration, at most one per activity. Wrapping a parameter, timer, signal or already compensated activity rejects. Ordinary expressions between durable nodes remain unsupported by this bounded source profile.

Use the existing zl_flow_engine_start_module_i64_v2 and zl_flow_engine_attach_module_i64_v2 boundaries. Definitions with registrations select the v3 native descriptor automatically. The existing zl_flow_run_execute_definition_activity_i64_v2 dispatches either the current forward callback or its registered compensation, with that phase’s receipt policy. A callback failure is durably recorded and consumes its retry attempt. The scalar Flow evaluator does not execute durable graphs without an engine.

The complete reachable implementation closure includes undo callbacks and their helpers, even before unwind begins. Changing a reachable undo implementation rejects attachment and live execution with HISTORY_MISMATCH, without appending. Source/artifact round trips, package-file reordering, and edits confined to existing unreachable helpers preserve the epoch. There is no implicit migration to changed callback code.

Checked source artifact allocation

Schema 2.26 is allocated in spec/extensions/core1-flow-compensation.json and available through zlc-core zlm2-schema --minor=26 or zl_zlm2_wire_schema_v8. Earlier descriptor APIs retain their closed ranges; v8 returns byte-identical descriptors for minors 13–25. The default CLI minor remains 20.

Semantic kind 34 (flow-compensate) has one operand and recorded-activity operation 8. Its generated carrier retains the checked callback reference. In 2.26 each FlowIR row grows from 40 to 72 bytes; the first 40 bytes are unchanged:

Offset	Width	Meaning
40	8	Compensation node ID, or zero for no registration
48	4	Relocated compensation callback function index
52	4	Compensation activity domain
56	4	Compensation schema version
60	4	Compensation maximum attempts
64	1	Compensation deduplication boolean
65	7	Zero

The row’s forward node ID is the owner. Without a registration, the entire extension is zero. All other outer rows retain their allocations. ZLA2 1.18 uses the existing typed/owned structural records and admits the new canonical carrier name. Loaders re-lower the checked tree and compare every FlowIR field; rechecksummed edits to a registration or its association cannot bypass this redundancy check. Body identity binds the compensation contract and policy.

Public message reflection includes flow_compensations with the owning activity_id, compensation id, typed target, domain, schema_version, maximum_attempts, and deduplicated flag. These are checked declarations, not live engine authority. Runtime phase, attempts and outcomes remain visible through canonical history, polling and outcome APIs.

Native contract

Construct zl_flow_compensated_definition_v3, set its base.abi_version to ZL_FLOW_ABI_VERSION_V3 and its base.struct_size to the size of the whole extended descriptor, then pass &definition.base to zl_flow_engine_start_v2 or zl_flow_engine_attach_v2. The existing opaque engine/run handles and operation entry points remain applicable. The engine owns copies of both the forward nodes and compensation descriptors.

Each zl_flow_compensation_v3 associates a forward activity’s ID with a distinct compensation ID, activity domain, positive schema version, positive maximum attempts and independent deduplication flag. Registrations may be supplied in any order. Duplicate registrations, unknown or non-activity owners, zero or colliding IDs and invalid activity boundaries reject before a run is started. There may be no registrations, including on an empty forward graph.

Cancellation or exhausted forward retries initiate unwind. The observable behavior is:

1. Visit only completed, registered forward activities, in reverse graph order.
2. Poll ZL_FLOW_COMMAND_COMPENSATION_V3 with that registration’s ID, domain, schema and attempt. Its input is the recorded successful forward result, not the declared input, a failed forward attempt or a preceding undo result.
3. Complete or deliver through the existing activity APIs. Compensation failures retry under their own attempt bound. Successful compensation advances to the previous completed registration.
4. When unwind succeeds, expose the initiating CANCELLED/0 or FAILED/value. If compensation itself exhausts retries, stop with FAILED and its final failure value. Remaining registrations are not executed; history preserves both the original reason and the failed compensation.

Outcome remains RUNNING/0 while compensation is pending. Successful forward completion never compensates. A second cancellation, timer or signal during unwind rejects without an append. Dropping a handle does not cancel the run. The bounded zl_flow_run_execute_module_activity_i64_v2 adapter can execute an explicitly selected checked scalar callback; it checks the active compensation’s receipt policy and records callback success or failure before returning.

Compensation activities remain at-least-once. Exact repeated deduplicated deliveries acknowledge without appending, even across phases, terminal outcomes and reloads. Forward and compensation IDs are distinct, and each phase has its own receipt policy. Conflicting outcomes, values, receipts or future attempts reject. Non-deduplicated stale deliveries reject even with a receipt. These rules deduplicate durable acknowledgements, not arbitrary external effects; compensation is not an atomic rollback of external systems.

Canonical v3 allocation

ZFH3 retains the 40-byte header and 56-byte event rows used by ZFH2. ZFJ3 retains 128-byte frames with SHA-256 over bytes 0–95 and the checksum at 96–127. All existing fields retain their widths and offsets. Event kinds 0–5 keep their existing field meanings; kind 6 is compensation success and kind 7 compensation failure. Compensation event node_index is the forward graph index owning the registration, not its position in the registration table. Attempt is positive, value is Int64 and receipt is the same unrestricted 32-byte field.

The v3 start receipt is SHA-256 of ZFDEF3\0\0 followed by one 80-byte row per forward node in graph order. The first 40 bytes preserve the v2 canonical node allocation. The remaining fields are:

Row offset	Width	Meaning
40	8	Owning forward activity ID
48	8	Compensation activity ID
56	4	Compensation activity domain
60	4	Compensation schema version
64	4	Compensation maximum attempts
68	1	Compensation deduplication flag, 0 or 1
69	11	Zero

Integers are little-endian. Bytes 40–79 are all zero when the forward node has no compensation. Host padding, pointers and registration-table order never participate. Native definition IDs and code epochs remain caller-owned. Reattachment checks all semantic fields and the recorded digest before installing metadata. Rejected attachment does not make a later attempt valid.

An engine can read v2 and v3 frames, but a single run cannot change versions. Unknown versions and v2 compensation events reject. Checksums alone do not admit impossible phase transitions, orderings, attempts or post-terminal events. Older v2-only readers reject v3 frames; there is no implicit conversion.

Verification

The public native tests cover cancellation, exhausted forward retries, successful completion, empty completed prefixes, compensation exhaustion, owned metadata, invalid registrations, receipt policy, checked callback execution and checksum-valid forged histories.

The differential property runner destroys all engine state between transitions, reopens both independent read-only replay providers and the file writer, and compares commands, outcomes and exact history bytes. It permutes registration order, varies sparse registration subsets, retries both phases and uses signed inputs. Hypothesis runs 60 generated cases.

The existing parent-driven SIGKILL harness adds 30 frame-write/flush/fsync windows and 18 checkpoint-fsync/rename/directory-fsync windows. The previous published checkpoint is tested separately from the live journal. Unwind success and compensation exhaustion both participate.

The native extension was introduced test-first: commit c1f79be899 builds the new public descriptors and fails at runtime admission with expected status 0, got 1. Verification of the implementation passes all 47 selected Flow CTest groups on portable, LLVM and ASan/UBSan builds, plus 60 generated cases on each. The behavior-mapped mutation gates kill 20 compensation controls, 27 existing history-admission controls, 18 existing delivery-sequence controls and three re-anchored legacy retry controls. The 23 mutation-harness tests also pass.

ctest --test-dir out/build/core2-portable -R '^(runtime.core1_flow|benchmark.flow_)' --output-on-failure
ZERGLANG_FLOW_COMPENSATION_EXECUTABLE=out/build/core2-portable/zl_test_core1_flow_compensation_replay python -m pytest tests/properties/test_flow_compensation.py
python tools/run_core1_mutation_checks.py --suite flow-compensation --report out/verification/flow-compensation-mutants.json

The source tests also cover independent replay after checkpoint and full-state destruction, callback failures in all three permitted domains, compensation exhaustion, unchanged successful completion, imported undo closures, invalid owners and duplicate IDs, reflection, and checksum-valid artifact forgeries. The source/artifact property varies signed run inputs across all three engines. The source implementation follows behavioral-red commit 01a641cb31, which rejects Flow.compensate with ZL-FLOW-0001. All 53 selected Flow groups pass on LLVM and ASan/UBSan. Three property tests generate 130 cases on each of the portable, LLVM and sanitized builds, covering source/artifact replay, descriptor key-order invariance and rejection of allocation drift. All 25 targeted source compensation mutations are killed, with no invalid or surviving mutants. The public documentation projection is covered by the site’s 326 passing tests and TypeScript check. No new test framework is introduced. The full portable run passes 260 of 261 groups; the unrelated native-synthesis timeout-injection case passes its isolated rerun (27.69 seconds). Under the parallel full run, its two-second deadline expired one phase before the deliberate timeout. No compensation gate fails.

ctest --test-dir out/build/core2-portable -R 'flow_source_compensation|flow_compensation_artifact|zlm2_flow_compensation_layout' --output-on-failure
ZERGLANG_FLOW_SOURCE_COMPENSATION_EXECUTABLE=out/build/core2-portable/zl_test_core1_flow_source_compensation python -m pytest tests/properties/test_flow_source_compensation.py
python tools/run_core1_mutation_checks.py --suite flow-source-compensation --report out/verification/flow-source-compensation-mutants.json

The Exec/11 and Task/8 scenario protocol observes v3/v4 compensation, checked undo execution, retry exhaustion and fresh replay. Benchmark-level kill-point orchestration and fixture-by-fixture promotion remain follow-ups. The separate owned checkpoint profile adds v4 snapshot values and the ZLM2 2.27 source Dynamic boundary; broader Durable type admission remains outstanding. No existing catalog is rewritten by the scenario adapter.