Home/Unified Causal Log Cutover Plan

Unified Causal Log Cutover Plan

C0 shadow strategy, wire protocol, divergence marker, and the e2e canary gate that gates C1

Overview

The unified-causal-log cutover is staged across multiple milestones, conventionally labelled C0, C1, and beyond. This page documents the C0 shadow plan — the phase in which the new causal log is computed and propagated in parallel with the existing log without replacing it. No reads are served from the shadow log during C0; its sole purpose is to validate correctness before any irreversible cutover step is taken.

The design doc at docs/design/unified-causal-log-cutover-plan.md has been updated with a dedicated C0 section. The material here reflects that section and records the rationale so maintainers can judge when it is safe to proceed to C1.

What C0 does

During C0 every node computes the unified causal log alongside the legacy log. The two logs are kept strictly separate:

  • The legacy log continues to drive all reads, consensus decisions, and state machine transitions.
  • The shadow log is computed from the same sequence of events and stored in a parallel structure that is never exposed to application logic.
  • At the end of each epoch boundary the node compares the root hash of the shadow log against the root hash transmitted by peers and records any divergence via the scope_root_shadow_divergence marker (described below).

C0 is considered in-progress until the e2e canary gate defined in this document passes all required conditions. C1 must not begin before that gate clears.

Shadow log computation method

The shadow log is built by replaying every committed event through the new causal-log assembler while the legacy path runs concurrently. The key properties of this computation are:

  1. Deterministic ordering — events are fed to the assembler in the same total order that the consensus layer delivers them, so the resulting DAG is identical on every correctly-behaving replica.
  2. Incremental hashing — each new event is appended to the shadow log using the same incremental Merkle construction that C1 will use for the live log, ensuring the hash function under test is the production one.
  3. Isolation — the assembler writes to a separate storage column family (or equivalent namespace) tagged shadow:. Reads from application state never touch this namespace during C0.
  4. No side effects — the shadow assembler is read-only with respect to consensus; it neither proposes nor votes and does not influence leader election.

The computation is gated behind a feature flag unified_causal_log_shadow that defaults to on for all nodes running the C0 release and can be disabled per-node for performance triage without affecting correctness of the live path.

Wire protocol additions for C0

C0 introduces a small extension to the gossip envelope so that peers can exchange shadow root hashes without polluting the legacy log protocol:

// Protobuf excerpt — added in C0
message ShadowRootAnnouncement {
  // Epoch sequence number this root covers.
  uint64 epoch_seq = 1;

  // Scope identifier — matches the existing ScopeId type.
  bytes scope_id = 2;

  // Merkle root of the shadow causal log at epoch_seq.
  bytes shadow_root = 3;

  // Sender's node identity for divergence attribution.
  bytes sender_node_id = 4;
}

Nodes broadcast a ShadowRootAnnouncement immediately after sealing each epoch. Receipt of an announcement triggers a local comparison:

  • If the received shadow_root matches the locally computed root, no action is taken.
  • If they differ, the node emits a scope_root_shadow_divergence structured log event and increments the corresponding Prometheus counter.

The announcement messages are not stored in the legacy log and carry no consensus weight. They are fire-and-forget gossip with no acknowledgement requirement during C0.

The scope_root_shadow_divergence marker

The scope_root_shadow_divergence marker is the primary observability signal for C0. It is emitted as a structured log event at WARN level and exported as a Prometheus counter with the following labels:

  • scope_id — the affected scope.
  • epoch_seq — the epoch at which the divergence was first observed.
  • local_root — hex-encoded local shadow root (first 8 bytes for brevity).
  • remote_root — hex-encoded remote shadow root (first 8 bytes).
  • remote_node_id — the peer whose announcement triggered the comparison.
// Example structured log line
{
  "level": "WARN",
  "target": "calimero_causal_log::shadow",
  "message": "scope_root_shadow_divergence",
  "scope_id": "abc123",
  "epoch_seq": 4821,
  "local_root": "d4e8f1a2...",
  "remote_root": "9c3b77e0...",
  "remote_node_id": "node-07"
}

A divergence does not halt the node or trigger any recovery during C0. It is purely diagnostic. The canary gate (see below) aggregates these counters across the test fleet to determine whether the shadow implementation is ready for C1.

Hash-neutral governance rotation — the critical canary scenario

A hash-neutral rotation is a governance event that changes the membership or key material of a scope without altering the application state hash. Examples include:

  • Rotating a node's signing key while keeping its identity stable.
  • Adding or removing a passive observer that holds no voting weight.
  • Updating scope metadata fields that are excluded from the state Merkle tree.

Hash-neutral rotations are the hardest class of event for the shadow log to handle correctly because the legacy log's root hash does not change across the rotation, but the unified causal log's root must change — it encodes governance events explicitly in its DAG structure.

The canary gate therefore requires the test harness to inject at least one hash-neutral governance rotation per test run and assert that the shadow divergence counter fires on nodes that have not yet processed the rotation and clears once all nodes converge. A canary that never fires on a rotation indicates the shadow log is silently ignoring governance events, which would be a correctness defect.

The e2e canary gate — pass conditions

The canary gate is an end-to-end test suite that must reach a steady-state pass before C1 is permitted to begin. The gate has two distinct pass conditions that must both be satisfied:

Condition 1 — zero false positives at steady state

After a fully-converged cluster has processed all pending events and all nodes agree on epoch boundaries, the scope_root_shadow_divergence counter must read zero across all nodes for a sustained observation window (currently defined as 30 consecutive epochs with no divergence events).

  • Transient divergences during event processing are expected and do not fail this condition as long as they resolve before the window closes.
  • A persistent non-zero counter at steady state indicates a non-determinism bug in the shadow assembler and must be fixed before proceeding.

Condition 2 — fires on hash-neutral governance rotation

The test harness must confirm that injecting a hash-neutral governance rotation causes the divergence counter to increment on at least one node during the convergence window, and that the counter returns to zero after all nodes process the rotation.

  • This condition validates that the shadow log is sensitive to governance events that the legacy hash does not capture.
  • A canary run in which no divergence fires during a rotation is treated as a gate failure, not a pass, because it indicates the event was silently dropped.

Both conditions must pass on three consecutive canary runs before C1 work is scheduled.

Gate failure modes and remediation

The following table describes the failure modes the canary gate is designed to catch and the expected remediation path for each:

  • Persistent divergence at steady state — indicates non-deterministic event ordering or a hash function mismatch between the local assembler and peers. Remediation: audit the event ingestion pipeline for ordering guarantees and verify the assembler's hashing matches the wire spec.
  • No divergence on hash-neutral rotation — indicates governance events are not being fed to the shadow assembler. Remediation: verify the governance event bus is wired to the shadow path behind the feature flag.
  • Divergence that never clears — indicates a convergence bug where the shadow DAG cannot reconcile after a fork. Remediation: inspect the merge logic in the causal-log assembler and check for missing causal links in the gossip path.
  • Counter fires on non-divergent epochs — indicates a clock or epoch-boundary mismatch causing nodes to compare roots from different epochs. Remediation: validate that epoch_seq in ShadowRootAnnouncement is derived from the same consensus clock on all nodes.

C0 to C1 transition criteria — summary

A maintainer may schedule C1 work only when all of the following are true:

  1. The e2e canary gate has passed three consecutive runs with zero false positives at steady state.
  2. The canary has fired and cleared correctly on a hash-neutral governance rotation in each of those runs.
  3. No open issues are tagged c0-blocker in the issue tracker.
  4. The scope_root_shadow_divergence Prometheus counter has been reviewed by at least one maintainer who is not the author of the shadow assembler.
  5. The design doc section for C1 has been written and reviewed, incorporating any lessons learned during C0.

Do not begin C1 speculatively. The shadow plan exists precisely to surface correctness issues at zero operational risk. Cutting over before the gate passes converts a correctable bug into a potential data-integrity incident.

Feature flag lifecycle

The unified_causal_log_shadow feature flag introduced in C0 follows the standard flag lifecycle for this codebase:

  • C0 release — flag defaults to on; can be disabled per-node via config for performance triage.
  • C1 release — flag is removed; shadow path is promoted to the live path; the legacy path becomes the shadow for a brief overlap window (the C1 shadow plan will describe this separately).
  • Never ship C1 with the C0 flag still present — having two shadow flags simultaneously creates an untestable combination space.

Config key for the flag:

# calimero-node.toml
[experimental]
unified_causal_log_shadow = true   # default; set false to disable C0 shadow

Relationship to the existing design doc

The canonical design document lives at docs/design/unified-causal-log-cutover-plan.md in the repository root. The C0 section added by this change is the authoritative specification; this architecture page is a derived summary intended for browsing alongside other architecture pages.

If this page and the design doc disagree, the design doc takes precedence. File a documentation bug to reconcile the two.

  • docs/design/unified-causal-log-cutover-plan.md — authoritative cutover plan; C0 section is in-progress.
  • crates/causal_log/src/shadow/assembler.rs — shadow assembler implementation.
  • crates/causal_log/src/shadow/gossip.rs — ShadowRootAnnouncement broadcast logic.
  • crates/causal_log/src/shadow/divergence.rs — scope_root_shadow_divergence marker emission.
  • e2e/causal_log_canary/ — end-to-end canary test suite including the hash-neutral rotation scenario.

C1 incremental landing plan — C1a and C1b

C1 is split into two slices to reduce the blast radius of the live-log cutover and to defer subsystems that carry their own staleness semantics.

C1a (current PR)

C1a lands the HashComparison end-of-session verdict: the unified causal log is promoted to the live path for the HashComparison comparison site, which produces a verdict at the end of each session. This is the lowest-risk cutover point because session boundaries are already natural quiesce points with no in-flight reads.

  • The C0 canary gate (three consecutive passing runs, as described above) is the merge gate for C1a. C1a must not land until that gate clears.
  • The unified_causal_log_shadow feature flag is removed in C1a; the shadow path is promoted to live.
  • All other comparison sites remain on the legacy log after C1a lands.

C1b (completed 2026-06-24)

C1b updated the remaining comparison sites that were intentionally left on the legacy log in C1a. The LevelWise advisory handshake-root verdict was folded onto scope_root as planned. Two sites were deliberately excluded from the scope_root migration and remain entity-root-based by explicit design decision:

  • Snapshot streaming-integrity boundary (snapshot.rs) — kept entity-root-based because the snapshot boundary check validates the integrity of streamed entity data at ingestion time. Threading scope_root through this path would cause the check to accept or reject a snapshot based on causal-log state that may not yet reflect the snapshot's own epoch, silently masking ingestion of a snapshot built against a stale causal root.
  • Protocol selector 'roots match → skip sync' arm — kept entity-root-based because this no-sync fast path compares entity roots as a proxy for whether a full sync can be skipped. Switching it to scope_root would conflate governance-event divergence (which the unified causal log captures) with data-state divergence (which drives the sync decision), potentially causing unnecessary syncs or, worse, skipping a sync when entity state has genuinely diverged.

These exclusions are permanent for C1b. If a future milestone determines that either site should be moved to scope_root, that work requires its own design pass and canary gate.

C1b also introduces the reversed-shadow arrangement: the legacy log becomes the shadow checked against the now-live unified log. The existing scope_root_shadow_divergence marker infrastructure was evaluated for polarity changes as part of the C1b design doc.

Open questions for C1 planning

The following questions are deliberately left open until the C0 canary gate passes. They should be resolved as part of writing the C1 design doc section:

  • Read cutover ordering — should reads switch to the unified log atomically at an epoch boundary, or is a gradual per-scope migration safer?
  • Legacy log retention — how long must the legacy log be retained after C1 for rollback purposes, and what is the storage cost on large scopes?
  • C1 shadow direction — during C1 the roles reverse: the unified log becomes live and the legacy log becomes the shadow. Does the existing divergence marker infrastructure need changes to support this reversed polarity?
  • Governance event backfill — nodes that were offline during C0 will have gaps in their shadow log. Does C1 require a backfill protocol, or is a fresh shadow-log start at C1 cutover acceptable?
  • C1b comparison-site ordering — resolved: LevelWise advisory verdict was migrated to scope_root; snapshot boundary verdict and protocol selector converged-decision were explicitly kept entity-root-based (see C1b section above).

None of these questions need to be answered before C0 is declared complete. They are recorded here so they are not forgotten when C1 planning begins.