Sync Engine
Sync protocols from calimero-node · calimero-node-primitives
Purpose
Two independent synchronization systems keep peers converged: application state sync (context DAG) and group governance sync (governance DAG). The SyncManager runs periodic and on-demand sync cycles. Protocols are selected dynamically via handshake negotiation based on divergence metrics.
Protocol Selection
Peers exchange a SyncHandshake to compare state. The comparison metrics determine which protocol minimizes bandwidth and latency for the current divergence level.
Application State Protocols
Four protocols for synchronizing context application state. Each is optimized for a different divergence scenario.
Hash Comparison — Merkle tree walk
The lightest sync protocol. Both peers share their Merkle tree root hash. If roots differ, they exchange hashes at progressively deeper levels to locate the divergent subtrees. Only the changed subtrees are transferred.
Algorithm
Root Compare
Exchange root hashes. If equal, sync is a no-op. If different, descend to the next level.
Level Descent
Exchange hash nodes at the current depth. Mark subtrees with matching hashes as "synced" and skip them. Continue descending into differing subtrees.
Leaf Transfer
At leaf level, exchange the actual data entries for divergent nodes. Apply incoming entries to local storage and update the Merkle tree.
Best for: Small, localized changes — a few keys updated in a large state tree.
crates/node/src/syncLevel-wise — wide-tree traversal
Optimized for broad, shallow divergence where many branches differ but the tree isn't deep. Instead of drilling into individual branches, this protocol processes entire levels in parallel.
Algorithm
Level Exchange
Both peers serialize their entire tree at the current level and exchange the batch. This avoids the round-trip overhead of per-branch negotiation.
Batch Diff
Compare the received level against the local level. Identify additions, deletions, and modifications in a single pass.
Descend Parallel
For each differing node, descend to the next level. Process all descendants in parallel rather than sequentially.
Best for: Many keys changed across different parts of the tree — bulk updates, migrations.
Snapshot — full state transfer
Fallback protocol for maximum divergence or when a peer has no prior state (new joiner). Transfers the entire state as paged chunks with a DeltaBuffer to capture mutations during transfer.
Algorithm
Snapshot Begin
Source peer takes a consistent read snapshot and begins chunking the state into pages of snapshot_chunk_size bytes. A DeltaBuffer is opened to capture any writes that occur during the transfer.
Chunk Transfer
Pages are streamed to the target peer. Each page includes a sequence number and hash for integrity verification. The target applies pages to a staging area.
DeltaBuffer Replay
After all pages are transferred, the accumulated DeltaBuffer entries (mutations that occurred during the snapshot) are replayed on top of the restored state.
Commit
The staging area is atomically swapped into the live state. The Merkle tree is rebuilt from the new state.
Best for: New peers joining a context, or peers that have been offline for a very long time.
Delta Sync — direct DAG exchange
Direct exchange of causal delta entries between peers. Uses bloom filters to efficiently identify which deltas the remote peer is missing, and subtree prefetch to minimize round-trips.
Algorithm
Bloom Filter Exchange
Each peer builds a bloom filter of their DAG entry hashes and exchanges it. This allows each side to quickly identify entries the other is likely missing without enumerating the full DAG.
Delta Request
Based on bloom filter results, the receiver requests specific delta entries by hash. The sender responds with the full CausalDelta payloads.
Subtree Prefetch
When a requested delta has parent hashes that the receiver also lacks, the sender proactively includes the parent chain (subtree prefetch) to reduce additional round-trips.
Apply + Verify
Received deltas are verified for causal consistency (all parents present) and applied to the local DAG in topological order. The Merkle tree is incrementally updated.
Best for: Moderate divergence with known recent activity — the most commonly used protocol during normal operation.
Group Governance Sync
Governance operations form a separate DAG and use dedicated sync mechanisms. Three complementary approaches ensure governance state converges across all group members.
Projection-Authoritative Gossip Write Authorization (authorize_delta_at_edge_projected)
The gossip state-delta path (handle_state_delta) calls authorize_delta_at_edge_projected in verify.rs. This function is now the sole authorizer for all delta authorization paths — the live authorize_delta_at_edge function backed by acl_view_at has been deleted from verify.rs and is no longer re-exported from state_delta/mod.rs. No live resolver is consulted on any path; no divergence cross-check or shadow logging is performed.
authorize_delta_at_edge_projected accepts a caller-supplied resolve closure that returns a CutMembership enum. All call sites — gossip apply, parent-fetch (request_missing_deltas / verify_fetched_parent), snapshot-replay (replay_buffered_delta), and DAG-catchup in SyncManager — now pass a closure backed by the shared resolve_cut_membership renderer (see below).
The three CutMembership variants map directly to authorization outcomes:
- Member(role) → Authorized — projection confirms membership with a known role; write proceeds normally.
- NotMember → MembershipReject — projection definitively rejects; delta is discarded.
- Incomplete → Buffer — governance ancestry is not yet fully folded (no embedded needed set; the buffer's needed heads are populated from the governance position's heads); delta is held in the pending buffer and re-evaluated when governance catches up.
The entire previous co-authorizer / shadow-logging block — live authorize_delta_at_edge call, divergence markers, role-shadow logging (unified_projection_divergence / plane=data-write-role), decision-shadow logging (data_write_decision_shadow), and the MembershipReject grant-override via projection_member_at_cut_authoritative — has been removed.
DeltaAuthOutcome::MembershipReject no longer carries a group: ContextGroupId field. Code matching on this variant must be updated to omit the field (use .. if forward-compatibility is needed).
Scope: This change now applies to all authorization paths. Parent-fetch (request_missing_deltas and verify_fetched_parent) and snapshot-replay (replay_buffered_delta) gain a new node_state: &NodeState parameter to carry the projection reference. The DAG-catchup head-pull in SyncManager is similarly updated.
Visibility: projection_member_at_cut is refactored to delegate to resolve_cut_membership (discarding the role) so both helpers cannot drift. resolve_cut_membership is pub(crate) and is the single projection verdict renderer.
// authorize_delta_at_edge (live/acl_view_at-backed) has been deleted
pub fn authorize_delta_at_edge_projected<F>(
ctx: &DeltaContext,
resolve: F,
) -> DeltaAuthOutcome
where
F: FnOnce() -> CutMembership
// Shared projection verdict renderer (state_delta/mod.rs)
// Refreshes scope projection (write lock), then reads member + role
// under a single read guard for epoch consistency.
// Called by gossip-apply, parent-fetch, snapshot-replay, DAG-catchup.
pub(crate) fn resolve_cut_membership(
node_state: &NodeState,
ctx: &DeltaContext,
) -> CutMembership
// CutMembership — Incomplete carries no embedded needed set
pub enum CutMembership {
Member(Role), // → Authorized
NotMember, // → MembershipReject
Incomplete, // → Buffer (governance ancestry still folding; no needed set)
}
// MembershipReject no longer carries a group field
// DeltaAuthOutcome::MembershipReject (no group: ContextGroupId)
Operators: Gossip deltas whose governance ancestry is not yet folded are now held in the pending buffer rather than discarded. No unified_projection_divergence or data_write_decision_shadow warnings are emitted from any write-auth path. These log markers no longer appear for the data-write plane and any alerts configured on them for this path may be retired. Doc comments in peer_identity_cache.rs and namespace_sync.rs have been updated to reference projection-based membership resolution rather than the deleted live resolver.
Projection Rebuild Indirection — ops_for_namespace
ScopeProjections::ops_for_namespace is the new stable indirection point for loading ops at every projection-rebuild site. It currently delegates directly to collect_namespace_ops (the governance-DAG walk) — it does not yet read from the unified op-store (load_scope_ops). The C2.2b flip to the op-store is explicitly deferred until late-decrypted membership is fixed.
Root cause for the deferral: an encrypted MemberAdded op can be applied to the governance DAG before its group key arrives. The apply-time dual-write freezes it as Noop in the op-store, while collect_namespace_ops re-decrypts at read time once the key lands. Using the op-store as the projection source would therefore silently drop members and break joiner sync.
All three internal projection-rebuild call sites in scope_projection.rs — backfill_namespace, member_now_ephemeral/ephemeral_fold, and the scope_root reconstruction — and the call site in refresh_projection_for_cut in the node crate now call ops_for_namespace instead of collect_namespace_ops directly. collect_namespace_ops remains as the implementation that ops_for_namespace currently delegates to.
Inside refresh_projection_for_cut, after ops_for_namespace returns the governance-DAG fold, ScopeProjections::check_op_store_completeness is called with those ops before apply_backfill. This adds one op-store read on cold/refresh cuts (warm cuts skip via namespace_to_refresh returning None). The result is ignored by the caller; the gate is purely diagnostic and never affects whether or how the backfill is applied. The projection still folds via collect_namespace_ops.
// Currently delegates to collect_namespace_ops (governance-DAG walk)
// Will be flipped to load_scope_ops (op-store) once late-decrypt is fixed
pub(crate) fn ops_for_namespace(
&self,
namespace: &Namespace,
) -> Result<Vec<Op>>
// Observe-only completeness gate — called in refresh_projection_for_cut
// after ops_for_namespace, before apply_backfill. One op-store read on
// cold/refresh cuts; skipped on warm cuts. Result is always ignored.
pub(crate) fn check_op_store_completeness(
&self,
ops: &[Op],
) -> Result<()>
// Implementation (unchanged); called only via ops_for_namespace
fn collect_namespace_ops(
&self,
namespace: &Namespace,
) -> Result<Vec<Op>>
// Removed from ScopeProjections:
// shadow_compare_op_store — cross-check was unreliable (see above)
// scope_root_from_governance_dag — companion to shadow_compare_op_store
Operators: No shadow_compare_op_store log output is emitted from the backfill path. Any monitoring rules that relied on post-backfill op-store comparison metrics from this path may be retired. The ops_for_namespace indirection is the future flip point; watch for a follow-up change that routes it to load_scope_ops once late-decrypted membership handling is resolved. check_op_store_completeness is diagnostic only — its result is discarded and it does not block or alter the backfill.
Op-Store Refresh on Late Key Delivery (recover_missing_group_keys) — sole persist trigger
SyncManager::recover_missing_group_keys is the single runtime chokepoint for direct group-key delivery. Immediately after a successful key delivery and the existing drain/reconcile steps, ScopeProjections::repersist_namespace_ops is called for the affected namespace_id. This ensures that any MemberAdded ops that were frozen as Noop in the op-store (because their group key had not yet arrived at apply time) are re-decrypted and written into the op-store immediately when the key lands — without waiting for the next projection rebuild cycle.
ScopeProjections::persist_namespace_head_ops — a thin alias that previously documented a per-site 'C3 Stage 1/2/3' persist pattern — has been deleted. All direct call sites (create_group, delete_group, join_context, join_subgroup_inheritance, remove_group_members, join_namespace, reparent_group, create_group_in_namespace) have been removed. repersist_namespace_ops called from recover_missing_group_keys is now the sole trigger for op-store persistence of governance ops.
Because all late-arriving keys flow through recover_missing_group_keys, no other call site needs to trigger repersist_namespace_ops for the key-delivery case. The call is placed after drain/reconcile so the projection is already updated before the op-store is refreshed.
pub(crate) fn repersist_namespace_ops(
&self,
namespace_id: NamespaceId,
) -> Result<()>
Operators: After this change, late-decrypted MemberAdded ops are persisted to the op-store as soon as their key arrives rather than on the next projection rebuild. No new log markers are emitted; the existing key-delivery trace spans cover this path.
Post-Backfill Shadow Compare (refresh_projection_for_cut) — Removed
After apply_backfill writes the governance-DAG fold into the maintained projection inside refresh_projection_for_cut, a post-fold cross-check against the op-store was previously triggered via shadow_compare_op_store. This call and both shadow_compare_op_store and scope_root_from_governance_dag have been removed from ScopeProjections.
The cross-check was flawed: the maintained projection holds live ingest_op-fed ops that may not yet be in the op-store (an encrypted MemberAdded op can be applied to the governance DAG before its group key arrives, causing the apply-time dual-write to freeze as Noop in the op-store while collect_namespace_ops re-decrypts at read time once the key lands). The op-store gap was therefore silently masked by the live apply path, making the shadow compare unreliable. No observational backfill check of this kind is performed.
Projection-Authoritative Drain (drain_governance_pending)
State deltas that arrived before their governance ancestry was locally available are buffered and re-evaluated by drain_governance_pending once new governance ops are applied. This drain now uses member_at_cut_authoritative (projection) as its sole decision authority, replacing the previous acl_view_at (live DAG walk) primary path.
The three projection outcomes map directly to the three drain outcomes:
- Some(true) → re-apply — delta is re-submitted to the normal apply path. No cross-check is performed; the re-apply decision is purely from member_at_cut_authoritative.
- Some(false) → drop — delta is discarded; the author is definitively not a member at that cut. acl_view_at is called after the drop decision solely to label the drop metric ("removed" vs "never_member"); no unified_projection_divergence warn is emitted.
- None → re-buffer — governance ancestry is still incomplete; the delta stays pending for the next drain cycle.
Because the projection never returns an error, the previous Err match arm that dropped a buffered delta and recorded a "lookup_error" metric has been eliminated entirely.
acl_view_at Usage
acl_view_at is called only in the Some(false) (drop) arm, and only to obtain a best-effort metric label. It is not called in the Some(true) arm (no grant-plane cross-check) and not called on None (to avoid false positives while ancestry is still being folded). No unified_projection_divergence warning is emitted on any drain path.
Metric Changes
The governance_drain_outcome metric label for drop events is best-effort. It is derived from acl_view_at's response in the drop arm: "removed" if live says Removed, "never_member" if live says NeverMember, and the fallback "not_member" if the live resolver is inconclusive. The "lookup_error" label is no longer emitted. Grant-arm (Some(true)) drop metrics are unaffected as acl_view_at is no longer called there.
The debug log emitted when a delta is re-buffered no longer includes a needed_count field, because the projection returns a boolean None rather than a structured variant enumerating missing DAG heads.
// Some(true) → re-apply
// Some(false) → drop (metric label: "removed" | "never_member" | "not_member")
// None → re-buffer (no needed_count in log)
pub fn member_at_cut_authoritative(
ctx: &DeltaContext,
cut: &Cut,
) -> Option<bool>
Operators: No unified_projection_divergence warns are emitted from the drain path. Update any metric dashboards to handle the new "not_member" label and the absence of "lookup_error" on governance_drain_outcome. Deltas that previously reached the drain via the live-path MembershipReject drop-with-no-recovery will now instead arrive in the drain via the gossip path's Incomplete → Buffer outcome, so drain throughput may increase during governance catch-up periods.
Inbound-Sync Peer Authorization (peer_is_group_member)
A new private method SyncManager::peer_is_group_member encapsulates membership authorization for inbound sync peers. It resolves the current governance heads via ScopeProjections::namespace_current_heads and calls projection_member_at_cut to obtain the projection's verdict. If the projection returns None (cold or partially folded governance), the method falls back to the live result (MembershipRepository::is_member) so no peer is spuriously rejected while governance ancestry is still being caught up. When the projection returns Some(x) it is used as the authoritative decision with no cross-check warn emitted.
Two call sites inside SyncManager that previously called MembershipRepository::new(store).is_member(...) directly — the materialization-wait verification and the inherited-member check in verify_inbound_member — now call self.peer_is_group_member(...), so the projection-first logic applies uniformly to all inbound peer authorization.
async fn peer_is_group_member(
&self,
group_id: GroupId,
peer_id: PeerId,
) -> Result<bool>
// projection returns None → fall back to live (projected.unwrap_or(live))
// Some(x) → authoritative; no cross-check warn emitted
Operators: A None projection result is not an error — it means governance ancestry is still being folded and the live result is used as a safe temporary answer. No unified_projection_divergence warning is emitted on this path.
Real-time Gossip
primarySignedGroupOpV1 operations are published via gossipsub on dedicated group topics. Each operation is verified on ingress — signature check, capability check, parent hash validation. Valid operations are fed directly into the local DagStore.
pub fn publish_group_op(
topic: &GroupTopic,
op: &SignedGroupOpV1,
) -> Result<()>
Heartbeat Comparison
periodicGroupStateHeartbeat is broadcast every 30 seconds. Contains the group's current dag_heads (latest operation hashes). Peers compare heads and trigger catch-up if they differ.
group_id: GroupId,
dag_heads: Vec<Hash>,
op_count: u64,
}
Stream Catch-Up
on-demandWhen heartbeat comparison reveals a mismatch, a direct stream is opened to the divergent peer. The peer exchanges GroupDeltaRequest specifying known heads, and receives a GroupDeltaResponse with the missing operations scanned from the remote OpLog.
group_id: GroupId,
known_heads: Vec<Hash>,
}
pub struct GroupDeltaResponse {
ops: Vec<SignedGroupOpV1>,
has_more: bool,
}
Startup Recovery
bootOn node startup, reload_group_dags reads the persistent OpLog from RocksDB and reconstructs all group DAGs in memory. The DAG heads are then compared with peers via the first heartbeat cycle to catch up on any operations missed while offline.
pub async fn reload_group_dags(
store: &RocksDBStore,
) -> Result<HashMap<GroupId, DagState>>
SyncManager
Long-running async task (not an Actix actor) that orchestrates all synchronization. Runs periodic sync cycles and handles on-demand sync triggered by incoming streams.
Configuration
timeout: Duration, // per-sync timeout
interval: Duration, // periodic sync interval
frequency: u32, // syncs per interval
max_concurrent: usize, // max parallel sync streams
snapshot_chunk_size: usize, // bytes per snapshot page
delta_sync_threshold: u64, // entity count for delta vs snapshot
}
Core Loop
start() — Spawn Loop
The start() method spawns the main sync loop as a Tokio task. It runs indefinitely, waking on a timer tick or when an incoming stream is opened by a remote peer.
Periodic Tick
On each interval tick, iterates over all active contexts. For each context, builds a SyncState (current root_hash, dag_heads, entity count) and selects a peer for synchronization.
Peer Selection
Selects a sync peer based on: most recently seen heartbeat, lowest latency, and whether they reported a different state hash. Avoids peers already in an active sync session.
Concurrent Futures
Sync operations run as bounded concurrent futures via FuturesUnordered, capped at max_concurrent. Each future handles one context-peer sync session from handshake through completion.
handle_opened_stream()
When a remote peer opens a stream, reads the StreamMessage::Init frame, extracts the InitPayload, and dispatches to the appropriate sync protocol handler based on the negotiated protocol type.
Stream Wire Types
Init(InitPayload),
Message(MessagePayload),
OpaqueError,
}
context_id: ContextId,
protocol: SyncProtocol,
handshake: SyncHandshake,
}
HashNodes(Vec<HashNode>),
LevelData(LevelBatch),
SnapshotChunk(PageData),
DeltaEntries(Vec<CausalDelta>),
BloomFilter(BloomData),
// DagHeadsResponse carries an optional governance scope root
// None = responder could not fold scope (non-group ctx or store fault)
DagHeadsResponse {
scope_root: Option<Hash>,
},
}
Post-Sync Governance Reconciliation (handle_dag_sync)
Governance divergence check moved to handle_dag_sync
Governance reconciliation after a sync session is now owned exclusively by handle_dag_sync in the sync manager, not by individual protocol backends. After execute returns Ok(Some(_)) — meaning a data backend actually ran (Hash Comparison, LevelWise, Snapshot, Delta Sync, or any future backend) — the manager re-reads the post-sync local scope_root and compares it byte-for-byte against the peer's handshake scope_root. If they differ, pull_namespace_governance is called immediately. A store fault on the post-sync root read causes the check to be skipped with a warn and deferred to the next tick.
Ok(None) is excluded: when execute returns Ok(None) (i.e., SyncProtocol::None — entities were already in sync), the post-sync check does not run, avoiding a redundant store read and scope_root fold on every converged tick. The pre-sync GovDiverged path (described in the ProtocolDispatch section) handles that case.
All backends covered: Previously, only Hash Comparison and LevelWise could trigger a governance pull (via a gov_divergence_detected flag on their stats structs). Snapshot and Delta Sync never triggered such a pull, leaving peers that converged entity state via those backends stuck with a diverged governance plane. The manager-level check fires after all data backends.
gov_divergence_detected: bool has been removed from both HashComparisonStats and LevelWiseStats. HC and LevelWise still compute the ScopeVerdict for root_hash_verified and the scope_root_governance_divergence warning log, but no longer propagate a divergence flag to the selector.
// if post_sync_scope_root != peer_handshake_scope_root {
// let ops = self.pull_namespace_governance(context_id, peer).await;
// info!(marker = "gov_divergence_pull_triggered", ops_pulled = ops, ...);
// }
// gov_divergence_detected removed from both stats structs
pub struct HashComparisonStats {
// ... existing fields (no gov_divergence_detected) ...
}
pub struct LevelWiseStats {
// ... existing fields (no gov_divergence_detected) ...
}
Operators: The gov_divergence_pull_triggered info marker is now emitted by the sync manager after any data-backend sync session, not only HC/LevelWise. Any dashboards or alerts previously relying on gov_divergence_detected in protocol stats should be retired; watch the manager-level gov_divergence_pull_triggered and gov_divergence_pull_complete markers instead.
pull_namespace_governance — inherent method & op-count return
SyncManager::pull_namespace_governance — inherent method, returns usize
pull_namespace_governance is an inherent async method on SyncManager. It is best-effort: resolves the namespace ID for the given context via namespace_sync::resolve_namespace_id, calls sync_namespace_from_peer targeting the specific peer, and returns the number of governance ops delivered. All early-return and error paths return 0.
pull_namespace_governance is no longer a method on the ProtocolDispatch trait. The trait method and its implementation have been removed. The manager calls the inherent method directly from handle_dag_sync (post-sync path) and from the pre-sync entities-agree path.
sync_namespace_from_peer and pull_namespace_governance previously returned Result<()>. They now return Result<usize> — the number of governance ops delivered by the backfill response. The SyncDriverDispatch::sync_namespace_from_peer wrapper discards the count with let _ops = ....
Pre-sync governance divergence (entities-agree path): When select_protocol returns SyncProtocol::None (entity roots agree, no entity walk needed) and the peer advertised a non-None scope_root, the sync manager computes a scope_verdict. If the verdict is ScopeVerdict::GovDiverged, it calls pull_namespace_governance and returns early, bypassing the entity walk. The gov_divergence_pull_complete debug log now includes an ops_pulled field. The store read for scope_verdict is gated on peer_scope_root being Some: when a peer sends scope_root: None the datastore is never touched.
Post-sync path (all backends): After execute returns Ok(Some(_)), handle_dag_sync re-reads the local scope_root, compares it to the peer's handshake scope_root, and calls pull_namespace_governance when they differ. The returned op count is included in the gov_divergence_pull_triggered info log. The pull is self-limiting: once governance converges, roots match and no further pull is issued. The warn! log in run_initiator_impl changed from "awaiting governance sync to propagate the rotation" to "pulling governance from the peer to propagate the rotation".
// Returns the number of governance ops delivered (0 on error/no-op)
async fn SyncManager::pull_namespace_governance(
&self,
context_id: ContextId,
peer: PeerId,
) -> Result<usize>
// sync_namespace_from_peer now returns usize (op count)
// SyncDriverDispatch wrapper discards with: let _ops = ...;
// Pre-sync entities-agree path (SyncProtocol::None + peer scope_root Some)
// if scope_verdict == ScopeVerdict::GovDiverged {
// let ops = self.pull_namespace_governance(context_id, peer).await?;
// debug!(marker = "gov_divergence_pull_complete", ops_pulled = ops, ...);
// return; // bypass entity walk
// }
// Post-sync path in handle_dag_sync — fires on Ok(Some(_)) only
// if post_sync_scope_root != peer_handshake_scope_root {
// let ops = self.pull_namespace_governance(context_id, peer).await?;
// info!(marker = "gov_divergence_pull_triggered", ops_pulled = ops, ...);
// }
Operators: The gov_divergence_pull_triggered info marker (post-sync) and gov_divergence_pull_complete debug marker (pre-sync) both now include an ops_pulled field. ops_pulled == 0 indicates a best-effort failure or an already-converged peer. Monitor both markers together to verify governance convergence. The pull_namespace_governance trait method on ProtocolDispatch no longer exists; any custom ProtocolDispatch implementations must remove it.
scope_root Shadow Divergence
local_scope_root helper & scope_root_shadow_divergence marker
A new private helper local_scope_root looks up the group that owns a context and then calls ScopeProjections::group_scope_root_ephemeral to fold the current governance scope root. Returns None for non-group contexts or store faults. A TODO(perf, C1+) note marks the current full DAG walk per session as temporary — it should be replaced with a read from the maintained projection before C1 makes this path load-bearing.
DagHeadsResponse gains a scope_root: Option<Hash> field. The responder side in delta_request.rs and hash_comparison.rs calls local_scope_root and includes the result in every DagHeadsResponse sent. The deterministic sync simulator in hash_comparison_protocol.rs sends None because it has no governance projection.
query_peer_dag_state return type extended: The return type of query_peer_dag_state was extended from Option<(Hash, Vec<[u8; DIGEST_SIZE]>)> to Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)>. The scope_root field from DagHeadsResponse, previously matched as scope_root: _ (discarded), is now captured and threaded back to all callers. This allows the pre-sync entities-agree path to inspect the peer's governance scope root without an additional round-trip.
query_peer_current_root now returns (root_hash, Option<scope_root>) and is pub(crate) so the LevelWise module can call it directly, sharing a single implementation of the mid-session DagHeadsRequest/DagHeadsResponse round-trip. After the HC session completes, the convergence verdict (stats.root_hash_verified) on the re-query path now uses scope_root as the authoritative signal when both peers resolve one. Both scope roots are bound by destructuring (Some(local), Some(peer)); if either is None (cold projection or non-group context), the check falls back to bare entity-root comparison against the freshly re-queried peer root (not the stale handshake root). Transport faults on the re-query are logged at debug and fall back to the handshake root; an older peer returning Ok(None) falls back silently. This means two nodes can no longer declare convergence when they share identical entity data but have diverged governance state (e.g., after a hash-neutral membership rotation).
The LevelWise initiator follows the same re-query pattern: at end-of-session it calls query_peer_current_root to obtain the peer's current entity root plus scope_root, then applies the same scope_root-preferring convergence decision described above (falling back to the freshly re-queried entity root, not the stale handshake root).
The LevelWise responder loop now has an explicit InitPayload::DagHeadsRequest arm. When triggered, it re-reads the local entity root after all leaf/tombstone merges, folds the governance projection (local_scope_root), and replies with a DagHeadsResponse carrying both root_hash and scope_root. On a non-group or cold projection, scope_root is None, signalling the initiator to fall back to entity-root comparison. Previously this arm fell through to the catch-all _ that broke out of the loop.
The C0 observe-only block that emitted scope_root_shadow_divergence when entity roots agreed but scope roots differed — without acting on the divergence — has been removed. It is superseded by the authoritative scope_root convergence check and the new scope_root_governance_divergence warning.
A new WARN log with marker = "scope_root_governance_divergence" is emitted by both the Hash Comparison and LevelWise paths when: (1) both peers resolved a scope_root, (2) root_hash_verified is false, and (3) the bare entity roots agree. This distinguishes pure ACL/governance-plane divergence — where the entity tree-walk has nothing to reconcile — from a data-plane divergence. All other non-convergence cases emit the general warning with peer_hash taken from the re-queried root (replacing the old stale remote_hash).
// TODO(perf, C1+): replace full DAG walk with maintained-projection read
fn local_scope_root(
store: &Store,
context_id: ContextId,
) -> Option<Hash>
// Returns entity root + optional governance scope root
// pub(crate) — shared by Hash Comparison and LevelWise initiators
pub(crate) async fn query_peer_current_root(
&self,
peer_id: PeerId,
context_id: ContextId,
) -> Result<(Hash, Option<Hash>)>
// query_peer_dag_state — extended return type (used in select_protocol)
// Now returns Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)>
// Third element is scope_root from DagHeadsResponse (previously discarded)
// scope_root: None → peer is cold/non-group; skip scope_verdict check
// scope_root: Some(h) → used by pre-sync GovDiverged detection
// Authoritative convergence verdict (run_initiator_impl re-query path,
// used by both Hash Comparison and LevelWise):
// Both scope roots bound by destructuring (Some(local), Some(peer))
// if both Some → require scope_root equality
// if either None → fall back to freshly re-queried entity-root comparison
// root_hash_verified = match (local_scope_root, peer_scope_root) {
// (Some(l), Some(p)) => l == p,
// _ => local_root_hash == peer_current_root, // re-queried, not stale
// }
// LevelWise responder: explicit DagHeadsRequest arm in run_responder_loop
// Re-reads entity root post-merge, folds local_scope_root, replies with
// DagHeadsResponse { root_hash, scope_root }. scope_root=None for non-group
// or cold projection → initiator falls back to entity-root comparison.
// WARN emitted (HC and LevelWise) when both scope_roots are Some,
// root_hash_verified is false, but bare entity roots agree:
// marker=scope_root_governance_divergence
// entity_root=<8-byte hex>, local_scope=<8-byte hex>, peer_scope=<8-byte hex>
// Other mismatches: general non-convergence warn with peer_hash from re-query
Operators: The scope_root field is additive and optional — responders that cannot fold the scope send None, and the initiator falls back to bare entity-root comparison in that case, so the protocol is fully backward compatible. The C0 scope_root_shadow_divergence WARN marker is no longer emitted; update any log-aggregator rules or alerts that referenced it. Watch instead for the new scope_root_governance_divergence WARN marker on both Hash Comparison and LevelWise warn logs: a hit indicates real ACL/membership divergence on a context whose entity data is already in sync — these are awaiting governance sync propagation and require no entity-level reconciliation. Add a separate alert rule for this marker on LevelWise paths. Downstream code matching on MessagePayload::DagHeadsResponse must be updated to handle or ignore the scope_root field.
Transport
The sync transport layer wraps libp2p streams with framing, encryption, and authentication.
SyncTransport
Wraps a raw libp2p stream with length-delimited framing. Each frame is a Borsh-serialized StreamMessage. Handles backpressure via Tokio's AsyncWrite flow control.
framed: Framed<Stream, LengthCodec>,
encryption: EncryptionState,
}
EncryptionState
Manages per-stream encryption state. After the initial handshake, streams are encrypted with a shared key derived from the context's encryption domain. Supports both Encrypted and Plaintext modes (configurable per context).
Plaintext,
Encrypted {
cipher: ChaCha20Poly1305,
nonce_counter: u64,
},
}
Challenge Domain
Each sync stream authenticates with a challenge derived from the context's encryption domain. The initiating peer signs a challenge nonce, and the responder verifies membership before accepting the stream. This prevents unauthorized peers from syncing context data.
--mock-tee flag (dev/test only) — ⚠ NOT FOR PRODUCTION
merod run gains a --mock-tee flag (also settable via the MEROD_MOCK_TEE environment variable). When active, the fleet-join and attest handlers produce and accept mock attestation quotes instead of requiring real TDX hardware. This is intended exclusively for local development and CI environments where TDX hardware is unavailable.
The flag is refused at startup (bail!) if the node's TeeConfig has real attestation configured (TeeConfig::has_real_attestation returns true). A loud tracing::warn is emitted on startup whenever mock-tee is active. An additional warning fires if a Phala KMS provider is configured but real attestation is disabled, indicating a likely misconfiguration.
The flag is threaded through NodeConfig::mock_tee → AdminState::mock_tee (via AdminState::new) and is never persisted to the node config file.
// merod run --mock-tee | MEROD_MOCK_TEE=1 merod run
// Guard — refuses mock-tee when real KMS attestation is configured
impl TeeConfig {
pub fn has_real_attestation(&self) -> bool { /* ... */ }
}
// Propagation path (never persisted)
RunCommand::mock_tee: bool
→ NodeConfig::mock_tee: bool
→ AdminState::mock_tee: bool // via AdminState::new
Operators: Never set --mock-tee or MEROD_MOCK_TEE in production deployments. The startup bail! guard prevents accidental use when TeeConfig::has_real_attestation is true, but nodes without a real KMS configured will not be protected by that guard — rely on deployment policy and the emitted startup warning instead.