Node Crate
calimero-node + calimero-node-primitives
Purpose
NodeManager is an Actix actor that orchestrates the entire node. It receives network events via a dedicated channel bridge, manages blob caching, periodic heartbeats, delta handling, and routes streams to the sync manager.
It does not run libp2p directly — NetworkManager does that. NodeManager communicates with the network through NodeClient, which wraps a LazyRecipient<NodeMessage> and a NetworkClient handle. The crate also contains the SyncManager (a long-lived async task, not an Actix actor) and a GarbageCollector actor for storage tombstone cleanup.
2
crates
30+
source files
3
actors / tasks
4
sync protocols
Module Structure
File layout across the two crates. Each handler lives in its own focused file following the Single Responsibility Principle.
calimero-node
NodeManager actor definition, Actix lifecycle hooks
NodeManager struct with SRP decomposition: NodeClients, NodeManagers, NodeState
Startup hooks: setup_startup_subscriptions(), setup_maintenance_intervals(), setup_hash_heartbeat_interval()
NodeState: blob cache (multi-phase LRU eviction), delta stores (DashMap), sync sessions, pending invites
start() entry point, NodeConfig, wires Store / BlobManager / actors / server / GC
Handler<NodeMessage> dispatch — routes GetBlobBytes, specialized node invites
Handler<NetworkEvent> — gossip, stream, subscription, broadcast message routing
Namespace event handlers: handle_namespace_governance_delta(), handle_namespace_state_heartbeat(), backfill protocol
Specialized node: discovery, TEE attestation, join confirmation
handle_state_delta — DeltaStore buffering, context routing, sync session awareness, bounded KeyDelivery wait; gossip write-auth now delegated entirely to authorize_delta_at_edge_projected (projection-only, no live resolver called); refresh_projection_for_cut (DAG-walk/backfill for projection reads; calls ops_for_namespace — the stable indirection layer that currently delegates to collect_namespace_ops (governance-DAG walk) and will be flipped to load_scope_ops once late-decrypted membership is resolved; after ops_for_namespace returns the gov-DAG fold, check_op_store_completeness is called with those ops as a purely diagnostic gate — one op-store read on cold/refresh cuts (warm cuts skip via namespace_to_refresh returning None) — whose result is ignored by the caller and never affects whether or how apply_backfill proceeds; the observe-only shadow_compare_op_store cross-check and its companion scope_root_from_governance_dag have been removed — the comparison was flawed because the maintained projection holds live ingest_op-fed ops that may not be in the op-store); resolve_cut_membership (pub(crate) — single shared projection verdict renderer: performs a scoped projection refresh, then reads both member_at_cut and role_at_cut_for_group under a single read guard for epoch consistency, returns a CutMembership value; used by the gossip apply path, parent-fetch in request_missing_deltas, snapshot-replay in replay_buffered_delta, and DAG-catchup in SyncManager); projection_member_at_cut (pub(crate) — delegates to resolve_cut_membership and discards the role, ensuring the two paths cannot drift; used by SyncManager::peer_is_group_member); drain_governance_pending — re-applies, drops, or re-buffers buffered deltas based on three projection outcomes (Some(true)→apply, Some(false)→drop, None→re-buffer); the live authorize_delta_at_edge function is deleted — no acl_view_at calls remain on any delta authorization path
StateDeltaActor on dedicated Arbiter, bounded mailbox, drop-on-overflow with rebroadcast recovery
SyncSessionActor on dedicated Arbiter — both initiator and responder sync sessions; bounded mailbox + semaphore (max_concurrent), per-session timeout
Group-key delivery (pull-based): on a join request the responder validates the invitation and serves the ECDH-wrapped key (handle_namespace_join_request → build_group_key_delivery); joiners recover missing keys via recover_missing_group_keys. After a successful direct key delivery and the existing drain/reconcile steps, ScopeProjections::repersist_namespace_ops is called for the affected namespace_id so that any previously frozen Noop ops (encrypted MemberAdded entries that could not be decrypted at ingest time) are re-decrypted and written into the op-store. Because recover_missing_group_keys is the single runtime key-delivery chokepoint, all late-arriving keys trigger this op-store refresh automatically. RootOp::KeyDelivery is published only by admin-initiated adds (add-members / TEE), not on join. After sign_and_publish_without_apply successfully publishes the MemberJoinedAt op during await_namespace_ready (Stage 2 of namespace join), the op is written into the unified op-store via ScopeProjections::repersist_namespace_ops; previously this locally-authored publish wrote to the gov-DAG but skipped the op-store because the receive-handler dual-write does not cover locally-authored publishes. Note: the thin alias persist_namespace_head_ops has been deleted and all post-author mirror calls at create_group, delete_group, join_context, join_subgroup_inheritance, remove_group_members, reparent_group, create_group_in_namespace, and join_namespace have been removed; repersist_namespace_ops is the canonical method and is called only at the key-delivery and join-completion chokepoints described above.
sync/manager/mod.rs (peer_is_group_member)
SyncManager::peer_is_group_member — encapsulates inbound-peer membership authorization for all sync call sites. Checks MembershipRepository::is_member (live) first, then resolves current governance heads via ScopeProjections::namespace_current_heads and calls projection_member_at_cut (now pub(crate)) for the projection verdict. The projection's answer is authoritative; if it returns None (cold or partially folded ancestry) the method falls back to the live result via projected.unwrap_or(live) so no peer is spuriously rejected — the existing catch-up retry will re-resolve once governance advances. No divergence warning is emitted when projection and live disagree; the live read is retained solely as the None fallback. SyncManager::verify_inbound_member and the materialization-wait verification now both route through peer_is_group_member instead of calling MembershipRepository::is_member directly.
Blob transfer handler — serves blob data to requesting peers
Per-context delta buffering for out-of-order delta arrival
GarbageCollector actor — periodic tombstone cleanup across all contexts
calimero-node / sync
SyncManager, SyncConfig, protocol re-exports, metrics collector traits
Periodic sync loop, protocol selection, stream handling, peer tracking. pull_namespace_governance(context_id, peer) is now an inherent async method on SyncManager — resolves the namespace for the context via namespace_sync::resolve_namespace_id and calls sync_namespace_from_peer targeting the specific diverging peer, reusing the existing NamespaceBackfillRequest machinery edge-triggered on the verdict; best-effort (no-op if the context has no resolvable namespace). The ProtocolDispatch trait method pull_namespace_governance is now a one-line delegate to SyncManager::pull_namespace_governance. Pre-sync governance divergence check (P6.S3): when select_protocol returns SyncProtocol::None (entity roots agree, no entity walk needed) and the peer advertises a non-
None scope_root, the manager now computes a scope_verdict. If the verdict is ScopeVerdict::GovDiverged (local and peer scope roots differ despite identical entity roots), it calls pull_namespace_governance and returns early, bypassing the entity walk entirely. This fills the gap where a governance rotation or ACL change — invisible to the entity Merkle — would otherwise persist uncorrected because the post-sync P6.S2 hook only fires after HC/LevelWise actually runs. The datastore read is gated on peer_scope_root being Some; when a peer sends scope_root: None (cold projection or non-group context) the store is never touched and scope_verdict is never computed. After a successful pre-sync governance pull the manager emits a debug-level gov_divergence_pull_complete marker, pairing with the existing gov_divergence_pull_triggered info log. After a successful HashComparison or LevelWise session, if gov_divergence_detected is true, the ProtocolSelector likewise calls pull_namespace_governance and logs gov_divergence_pull_triggered; previously only a passive warning was emitted with no corrective action. The pull is self-limiting: once governance converges the verdict is no longer GovDiverged and no further pull is issued. Implementors of ProtocolDispatch must implement pull_namespace_governance (delegating to the inherent method).Merkle DFS traversal — cheapest sync protocol; run_initiator_impl calls query_peer_current_root (pub(crate) — also reused by the LevelWise initiator; returns
(root_hash, Option<scope_root>)); query_peer_dag_state return type extended from Option<(Hash, Vec<[u8; DIGEST_SIZE]>)> to Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)> — the scope_root field from DagHeadsResponse (previously discarded as scope_root: _) is now captured and threaded back to all callers; on the re-query convergence path, stats.root_hash_verified is now set via helpers::scope_verdict(...).converged() — the inline match (local_scope_root, peer_scope_root) block is replaced by a call to the shared scope_verdict helper; the scope_root_governance_divergence WARN is emitted by pattern-matching ScopeVerdict::GovDiverged(local, peer) directly, eliminating the prior if let (true, Some(...), Some(...)) guard; HashComparisonStats gains a gov_divergence_detected: bool field set to true when the verdict is ScopeVerdict::GovDiverged, exposing the signal to the ProtocolSelector; the WARN text changes from "awaiting governance sync to propagate the rotation" to "pulling governance from the peer to propagate the rotation"; the deterministic sync simulator sends scope_root: NoneBFS level-wise sync for wide trees. The initiator calls query_peer_current_root (pub(crate)) at end-of-session for a fresh
(root_hash, Option<scope_root>) rather than comparing against the stale handshake root; convergence is now decided via helpers::scope_verdict(...).converged() — the inline match (local_scope_root, peer_scope_root) block is replaced by the shared helper, mirroring HashComparison. Transport faults fall back to the handshake root; an older peer returning Ok(None) falls back silently. The responder loop has an explicit InitPayload::DagHeadsRequest arm: after all leaf/tombstone merges it re-reads the local entity root, folds the governance projection to obtain scope_root (None for non-group or cold projection), and replies with a DagHeadsResponse carrying both fields. LevelWiseStats gains a gov_divergence_detected: bool field set to true when the verdict is ScopeVerdict::GovDiverged, mirroring HashComparisonStats. When root_hash_verified is false, the scope_root_governance_divergence WARN is emitted by pattern-matching ScopeVerdict::GovDiverged(local, peer); the WARN text changes from "awaiting governance sync to propagate the rotation" to "pulling governance from the peer to propagate the rotation"; all other mismatches emit the general non-convergence warning using the freshly re-queried peer root.Full snapshot transfer — heaviest fallback protocol
Blob sharing protocol
Sync helper utilities; includes local_scope_root — looks up the group owning a context, calls ScopeProjections::group_scope_root_ephemeral, returns
None for non-group contexts or store faults (TODO(perf, C1+): replace full DAG walk with maintained projection reads). Also defines ScopeVerdict { Converged, GovDiverged([u8;32],[u8;32]), DataDiverged } and the free function scope_verdict(local_scope_root, peer_scope_root, local_entity_root, peer_entity_root) -> ScopeVerdict — the single convergence authority used by both HashComparison and LevelWise protocols, and by the pre-sync SyncProtocol::None divergence check in SyncManager. GovDiverged carries both resolved scope roots directly so callers can pattern-match and log them without re-destructuring Options. When either scope root is None (cold projection or non-group context), scope_verdict falls back to bare entity-root comparison (treating asymmetric None as None/None would cause false GovDiverged alarms on partially-warmed nodes). The pre-sync path gates the store read on peer_scope_root being Some, so the common already-in-sync path for cold/non-group peers incurs no I/O.Per-peer sync history tracking
SyncMetricsCollector trait, PhaseTimer, NoOpMetrics
Production Prometheus metric implementation
calimero-node-primitives
NodeClient façade — subscribe, broadcast, heartbeat, group ops, sync trigger
Application module root — get/has app, module declarations
All installation paths (path, URL, blob, bundle), uninstall, artifact extraction
Bundle manifest verification, signature validation, path safety checks
List applications, packages, versions; update compiled app
Blob add / get / delete / list / auth operations
Generic alias create / delete / lookup / resolve / list
NodeMessage enum — GetBlobBytes, RegisterPendingSpecializedNodeInvite, etc.
BroadcastMessage, sync wire types, protocol traits, state machine; MessagePayload::DagHeadsResponse now carries an additive scope_root: Option<Hash> field —
None means the responder could not resolve/fold the scope; initiators must skip the shadow comparison in that caseDeltaBuffer — bounded buffer for deltas received during snapshot sync
TopicManager — deduplication-aware gossipsub subscription management with RwLock
JoinBundle — data package for namespace join (group key envelope, governance snapshot, context IDs)
Key Types
Central structs in the node crate. NodeManager is the actor; its fields are split into three injected concerns: clients, managers, and mutable state.
NodeManager
pub struct NodeManager {
clients: NodeClients, // context + node façades
managers: NodeManagers, // blobstore + sync
state: NodeState, // mutable runtime state
}
clients: NodeClients, // context + node façades
managers: NodeManagers, // blobstore + sync
state: NodeState, // mutable runtime state
}
NodeClients
struct NodeClients {
context: ContextClient,
node: NodeClient,
}
context: ContextClient,
node: NodeClient,
}
NodeManagers
struct NodeManagers {
blobstore: BlobManager,
sync: SyncManager,
}
blobstore: BlobManager,
sync: SyncManager,
}
NodeState
struct NodeState {
blob_cache: Arc<DashMap<BlobId, CachedBlob>>,
delta_stores: Arc<DashMap<ContextId, DeltaStore>>,
pending_specialized_node_invites: PendingSpecializedNodeInvites,
accept_mock_tee: bool,
node_mode: NodeMode,
sync_sessions: Arc<DashMap<ContextId, SyncSession>>,
}
blob_cache: Arc<DashMap<BlobId, CachedBlob>>,
delta_stores: Arc<DashMap<ContextId, DeltaStore>>,
pending_specialized_node_invites: PendingSpecializedNodeInvites,
accept_mock_tee: bool,
node_mode: NodeMode,
sync_sessions: Arc<DashMap<ContextId, SyncSession>>,
}
NodeConfig
pub struct NodeConfig {
home: Utf8PathBuf,
identity: Keypair,
// namespace identity is auto-generated per root group in the datastore
network: NetworkConfig,
sync: SyncConfig,
datastore: StoreConfig,
blobstore: BlobStoreConfig,
context: ContextConfig,
server: ServerConfig,
gc_interval_secs: Option<u64>,
mode: NodeMode,
specialized_node: SpecializedNodeConfig,
mock_tee: bool, // dev/test only — refused if TeeConfig::has_real_attestation
}
home: Utf8PathBuf,
identity: Keypair,
// namespace identity is auto-generated per root group in the datastore
network: NetworkConfig,
sync: SyncConfig,
datastore: StoreConfig,
blobstore: BlobStoreConfig,
context: ContextConfig,
server: ServerConfig,
gc_interval_secs: Option<u64>,
mode: NodeMode,
specialized_node: SpecializedNodeConfig,
mock_tee: bool, // dev/test only — refused if TeeConfig::has_real_attestation
}
⚠ --mock-tee (dev/test only)
The --mock-tee flag (or MEROD_MOCK_TEE env var) on merod run makes fleet-join and attest handlers produce and accept mock TDX attestation quotes instead of requiring real hardware. When active, a loud tracing::warn is emitted at startup; a second warn fires if a Phala KMS provider is configured but real attestation is disabled (likely misconfiguration). The flag is refused (bail!) when TeeConfig::has_real_attestation() returns true — i.e. when real KMS attestation is configured. NodeConfig::mock_tee threads the value into AdminState (via AdminState::new); it is never persisted to the config file. Must not be used in production.