Node Crate

calimero-node + calimero-node-primitives

Purpose

NodeManager is an Actix actor that orchestrates the entire node. It receives network events via a dedicated channel bridge, manages blob caching, periodic heartbeats, delta handling, and routes streams to the sync manager.

It does not run libp2p directly — NetworkManager does that. NodeManager communicates with the network through NodeClient, which wraps a LazyRecipient<NodeMessage> and a NetworkClient handle. The crate also contains the SyncManager (a long-lived async task, not an Actix actor) and a GarbageCollector actor for storage tombstone cleanup.

crates

30+

source files

actors / tasks

sync protocols

Module Structure

File layout across the two crates. Each handler lives in its own focused file following the Single Responsibility Principle.

calimero-node

src/lib.rs

NodeManager actor definition, Actix lifecycle hooks

src/manager.rs

NodeManager struct with SRP decomposition: NodeClients, NodeManagers, NodeState

manager/startup.rs

Startup hooks: setup_startup_subscriptions(), setup_maintenance_intervals(), setup_hash_heartbeat_interval()

src/state.rs

NodeState: blob cache (multi-phase LRU eviction), delta stores (DashMap), sync sessions, pending invites

src/run.rs

start() entry point, NodeConfig, wires Store / BlobManager / actors / server / GC

src/handlers.rs

Handler<NodeMessage> dispatch — routes GetBlobBytes, specialized node invites

handlers/network_event/mod.rs

Handler<NetworkEvent> — gossip, stream, subscription, broadcast message routing

handlers/network_event/namespace.rs

Namespace event handlers: handle_namespace_governance_delta(), handle_namespace_state_heartbeat(), backfill protocol

handlers/network_event/specialized.rs

Specialized node: discovery, TEE attestation, join confirmation

handlers/state_delta/mod.rs

handle_state_delta — DeltaStore buffering, context routing, sync session awareness, bounded KeyDelivery wait; projection_member_at_cut_authoritative (grant-path projection override for MembershipReject and primary authority in drain_governance_pending); refresh_projection_for_cut (shared DAG-walk/backfill used by both deny and grant arms); projection_member_at_cut (deny-arm helper — now pub(crate) so the sync manager can call the same refreshing membership read); drain_governance_pending — re-applies, drops, or re-buffers buffered deltas based on three projection outcomes (Some(true)→apply, Some(false)→drop, None→re-buffer); acl_view_at used post-verdict only for unified_projection_divergence cross-check

src/state_delta_bridge.rs

StateDeltaActor on dedicated Arbiter, bounded mailbox, drop-on-overflow with rebroadcast recovery

src/sync_session_bridge.rs

SyncSessionActor on dedicated Arbiter — both initiator and responder sync sessions; bounded mailbox + semaphore (max_concurrent), per-session timeout

sync/manager/namespace_sync.rs

Group-key delivery (pull-based): on a join request the responder validates the invitation and serves the ECDH-wrapped key (handle_namespace_join_request → build_group_key_delivery); joiners recover missing keys via recover_missing_group_keys. RootOp::KeyDelivery is published only by admin-initiated adds (add-members / TEE), not on join.

sync/manager/mod.rs (peer_is_group_member)

SyncManager::peer_is_group_member — encapsulates inbound-peer membership authorization for all sync call sites. Checks MembershipRepository::is_member (live) first, then resolves current governance heads via ScopeProjections::namespace_current_heads and calls projection_member_at_cut (now pub(crate)) for the projection verdict. When the two disagree a unified_projection_divergence warning is emitted with plane = "membership-sync". The projection's answer is authoritative; if it returns None (cold or partially folded ancestry) the method falls back to the live result so no peer is spuriously rejected — the existing catch-up retry will re-resolve once governance advances. SyncManager::verify_inbound_member and the materialization-wait verification now both route through peer_is_group_member instead of calling MembershipRepository::is_member directly.

handlers/blob_protocol.rs

Blob transfer handler — serves blob data to requesting peers

src/delta_store.rs

Per-context delta buffering for out-of-order delta arrival

src/gc.rs

GarbageCollector actor — periodic tombstone cleanup across all contexts

calimero-node / sync

sync/mod.rs

SyncManager, SyncConfig, protocol re-exports, metrics collector traits

sync/manager/mod.rs

Periodic sync loop, protocol selection, stream handling, peer tracking

sync/hash_comparison_protocol.rs

Merkle DFS traversal — cheapest sync protocol

sync/level_sync.rs

BFS level-wise sync for wide trees

sync/snapshot.rs

Full snapshot transfer — heaviest fallback protocol

sync/blobs.rs

Blob sharing protocol

sync/helpers.rs

Sync helper utilities

sync/tracking.rs

Per-peer sync history tracking

sync/metrics.rs

SyncMetricsCollector trait, PhaseTimer, NoOpMetrics

sync/prometheus_metrics.rs

Production Prometheus metric implementation

calimero-node-primitives

primitives/src/client.rs

NodeClient façade — subscribe, broadcast, heartbeat, group ops, sync trigger

client/application.rs

Application module root — get/has app, module declarations

client/application/install.rs

All installation paths (path, URL, blob, bundle), uninstall, artifact extraction

client/application/bundle.rs

Bundle manifest verification, signature validation, path safety checks

client/application/query.rs

List applications, packages, versions; update compiled app

client/blob.rs

Blob add / get / delete / list / auth operations

client/alias.rs

Generic alias create / delete / lookup / resolve / list

primitives/src/messages.rs

NodeMessage enum — GetBlobBytes, RegisterPendingSpecializedNodeInvite, etc.

primitives/src/sync.rs

BroadcastMessage, sync wire types, protocol traits, state machine

primitives/src/delta_buffer.rs

DeltaBuffer — bounded buffer for deltas received during snapshot sync

primitives/src/topic_manager.rs

TopicManager — deduplication-aware gossipsub subscription management with RwLock

primitives/src/join_bundle.rs

JoinBundle — data package for namespace join (group key envelope, governance snapshot, context IDs)

Key Types

Central structs in the node crate. NodeManager is the actor; its fields are split into three injected concerns: clients, managers, and mutable state.

NodeManager

pub struct NodeManager {
    clients: NodeClients,     // context + node façades
    managers: NodeManagers,   // blobstore + sync
    state: NodeState,       // mutable runtime state
}

NodeClients

struct NodeClients {
context: ContextClient,
node: NodeClient,
}

NodeManagers

struct NodeManagers {
blobstore: BlobManager,
sync: SyncManager,
}

NodeState

struct NodeState {
    blob_cache: Arc<DashMap<BlobId, CachedBlob>>,
    delta_stores: Arc<DashMap<ContextId, DeltaStore>>,
    pending_specialized_node_invites: PendingSpecializedNodeInvites,
    accept_mock_tee: bool,
    node_mode: NodeMode,
    sync_sessions: Arc<DashMap<ContextId, SyncSession>>,
}

NodeConfig

pub struct NodeConfig {
    home: Utf8PathBuf,
    identity: Keypair,
    // namespace identity is auto-generated per root group in the datastore
    network: NetworkConfig,
    sync: SyncConfig,
    datastore: StoreConfig,
    blobstore: BlobStoreConfig,
    context: ContextConfig,
    server: ServerConfig,
    gc_interval_secs: Option<u64>,
    mode: NodeMode,
    specialized_node: SpecializedNodeConfig,
}

NodeClient primitives

Thin async façade in calimero-node-primitives. Used by Server and ContextManager to interact with node-level concerns. Wraps a Store, BlobManager, NetworkClient, and a LazyRecipient<NodeMessage>.

pub struct NodeClient {
    datastore: Store,
    blobstore: BlobManager,
    network_client: NetworkClient,
    node_manager: LazyRecipient<NodeMessage>,
    event_sender: broadcast::Sender<NodeEvent>,
    ctx_sync_tx: mpsc::Sender<(Option<ContextId>, Option<PeerId>)>,
    specialized_node_invite_topic: String,
}

Context & Group Subscriptions

subscribe(context_id)

Subscribe to a context's gossipsub topic via NetworkClient

unsubscribe(context_id)

Unsubscribe from a context topic

subscribe_group(group_id)

Subscribe to group/{hex} topic

unsubscribe_group(group_id)

Unsubscribe from group topic

broadcast(context, sender, artifact, …)

Publish a StateDelta on the context's gossipsub topic

publish_signed_group_op(group_id, op)

Publish a SignedGroupOpV1 on the group topic

publish_group_heartbeat(group_id, …)

Broadcast GroupStateHeartbeat for catch-up detection

get_peers_count(context)

Global peer count or per-topic mesh peers

Application & Blob Management

NodeClient exposes application lifecycle through four focused sub-modules (application/):

application.rs (module root) → install.rs (all install paths + uninstall) → bundle.rs (manifest verification + path safety) → query.rs (list/search)

Bundle signature verification: verify_and_extract_manifest (strict, production) vs extract_manifest_allow_unsigned (dev installs — verifies if present, allows unsigned).

Applications

install_application()
install_from_path()
install_from_url()
install_from_bundle_blob()
uninstall_application()
list_applications()
list_packages()

Blobs

add_blob()
get_blob() / get_blob_bytes()
delete_blob()
list_blobs()
find_blob_providers()
announce_blob_to_network()
create_blob_auth()

Aliases

create_alias()
delete_alias()
lookup_alias()
resolve_alias()
list_aliases()

Startup Flow

The start() function in run.rs bootstraps the entire node. Each component is initialized in dependency order, with LazyRecipients breaking circular initialization.

Event Handling

⚠ Migration note (F5 window): Monitor for unified_projection_divergence log events on the drain path. The governance_drain_outcome metric no longer emits the "lookup_error" label; consumers should add handling for the new "not_member" label. Additionally monitor for unified_projection_divergence events with plane = "membership-sync", which are emitted by the new peer_is_group_member helper when the projection and live membership disagree during inbound-sync authorization; a None projection result (cold or partially folded ancestry) silently falls back to the live result and will self-correct once governance advances.

NodeManager implements Handler<NetworkEvent> (via the bridge) and Handler<NodeMessage>. Network events arrive through a dedicated mpsc channel to avoid cross-arbiter message loss. BroadcastMessage::StateDelta dispatch is forwarded to a dedicated StateDeltaActor running on its own Arbiter so delta processing does not compete with sync, heartbeat, blob, or namespace handlers on the NodeManager mailbox. Inbound and outbound HashComparison/LevelWise sync sessions are likewise routed through a dedicated SyncSessionActor on its own Arbiter (responder via StreamOpened, initiator via SyncManager::start), so a slow session can’t starve the swarm-poll task and trigger gossipsub mesh starvation.

BroadcastMessage Dispatch

When a gossipsub message arrives, NodeManager deserializes it as a BroadcastMessage variant and routes accordingly:

StateDelta

Routed via StateDeltaSender::try_send to StateDeltaActor on a dedicated Arbiter. The actor's Handler<StateDeltaJob> calls handle_state_delta which checks sync session state — if a snapshot sync is active the delta is buffered, otherwise it's decrypted and applied. Mailbox bounded at STATE_DELTA_CHANNEL_CAPACITY (2048); on overflow the dispatch site logs a warn! and drops, relying on heartbeat-driven rebroadcast.

Projection-override on MembershipReject: When live membership returns DeltaAuthOutcome::MembershipReject, handle_state_delta no longer unconditionally drops the delta. It first calls projection_member_at_cut_authoritative (which refreshes the projection fold via refresh_projection_for_cut then reads member_at_cut_authoritative). If the projection returns Some(true), the delta falls through to the apply path and a unified_projection_divergence / membership-cut-grant warning is emitted; observe_peer_identity is also called so projection-only peers still feed sync peer selection. None or Some(false) still reject. The projection is therefore the authoritative membership decider on the primary gossip path; live's result is a cross-check, not the final word. Other MembershipReject sites (sync delta-request, parent-fetch replay) intentionally remain live-only.

Governance-drain projection authority: drain_governance_pending also uses member_at_cut_authoritative (projection) as the sole decision authority — not acl_view_at (live DAG walk). The three projection outcomes map directly to drain outcomes: Some(true) → re-apply the buffered delta; Some(false) → drop it (records a governance_drain_outcome metric); None → re-buffer (ancestry still being folded). After a decisive verdict, acl_view_at is called purely for a cross-check: on Some(true), a live result of Removed or NeverMember emits a unified_projection_divergence warn with plane = "membership-cut-grant"; on Some(false), a live result of Member emits the same marker with plane = "membership-cut". The live resolver is not called on None to avoid false positives. The old Err drop arm is eliminated because the projection never returns an error. The governance_drain_outcome drop metric label is now best-effort: "removed" / "never_member" if the live cross-check agrees, or the new fallback label "not_member" otherwise. The "lookup_error" label no longer exists. The re-buffer debug log no longer includes needed_count (the projection returns a boolean absence, not a structured missing-heads list).

Inbound-sync authorization (peer_is_group_member): projection_member_at_cut is now pub(crate) and is reused by SyncManager::peer_is_group_member for all inbound-sync peer authorization. That helper calls the live MembershipRepository::is_member first, then resolves governance heads and calls projection_member_at_cut with the same refreshing backfill logic used on the write path. The projection's answer is the authoritative decision; a None result (cold or partially folded) falls back to the live result so no peer is spuriously rejected. Divergences between the two are logged as unified_projection_divergence with plane = "membership-sync". Both the materialization-wait verification and verify_inbound_member inside SyncManager now route through peer_is_group_member instead of calling MembershipRepository::is_member directly.

HashHeartbeat

Periodic root hash announcement. Compared with local state; mismatches trigger sync via ctx_sync_tx channel.

SignedGroupOpV1

Forwarded to ContextManager's group operation handler. Signature verified, then applied to the local GroupStore DAG.

GroupStateHeartbeat

Group-level hash heartbeat for governance DAG catch-up detection.

GroupGovernanceDelta

Governance state delta. Routed to ContextManager for DAG ingestion.

GroupMutationNotification

Notification of group membership changes. Triggers re-evaluation of subscriptions and capabilities.

Stream Routing

When a peer opens a stream, stream_opened.rs inspects the protocol prefix to route:

/calimero/stream/…

Sync protocol streams — routed via SyncSessionSender::try_send to the dedicated SyncSessionActor Arbiter (issue #2316). The actor calls SyncManager::handle_opened_stream for hash comparison, level-wise sync, snapshot transfer, or key sharing. Bounded mailbox + a semaphore sized to sync_config.max_concurrent caps concurrent in-flight sessions; on overflow the inbound stream is dropped and peers retry.

/calimero/blob/…

Blob transfer protocol — handled by blob_protocol.rs. Serves blob data from BlobManager to requesting peers.

Subscription Lifecycle

NodeManager manages gossipsub subscriptions for both context topics (one per context) and group topics (group/{hex32}).

ctx

Context Topics

Subscribed when a context is joined/created via NodeClient. Carries StateDelta and HashHeartbeat messages. Unsubscribed on context leave.

grp

Group Topics

Subscribed for each group the node participates in. Carries SignedGroupOpV1, GroupStateHeartbeat, and GroupGovernanceDelta. On peer subscription events, the node auto-triggers group sync and alias re-broadcast.

Periodic Tasks

⏱

Heartbeats

SyncManager periodically broadcasts HashHeartbeat per context and GroupStateHeartbeat per group. Interval is configurable via SyncConfig.

⏱

Tombstone GC

GarbageCollector actor runs every gc_interval_secs (default 12 hours). Scans all contexts for expired CRDT tombstones past TOMBSTONE_RETENTION_NANOS and deletes them.

⏱

Blob Eviction

Cached blobs in NodeState::blob_cache are evicted based on last_accessed timestamps. The CachedBlob::touch() method refreshes access time on read.

⏱

Sync Loop

SyncManager runs a continuous async loop: listen for sync triggers from ctx_sync_rx, select the cheapest applicable protocol (hash comparison → level-wise → snapshot), execute, and track results per peer.

Dependencies

What the node crate depends on and what depends on it.

calimero-node depends on

calimero-context

ContextManager actor — started by node during bootstrap

calimero-context-primitives

ContextClient façade for sending messages to ContextManager

calimero-context-config

ContextGroupId, group configuration types

calimero-network

NetworkManager actor — started by node, runs libp2p swarm

calimero-network-primitives

NetworkClient, NetworkEvent, NetworkConfig, specialized invite types

calimero-node-primitives

NodeClient, NodeMessage, BroadcastMessage, sync wire types, DeltaBuffer

calimero-store

Store handle, typed keys, column families

calimero-store-rocksdb

RocksDB backend for Store

calimero-store-encryption

Optional AES encryption layer for the datastore

calimero-storage

CRDT storage engine, tombstone constants, EntityIndex

calimero-blobstore

BlobManager for binary object storage

calimero-server

Axum HTTP server, started by node as tokio task

calimero-primitives

Core types: ContextId, BlobId, PublicKey, events

calimero-crypto

SharedKey for encrypted sync handshakes

calimero-dag

DAG operations used by sync protocols

calimero-tee-attestation

TEE attestation verification for specialized node invites

calimero-utils-actix

LazyRecipient, ActorExt, Actix utilities

Depended on by

merod

Binary crate — calls calimero_node::start() from merod run command

calimero-server

Uses NodeClient (from primitives) for blob and application operations

calimero-context

Uses NodeClient (from primitives) for broadcast, subscribe, and sync triggers

Node

Context

Network

Storage

Server

Shared / Primitives

TEE