Networking & sealed channels

A node speaks HTTP and nothing else: one listener (--listen) serves clients, peers and the operator. There is no gossip, no peer discovery and no peer-to-peer stream protocol. Nodes learn each other’s URLs from the system group’s nodes table, seeded from --peers at bootstrap and extended through POST /ops/nodes.

Why seal on top of HTTP

Whatever TLS sits in front of a node ends outside the TD: at a load balancer, a reverse proxy, wherever the operator points it. Everything that crosses that hop — bearer tokens, call arguments, results — would be readable there, however genuine the node’s quote. So clients do not trust the transport. They encrypt each request to a key the node’s attestation binds, and only the attested process can open it.

The sealed channel

calimero-sealed implements it (wire format v2, byte-compatible with mero-js’s src/sealed/).

• Transport key. At start the node makes an X25519 key. It lives only in the process’s memory and is never written anywhere, so a restart replaces it. GET /attestation returns its public half and binds it into the quote’s report data.
• Handshake. POST /sealed/v2/handshake runs Noise NK_25519_AESGCM_SHA256 with prologue calimero/sealed-http/v2: the client knows the node’s static key in advance (from the quote), and both sides’ ephemeral keys make the session keys. The reply carries a session id and its lifetime.
• Exchange. The client wraps a whole HTTP request — method, path, headers (the bearer token among them), body — seals it under the session’s request key and posts it to /sealed/v2. Each request carries an id its session accepts once. The node opens it, routes the inner request through the same router a direct request takes, and streams the response back as sealed frames (a head, body frames, an end frame), so server-sent events are sealed too and a truncated response is detected.

Property	Value
Session lifetime	1 hour, however busy (the forward-secrecy window)
Idle session kept	10 minutes
Sessions held	16 384; past that, the least useful session is dropped and its client opens another
Replay window	4096 request ids per session; ids below the window are refused too
Largest sealed request	64 MiB

A proxy in the middle sees opaque POSTs. It cannot read a request or a response, forge or reorder a frame, or replay a request. When a node restarts, its old transport key is gone; the client gets stale_transport_key, attests the new key and opens a new session. WebSocket upgrades cannot cross the envelope and are refused inside it.

What is served in the clear

Only GET /attestation and GET /health (node id, system leader, release) answer outside the envelope; with sealed_required (every non---dev node) any other unsealed request is refused. Two outer layers sit in front of the envelope:

• App frontends. With --apps-domain, a request whose host is a package’s label under that domain is answered from the package’s signed frontend and never reaches a node route (Packages & frontends).
• CORS. The configured --origins and every package frontend origin may call the node from a browser. CORS is outermost so a preflight is answered before the envelope sees it; what travels sealed needs no header of its own.

The routes

Prefix	Who	What
/attestation, /health	Anyone, unsealed	The node’s quote; liveness
/auth/*	People and API keys	Passkey sign-up and sign-in, devices, sessions, API keys, aliases, approvals, usage
/packages*, /bundles	Installers	Install, list and remove releases; raw WASM in development
/apps*	Callers the app admits	Create, list, call, view, events, upgrade, access, metadata, blobs, presence, storage, ABI, attestation
/orgs*, /invites/{token}/redeem	Members	Organizations, teams, members, capabilities, invites, approvals, upgrade runs
/internal/*	Peers only	Raft RPCs, system writes and read index, forwarded calls and views, blobs, presence, events, storage stats
/ops/*	The operator’s token	Status, metrics, node registration, replica moves, backups, quotas, and the decisions governance gives the operator

Errors are JSON { "error": CODE, "message": …, "details"? } with an HTTP status.

Between nodes

A node talks to a peer exactly as a client does: it fetches the peer’s attestation, judges the quote against its own measurement policy, and opens a sealed session to the attested key (HttpAttestor). In TDX mode the policy is the signed mero-tee release the peer names, with the profiles in --peer-profiles (default: this node’s own MERO_TEE_PROFILE) and the floor --peer-min-release; a TDX node refuses to start without them. A node trusts a debug-profile peer only if it runs a debug image itself. The release each peer’s attestation named is remembered, which is what the release gate reads.

A node’s session to a peer is identified: handshake version 3, Noise_IK_25519_AESGCM_SHA256 instead of the client’s NK, where the caller also proves its own static key, which is its attested transport key. The peer hands that key to every request of the session, and the request names the calling node (x-calimero-node). The receiver believes the name only once it has attested that node’s registered address itself, with a fresh nonce, and found the same key there; the same quote gives it the caller’s release (Node::identify_peer, cached by key). Every internal request also carries x-cluster-auth, the cluster secret HMAC-SHA256(K_c, "calimero.cluster.v1|" ‖ cluster) (Encryption), compared in constant time, but the secret alone authorizes no data access: every release can compute it.

Each internal route judges the identified caller, the inbound twin of the release gate on what a node sends:

Route	The caller must be
internal/view, call, events, storage, blob/held, app-ready, presence	On a release the app’s owner approved (release approvals)
internal/blob/get	Any node of the cluster for code (a bundle, a release .mpk); for any other blob, on a release approved for an app hosted here that marks it
internal/blob/delete	Any node of the cluster, and no app hosted here may still mark the blob
internal/raft	The node the RPC names as its sender (leader or candidate); for an app group also on an approved release and a replica or member of the group. The message’s target must be this node (421 NOT_ADDRESSEE)
internal/system, system/index, blob, challenge/take	A registered node that is not removed (or a node still in the system group’s membership while its removal commits); what it proposes is judged where it applies (system group)

An anonymous peer session (Noise NK) is refused on every internal route.

Raft itself is an RPC: POST /internal/raft { group, target, rpc } carries an AppendEntries, Vote or InstallSnapshot request as JSON and returns the response. target is the node id the sender meant. The route checks both ends of the message: the sender must be the authenticated caller (above), and the addressee must be this node. A node reached under another id (a reused port, a node moved or replaced) refuses the message with 421 NOT_ADDRESSEE instead of taking it as its own. A message without target comes from a peer on an earlier release; it is still accepted, counted and warned about, and the release after refuses it. Forwarded calls go to internal/call, views to internal/view, system writes to internal/system. A forward to an app’s leader is bounded at 5 seconds; a system write keeps the 30-second HTTP timeout.