Server & API
calimero-server + calimero-server-primitives
Purpose
A single Axum HTTP server that binds multiple listeners and mounts all public-facing surfaces: Admin REST API, JSON-RPC 2.0, WebSocket, Server-Sent Events, Prometheus metrics, an optional embedded auth provider (mero-auth), and a static admin dashboard SPA. All business logic is delegated to ContextClient and NodeClient façades.
crates/serverHTTP Surfaces
The server exposes five distinct surface types on the same port. Each is mounted as a separate Axum router and merged into the main application.
Admin API Endpoints
The Admin REST API provides comprehensive management of contexts, groups, applications, blobs, and node identity. All protected routes require authentication (JWT or proxy).
Contexts
Groups
Namespaces
Applications
Blobs
Packages
Identity & Other
Authentication
The server supports two authentication modes. The choice is made at startup via configuration.
Proxy Mode default
No in-process JWT handling. The server assumes an external reverse proxy (e.g. nginx, Envoy, Cloudflare Access) terminates TLS and validates credentials before forwarding requests. The server trusts all incoming requests as authenticated.
Suitable for production deployments behind an identity-aware proxy with mTLS or OAuth2 termination.
Embedded Mode mero-auth
Full in-process JWT issuing and verification via the mero-auth crate. Mounts additional routes at /auth for login flows (challenge-nonce based). Issues JWT tokens on successful authentication.
Protected routes are guarded by an AuthGuardLayer (Tower middleware). Tokens can be passed via:
- Authorization: Bearer <token> header
- ?token=<token> query parameter
Execute Caller-Identity Enforcement
Both the JSON-RPC /jsonrpc surface and the WebSocket /ws surface share a common execute_request path in crates/server/src/execute.rs. Before any WASM method is invoked, the handler verifies that the authenticated caller's public key is a known member of the target context.
Before (vulnerable)
The executor identity was always auto-resolved to the node's own namespace identity for the context, regardless of which authenticated key made the request. Any valid token could invoke mutating methods on any context the node participated in.
After (fixed)
execute_request takes a caller: &PublicKey extracted from the verified AuthenticatedKey extension. It calls ctx_client.has_member(context_id, caller) first; a non-member receives FunctionCallError("Caller is not a member of this context") and the WASM runtime is never reached.
How the key is threaded per transport
- JSON-RPC: AuthGuardLayer injects AuthenticatedKey(pk) into Axum request extensions after JWT verification. The handle_request extractor pulls it out and passes it through the Request::handle trait to execute_request.
- WebSocket: Auth is connection-level (HTTP upgrade). ws_handler extracts AuthenticatedKey at upgrade time and stores the public key in ConnectionStateInner::caller. Each subsequent execute message reads the stored key and passes it to execute_request.
Executor identity vs. caller identity
After the membership check passes, the executor identity used to invoke the WASM method is still auto-resolved to the node's owned namespace identity for that context (the key whose private material the node holds). The caller's key gates access; the node's owned key drives execution. This distinction matters for applications that inspect the executor identity inside WASM.
Server → Actor Connection
The server holds an AdminState struct that is shared across all route handlers via Axum's state extraction. All business logic flows through the ContextClient and NodeClient façades — the server never accesses storage or actors directly.
store: Arc<dyn Database>, // read-only queries (list, get)
ctx_client: ContextClient, // execute, create, join, governance
node_client: NodeClient, // peers, blobs, boot
}
Request lifecycle example
HTTP Request Arrives
Axum receives a POST /admin-api/contexts request. Tower middleware extracts JWT claims (embedded mode) or passes through (proxy mode).
Route Handler
The route handler extracts AdminState and request body. Constructs the appropriate parameters for the operation.
Façade Call
Calls state.ctx_client.create_context(params).await. This sends a ContextMessage::CreateContext to the ContextManager actor via LazyRecipient.
Response
The actor processes the message, returns a Result. The route handler serializes it to JSON and sends the HTTP response.
Route Catalog
Admin API Routes
Auth Modes
Embedded Auth
Set network.server.embedded_auth = true in config. The server runs the mero-auth JWT service internally. Admin Dashboard SPA is served from /admin/. Frontend assets are baked in at build time via CALIMERO_AUTH_FRONTEND_DIR.
Proxy Auth
Run mero-auth as a separate service behind a reverse proxy (nginx/Traefik). The proxy validates tokens via /auth/validate before forwarding to the node. Node remains auth-unaware.
Subgroup Creation Behaviour (POST /namespaces/:id/groups)
As of the latest change, the create_group_in_namespace handler follows a strict key-first ordering:
- The subgroup signing key is written to SigningKeysRepository.
- The group encryption key is written to GroupKeyring.
- Only after both writes succeed is sign_apply_and_publish_namespace_op called to apply the GroupCreated op.
Key-provisioning failures are now hard errors. If either key write fails, the request is aborted immediately and the caller receives a 500-style error response. Previously, failures were warn-logged and the request continued, leaving the group in an inconsistent state.
Namespace root TEE members are now fully admitted into Restricted subgroups at creation time, because the signing key is available before the op is applied.
Visibility field: The request body (CreateGroupInNamespaceBody) accepts an optional visibility string. "open" sets restricted = false on the GroupCreated op; "restricted" (or absent/unrecognized) sets restricted = true. This allows callers to create born-Open subgroups in a single REST call without a follow-up update.
Op-store mirroring: After sign_apply_and_publish_namespace_op successfully authors the GroupCreated op, ScopeProjections::persist_namespace_head_ops is called to persist the op into the unified op-store, ensuring the new group's creation is durably recorded and visible to op-store consumers.
Op-Store Mirroring for Group Reparenting (POST /namespaces/:id/nest & unnest)
After sign_apply_and_publish_namespace_op successfully authors a GroupReparented op in the reparent_group admin handler, ScopeProjections::persist_namespace_head_ops is now called to persist the op into the unified op-store. This mirrors the same behaviour introduced for GroupCreated in create_group_in_namespace, ensuring that all structural group hierarchy changes are durably recorded and visible to op-store consumers immediately after the operation succeeds.
Wire-Fixture Round-Trip Canary Tests
Golden JSON files under crates/server/primitives/fixtures/wire/ capture the canonical on-wire shape of six HTTP request/response DTOs:
- CreateContextRequest / CreateContextResponseData
- ReparentGroupApiRequest / ReparentGroupApiResponse
- ExecutionRequest / ExecutionResponse (JSON-RPC)
A Rust integration test (wire_fixtures.rs) deserializes each fixture into its Rust type and re-serializes it. Any field rename, removal, or type change causes a diff and fails the test. Run UPDATE_FIXTURES=1 cargo test -p calimero-server-primitives --test wire_fixtures to regenerate the golden files; do this whenever an HTTP DTO field changes, then update the matching SDK type accordingly.
Route-Manifest Guard (endpoints.json)
A committed endpoints.json file enumerates all 72 admin HTTP routes. A Rust integration test (route_manifest.rs) parses .route("...", ...) calls from admin/service.rs and asserts they match the manifest exactly — adding a new route without updating the manifest causes a test failure.
Run UPDATE_MANIFEST=1 cargo test -p calimero-server --test route_manifest to regenerate endpoints.json after adding or removing routes.
Endpoint Coverage Enforcement (check-endpoint-coverage.sh)
The check-endpoint-coverage.sh script diffs endpoints.json manifest routes against concrete paths recorded by the mero-js e2e run (:param and *glob pattern matching). Known gaps are tracked in coverage-baseline.json:
- Routes present in the baseline are accepted as known gaps and reported as a burndown backlog.
- Any newly uncovered route not in the baseline fails the build.
- The baseline currently contains only /admin-api/ (the root catch-all), meaning 71 of 72 admin routes must be exercised by e2e tests.
New endpoints without e2e coverage must either be added to coverage-baseline.json with a documented reason, or have a mero-js e2e test added.
Fleet-Join Cold-Start: NoPeersSubscribedToTopic is Non-Fatal
When a node joins a fleet namespace and no peers are yet subscribed to the gossipsub topic (empty mesh at cold start), publish_on_namespace_now returns a NoPeersSubscribedToTopic error. This is now treated as a non-fatal expected condition:
- An info-level log is emitted instead of an error.
- The handler falls through to the re-announce polling loop, which retries until peers are available.
- The caller does not receive an HTTP 500, and the node does not unsubscribe from the topic.
- All other publish errors remain fatal and continue to abort the request.
This is the normal outcome on the very first fleet-join when the gossipsub mesh has not yet formed. The re-announce loop handles propagation once peers connect.
--mock-tee Flag (Dev/Test Only)
merod run gains a --mock-tee flag (also configurable via the MEROD_MOCK_TEE environment variable). When active:
- The fleet-join and attest handlers produce and accept mock attestation quotes instead of requiring real TDX hardware.
- A loud tracing::warn is emitted on startup to signal that mock TEE mode is active.
- An additional warn fires if a Phala KMS provider is configured alongside mock TEE (likely misconfiguration — real attestation is disabled).
- The flag is threaded through NodeConfig → AdminState as AdminState::mock_tee; it is never persisted to the on-disk config.
⚠ Security Warning
--mock-tee is intended for development and testing only. It is refused at startup (bail!) if the node's TeeConfig has real attestation configured (TeeConfig::has_real_attestation returns true). It must never be used in production environments.