Skip to content

The operator & the Ops API

The operator runs the machines: it starts nodes, gives them a network and disks, and keeps the cluster healthy. It is not trusted with app data. The Ops API (/ops/*) gives it what operating requires and nothing that opens an app. mdma’s OpsClient and meroctl --ops-token TOKEN ops … use it.

The operator holds a bearer token. A node is given either the token (--ops-token) or only its SHA-256 (--ops-token-sha256), so a machine’s metadata or configuration need not hold anything usable. The token is compared in constant time, inside the sealed channel like every other request. A node given neither has the Ops API disabled. The operator’s actions that the system log records carry the actor operator, which no account id can be.

The token reaches the node, not the log: every node of the cluster can append to the system log, so a decision the governance gives the operator must carry the operator’s own proof. That is an operator key, Ed25519, whose public half the nodes are given (--operator-key <hex>, MEROD_OPERATOR_KEY) and the first leader records at genesis. Every operator decision (the second table below) carries the key’s signature, checked where it applies on every node (system group).

The signed bytes are, with 0 a zero byte and every field UTF-8:

"calimero.operator.v1" 0 cluster 0 subject 0 method 0 args 0 nonce 0

subject is cluster (release and outbound approvals, deadlines, nodes, quotas, pools, prices, deployers, sign-up, a holder’s recovery), the app id (an app’s upgrade, replica move or restore), org:<id> (an organization’s run) or operator (its own backup key); method names the decision, among them __approve_release, __withdraw_release, __approve_outbound, __upgrade, __upgrade_org, __upgrade_org_retry, __release_deadline, __register_node ({ id, addr, zone }), __remove_node ({ id }), __set_quotas ({ quotas }, every limit after the change), __set_org_quotas ({ payer, quotas }), __set_pools, __set_price_table, __set_replicas ({ replicas }, the target set sorted; one signature covers both writes of the move), __restore_app ({ backup, sha256 }), __set_backup_key ({ publicKey }) and __recover_holder ({ holder, slice, apps }); args is the exact JSON text the node states; nonce is base64url, used once. A decision sent without a proof answers 428 OPERATOR_SIGNATURE_REQUIRED with details { subject, method, args, nonce, cluster, message }, message being the bytes above in base64: the caller signs message and sends the same request again with "operatorProof": { "nonce": <details.nonce>, "signature": <hex> }. A signature that does not verify is 403 BAD_OPERATOR_SIGNATURE; one used before is refused by the log. With no key recorded, operator decisions are refused (403).

Route What
GET /ops/status This node, the system group’s Raft status, the Raft status of the app groups hosted here (by app id), the node list, the governance, and counts of apps and accounts
GET /ops/metrics Prometheus text: nodes, apps, accounts, organizations, packages, bundle bytes; groups hosted and led here; open streams; system applied index, leadership, voters, learners; each hosted group’s applied index; the build release
GET /ops/placement Each app’s group and replica node ids; each node’s zone, attested release and app count
GET /ops/apps What each app runs and which releases may hold it (below)
GET /ops/orgs Each organization’s id and the releases approved to hold its apps
POST /ops/nodes Register a node, signed (428 first); the system leader then adds it as a learner (Replication). The nodes a cluster starts with are registered at genesis
DELETE /ops/nodes/{id} Remove a node, signed (428 first; the proof in a JSON body)
POST /ops/apps/{app}/replicas Move an app to a new replica set, onto approved releases only, signed (428 first) (Replica moves)
GET /ops/cluster The cluster’s identity key, which a job node is started with (merod run --job-node --cluster-identity)
GET /ops/job-nodes/{id}/attestation Whether the cluster accepts a job node: admitted with its release, refused (final: a 400 or 404 from the node, a 2xx that does not decode, a failed verification, or a 400 or 404 from the collateral source on the PCK certificate or TCB info request) or unreachable (transient: anything else, including any other status from the node and any other collateral failure) with the reason; 404 for an id that is no job node, 503 when this node’s directory is stale (Compute actions)
GET /ops/nodes/{id}/receipt-key The receipt key this node holds for a peer, to check an account proof from any node
GET / POST /ops/quotas Read or set the quotas (per account, per organization, and the developer’s fuel per day); { org, … } sets one organization’s override. Setting is signed (428 first; operatorProof beside the limits)
GET / POST /ops/pools, GET / POST /ops/prices Compute pools and the price table, setting signed (Compute actions)
GET /ops/served-origins Custom domains packages registered as served origins ({ origin, host, label, addedAt }, no package names), for certificates and routing
POST /ops/sign-up, /ops/sign-up/invites[/{key}] The sign-up policy and cluster sign-up invites, signed, until the cluster’s owner is named (Sign-up)
POST /ops/backup Write sealed snapshots of the app groups this node leads (below)
POST /ops/apps/{app}/restore Restore an app from a backup or a recovery file, signed (below)
GET / POST /ops/backup-key, GET /ops/recovery-files, POST /ops/recover The recovery disk: the operator’s backup key, the recovery files’ headers, founding the cluster again from them (below)
POST /ops/releases/{version}/deadline { after, operatorProof }, DELETE … Set (RFC 3339) or lift (after: null, or DELETE with the proof in its body) a release’s end-of-new-work deadline, signed with the operator key (428 first): after it, a party that has not approved the release creates, installs and upgrades nothing until it does; running apps go on (Release deadlines). Public at GET /releases
POST /ops/deployments/granted { operatorProof } Move the cluster’s deployments from open to granted, once, never back (Deployers)
POST /ops/deployers { deployer, until?, operatorProof }, DELETE /ops/deployers/{key} (or POST /ops/deployers/{key}/revoke, for a client whose DELETE carries no body) Grant (until: RFC 3339) or revoke a deployer ({ "account": id } or { "org": id }; the key is account:<id> or org:<id>), signed with the operator key, while the operator grants deployers
POST /ops/cluster-owner { owner, operatorProof } Where governance says deployers: owner: name the cluster’s owner, once; it grants deployers from then on
GET /deployers The deployments rule, the owner and every grant

The operator does not act inside a deployment: creating one, offering, withdrawing and deleting take a manager’s passkey (Deployments).

And, only where the deployment’s governance gives the operator that decision:

Route Decision
POST /ops/releases { version, approved } Release approvals, for every app
POST /ops/apps/{app}/upgrade, POST /ops/orgs/{org}/upgrade App upgrades, to a release it installed or one the app’s owner may see
POST /ops/packages, /ops/packages/install, GET /ops/packages, DELETE /ops/packages/{p}[/{v}] Installing the releases it upgrades to (it lists and removes only what it installed)
POST /ops/outbound { package, prefixes, replace } Outbound approvals, for every app

Elsewhere, the operator configures the node: the outbound policy (which hosts any app may reach, and the secrets requests may carry), the registry releases are installed from, the domain app frontends are served under, and the measurement policy for peers.

Allowing a host does not vouch for it: an app’s destinations are every host it and the apps it created declare, listed to the people who use it, and the operator has no list of trusted external endpoints, a TEE included (Where an update may send data).

GET /ops/apps is how the operator follows a rollout: per app, its id, group and scope ({ kind: "org", org }, with personal: true for a personal organization, never the account), the bundle digest it runs, its publisher (the package’s pinned signer, with the package), who holds app upgrades, its last upgrade (from and to digests, with each end’s release where the operator may see it, when, whether it succeeded, and the actor kind: owner, member or operator), the mero-tee releases approved to hold its data, and each replica’s node, release, and whether that release is approved for the app. GET /ops/orgs gives each organization’s approved releases. Both are paged (?after=ID&limit=N, at most MAX_REPORT_PAGE, 100, per page; next continues).

What it leaves out is deliberate:

  • Package names and versions only for releases the operator installed. Which package an app runs can say what an organization does (a package uploaded for one customer is named after it), so the report names a release only where sees_release lets the operator see it anyway: the releases it installed itself (/ops/packages, its catalog). For any other app, package is null and only the bundle’s digest (hex(SHA-256) of its WASM) appears: enough to group apps that run the same code and to see an upgrade happen, nothing that names it.
  • The actor’s kind, not the actor. An upgrade’s account, its error text (which the app’s code can write) and its upgrade run are left out.
  • Approved releases and governance are not secrets: every replica attests the governance, and the approved releases decide which nodes the operator can place an app on.

Which nodes hold an app’s data is decided twice: by the node that creates the app, once, and afterwards only by the operator’s replica moves. Neither the log nor a node moves replicas on its own.

POST /apps picks --replicas nodes (3 by default) before it proposes CreateApp, which records them (Node::place, placement::choose):

registered nodesn1 zone a · 4 appsn2 zone a · 1 appn3 zone b · 3 appsn4 zone b · 2 appsn5 no zone · 0n6 job noden7 zone c · 0 (release not approved)POST /apps, --replicas 31 · replica nodesn1 n2 n3 n4 n5 n7n6 outjob nodes hold no app(replica_nodes)2 · may hold itn1 n2 n3 n4 n5n7 outattested release approvedand not past a deadline3 · runtime floorn1 n2 n3 n4 n5none belowruntime ≥ the release'sminRuntimeVersion4 · chooseby (load, id):n5 n2 n4 n3 n1n5 · zone –n2 · zone an4 · zone b[2, 4, 5]a zone each first,then by load
  1. Replica nodes only. Of the registered nodes, only replica nodes (SystemState::replica_nodes). A job node never holds an app: it never holds the cluster key.
  2. Nodes that may hold it. The set of mero-tee releases approved for the app is that of the release the app will run: its publisher’s for a bound package, else the app’s organization’s, or the operator’s where governance gives it release approvals (Release approvals). A node stays only if its attested release is in that set and is not retired by a deadline that has passed. With no set recorded and no deadline passed, every replica node stays and none is attested for this.
  3. The runtime floor. A node whose runtime is known to be older than the release’s minRuntimeVersion is left out; a node that does not say (an older node, one that does not answer) is kept (The runtime a release needs).
  4. Zones first, then load. The nodes left are sorted by load (the number of apps they hold a replica of, then the lower id). The first pass takes the first node of each zone not yet covered; the second fills the remaining places by load alone. So the replicas span min(replicas, zones) zones, and within that the least-loaded nodes. A node’s zone is the one POST /ops/nodes registered; nodes without one (every node from --peers until it is registered with one) count as one unknown zone together, so a cluster without zones is placed by load alone.

Fewer nodes left than --replicas gives the app all of them. None left answers 503 (“no node runs a release this app’s owner approved”), or 409 RUNTIME_TOO_OLD naming the nodes when the runtime floor emptied the list.

Worked example (the figure). Seven nodes are registered: n1 to n4 in zones a and b with 4, 1, 3 and 2 apps, n5 without a zone and no apps, n6 a job node, n7 in zone c on a release the app’s publisher has not approved. n6 and n7 drop out. By load the rest are n5, n2, n4, n3, n1. The first pass takes n5 (the unknown zone), n2 (zone a) and n4 (zone b), which fills the three places: the app goes to [2, 4, 5], sorted by id. Had n5 not been there, the second pass would have filled the third place with n3, the least-loaded node left, and the app would span two zones.

After creation only the operator moves an app, with POST /ops/apps/{app}/replicas { replicas } (meroctl ops move <app> --replicas 2,4,5), signed with its key over <app> / __set_replicas / { replicas } (one signature for both writes of the move). The node refuses, before anything is written:

  • a node named twice (DUPLICATE_REPLICA), fewer than 3 nodes, or than the cluster’s replica nodes where it has fewer (TOO_FEW), an unregistered node (UNKNOWN_NODE) or a job node (JOB_NODE);
  • a node whose attested release is not approved for the app (403): a move is how data would reach a new image, so it obeys the same approvals as a new app;
  • a joining node whose runtime is older than the app’s release needs (409 RUNTIME_TOO_OLD; nodes the app is on already are not judged, so an app can always be moved off one);
  • on a node that does not lead the app’s group, 421 NOT_LEADER with details.leaderUrl; a second move of the same app while one runs, 409 MOVE_IN_PROGRESS. A move to the set the app has writes nothing (unchanged: true).

The leader hands the joining nodes the app’s bundle and the blobs its state marks, records the union of old and new nodes (SetReplicas), adds the new ones as learners, makes the target set the voters once they caught up, and records the target. The nodes that left stop the group and delete its data (Replica moves).

GET /ops/placement (meroctl ops placement) is what a rebalancer (mdma’s) reads before it moves anything:

{
"fresh": true,
"apps": [{ "app": "…", "group": 7, "replicas": [2, 4, 5] }],
"nodes": [{ "id": 2, "zone": "a", "role": "replica", "release": "2.3.131", "apps": 2 },
{ "id": 6, "zone": "", "role": "job", "release": "2.3.131", "apps": 0 }],
"clusterVersion": "…"
}

It is read after a read index (fresh: this node caught up with the system leader), and each node’s release is its attested one, asked for at most PLACEMENT_RELEASE_WAIT (null when it did not answer). Ids and counts only: no app names, organizations or state. A job node is listed with its role and no apps.

There is no Ops route that runs a view, reads state, streams events, lists an account’s apps, downloads a blob or a bundle, or returns a credential. Status, metrics and the reports above give counts, ids, digests, releases and Raft positions; app names, organization names, people’s names and app state never appear. The operator cannot sign in as anyone: sessions are minted only after a passkey ceremony, each signed by the attested node that ran it. Nor can the ops token alone make a decision the log accepts: that takes the operator key.

The operator does control availability. It can stop, freeze or restart nodes, withhold the network, or refuse to register nodes, and it sets release deadlines, which stop new work on old images but move no data and block no approval. An app stays available while a majority of its replicas run.

With --backup-dir, every node writes, every 60 seconds (and on POST /ops/backup), a copy of the current snapshot file of each app group it leads, first asking for a fresh snapshot if the last one is older than what it has applied. A backup is the log’s own snapshot file, sealed frame by frame under the cluster key K_c, stored as {dir}/{app}/{index:020}.{group}.{snapshot}.snap, with a manifest beside it (….snap.manifest) that names the app, group, index, the file’s SHA-256, the backup format and the merod and mero-tee release that wrote it, sealed under HMAC-SHA256(K_c, "calimero.backup.manifest.v1") and bound to the app and the file name. Only a node holding the cluster’s K_c can open either (backup::find, backup::latest). The directory stands in for object storage.

POST /ops/apps/{app}/restore puts an app back to a backup the receiving node holds, as the operator’s signed decision (<app> / __restore_app / { backup, sha256 }). The log records RestoreApp: the app moves to a new group whose first replica is that node, the seed. The seed forms the group alone, writes the backup’s rows into it (each opened and sealed again for the new group), appends componentRestored, snapshots and purges its log through that entry, then adds the other replicas, which can only catch up from that snapshot, and makes them voters; FinishRestore ends it. The new group’s Raft terms start again at 1, so componentRestored moves the base of run ids past every term the restored state carries (D52). The old group is dropped everywhere.

The system group’s leader backs up the system group the same way ({dir}/system/), with the code it names and K_c wrapped for its release; merod run --bootstrap --restore-from DIR founds the cluster again from them, and every app then waits for its own restore. A node of a cluster that lost its system majority backs up every group it holds with POST /ops/backup?all=true.

What the operator decides is which backup the app goes back to: a restore is a rollback of the app’s data, never a way to read it. A backup restores only on the cluster that sealed it (the same K_c), for the app its manifest names, on the release that wrote it or a later one (Restore an app).

Backups open only with K_c, which only a live mero-kms of a release they hold a copy for can unwrap. The recovery disk (merod --recovery-dir DIR) keeps a second set that outlives every mero-kms: every 5 minutes each node writes, for every app it holds (leader or not), the app’s state, the code it runs and its holder’s slice of the system state, each sealed to the backup holder’s X25519 public key (HPKE, RFC 9180) and signed by the node’s attested receipt key (Recovery files). The holder is whoever upgrades the app: the operator where it holds app upgrades, else the app’s organization (Backup key). No node holds a private half.

Route What
GET / POST /ops/backup-key { publicKey } The operator’s backup key (64 hex), where it holds app upgrades; setting is signed (operator / __set_backup_key / { publicKey })
GET /ops/recovery-files?holder= The headers and signatures of a holder’s recovery files on this node, never a key or a body, for the holder to check before it opens anything
POST /ops/recover { system, code } On a founder started with merod run --bootstrap --recover-from DIR: the data keys of the newest system file and the code files, opened on the operator’s machine; the system state is seeded as a founding from backups seeds it
POST /ops/apps/{app}/restore { recovery: { file, key } } Restore one app from its recovery file, signed
POST /recover/holder { holder, slice, apps, code } An organization’s apps on a cluster that does not hold them: the holder’s own device sends the data keys, and the operator admits the import with its signature (cluster / __recover_holder / { holder, slice, apps }; SystemCmd::ImportHolder)

Sealed files come back with their apps: their segments are copied to the disk as stored, checked against their hash when read back (409 RECOVERY_SEGMENTS_MISSING while one is missing), and each file’s data key is wrapped again under the new cluster’s file key. Every file a recovery uses must carry an attestation the node accepts under its peer policy (400 UNATTESTED_RECOVERY_FILE). meroctl ops recovery-keygen, ops backup-key, ops recover and ops restore --recovery-key-file drive the operator’s side (Recover after every key service is gone).

merod inspect reads a stopped node’s store: without the storage key, the layout and sealed sizes only; with it, each group’s metadata (vote, applied index, membership).