The operator & the Ops API
The operator runs the machines: it starts nodes, gives them a network and
disks, and keeps the cluster healthy. It is not trusted with app data. The
Ops API (/ops/*) gives it what operating requires and nothing that opens
an app. mdma’s OpsClient and meroctl --ops-token TOKEN ops … use it.
The credential
Section titled “The credential”The operator holds a bearer token. A node is given either the token
(--ops-token) or only its SHA-256 (--ops-token-sha256), so a machine’s
metadata or configuration need not hold anything usable. The token is compared
in constant time, inside the sealed channel like every other request. A node
given neither has the Ops API disabled. The operator’s actions that the system
log records carry the actor operator, which no account id can be.
The token reaches the node, not the log: every node of the cluster can append
to the system log, so a decision the governance gives the operator must carry
the operator’s own proof. That is an operator key, Ed25519, whose public
half the nodes are given (--operator-key <hex>, MEROD_OPERATOR_KEY) and the
first leader records at genesis. Every operator decision
(the second table below) carries the key’s signature, checked where it applies
on every node (system group).
The signed bytes are, with 0 a zero byte and every field UTF-8:
"calimero.operator.v1" 0 cluster 0 subject 0 method 0 args 0 nonce 0subject is cluster (release and outbound approvals, deadlines, nodes,
quotas), the app id (an app’s upgrade or replica move) or org:<id> (an
organization’s run); method is __approve_release, __withdraw_release,
__approve_outbound, __upgrade, __upgrade_org, __upgrade_org_retry,
__release_deadline, __register_node ({ id, addr, zone }),
__remove_node ({ id }), __set_quotas ({ quotas }, every limit after
the change) or __set_replicas ({ replicas }, the target set sorted; one
signature covers both writes of the move); args is the exact JSON text the
node states; nonce is base64url, used once. A decision sent without a proof
answers 428 OPERATOR_SIGNATURE_REQUIRED with details { subject, method, args, nonce, cluster, message }, message being the bytes above in base64: the
caller signs message and sends the same request again with
"operatorProof": { "nonce": <details.nonce>, "signature": <hex> }. A signature
that does not verify is 403 BAD_OPERATOR_SIGNATURE; one used before is
refused by the log. With no key recorded, operator decisions are refused
(403).
What it can do
Section titled “What it can do”| Route | What |
|---|---|
GET /ops/status |
This node, the system group’s Raft status, the Raft status of the app groups hosted here (by app id), the node list, the governance, and counts of apps and accounts |
GET /ops/metrics |
Prometheus text: nodes, apps, accounts, organizations, packages, bundle bytes; groups hosted and led here; open streams; system applied index, leadership, voters, learners; each hosted group’s applied index; the build release |
GET /ops/placement |
Each app’s group and replica node ids; each node’s zone, attested release and app count |
GET /ops/apps |
What each app runs and which releases may hold it (below) |
GET /ops/orgs |
Each organization’s id and the releases approved to hold its apps |
POST /ops/nodes |
Register a node, signed (428 first); the system leader then adds it as a learner (Replication). The nodes a cluster starts with are registered at genesis |
DELETE /ops/nodes/{id} |
Remove a node, signed (428 first; the proof in a JSON body) |
POST /ops/apps/{app}/replicas |
Move an app to a new replica set, onto approved releases only, signed (428 first) (Replica moves) |
GET / POST /ops/quotas |
Read or set the per-account quotas; setting is signed (428 first; operatorProof beside the limits) |
POST /ops/backup |
Write sealed snapshots of the app groups this node leads (below) |
POST /ops/deployments/granted { operatorProof } |
Move the cluster’s deployments from open to granted, once, never back (Deployers) |
POST /ops/deployers { deployer, until?, operatorProof }, DELETE /ops/deployers/{key} (or POST /ops/deployers/{key}/revoke, for a client whose DELETE carries no body) |
Grant (until: RFC 3339) or revoke a deployer ({ "account": id } or { "org": id }; the key is account:<id> or org:<id>), signed with the operator key, while the operator grants deployers |
POST /ops/cluster-owner { owner, operatorProof } |
Where governance says deployers: owner: name the cluster’s owner, once; it grants deployers from then on |
GET /deployers |
The deployments rule, the owner and every grant |
The operator does not act inside a deployment: creating one, offering,
withdrawing and deleting take a manager’s passkey (Deployments).
| POST /ops/releases/{version}/deadline { after, operatorProof }, DELETE … | Set (RFC 3339) or lift (after: null, or DELETE with the proof in its body) a release’s end-of-new-work deadline, signed with the operator key (428 first): after it, a party that has not approved the release creates, installs and upgrades nothing until it does; running apps go on (Release deadlines). Public at GET /releases |
And, only where the deployment’s governance gives the operator that decision:
| Route | Decision |
|---|---|
POST /ops/releases { version, approved } |
Release approvals, for every app |
POST /ops/apps/{app}/upgrade, POST /ops/orgs/{org}/upgrade |
App upgrades, to a release it installed or one the app’s owner may see |
POST /ops/packages, /ops/packages/install, GET /ops/packages, DELETE /ops/packages/{p}[/{v}] |
Installing the releases it upgrades to (it lists and removes only what it installed) |
POST /ops/outbound { package, prefixes, replace } |
Outbound approvals, for every app |
Elsewhere, the operator configures the node: the outbound policy (which hosts any app may reach, and the secrets requests may carry), the registry releases are installed from, the domain app frontends are served under, and the measurement policy for peers.
What runs where
Section titled “What runs where”GET /ops/apps is how the operator follows a rollout: per app, its id, group
and scope ({ kind: "org", org }, or { kind: "account" } for an
account’s own app, never the account), the bundle digest it runs, its publisher (the package’s pinned signer,
with the package), who holds
app upgrades, its last upgrade (from and to digests, when, whether it
succeeded, and the actor kind: owner, member or operator), the
mero-tee releases approved to hold its data, and each replica’s node, release,
and whether that release is approved for the app. GET /ops/orgs gives each
organization’s approved releases. Both are paged (?after=ID&limit=N, at most
MAX_REPORT_PAGE, 100, per page; next continues).
What it leaves out is deliberate:
- Package names and versions only for releases the operator installed.
Which package an app runs can say what an organization does (a package
uploaded for one customer is named after it), so the report names a release
only where
sees_releaselets the operator see it anyway: the releases it installed itself (/ops/packages, its catalog). For any other app,packageisnulland only the bundle’s digest (hex(SHA-256)of its WASM) appears: enough to group apps that run the same code and to see an upgrade happen, nothing that names it. - The actor’s kind, not the actor. An upgrade’s account, its error text (which the app’s code can write) and its upgrade run are left out.
- Approved releases and governance are not secrets: every replica attests the governance, and the approved releases decide which nodes the operator can place an app on.
What it cannot do
Section titled “What it cannot do”There is no Ops route that runs a view, reads state, streams events, lists an account’s apps, downloads a blob or a bundle, or returns a credential. Status, metrics and the reports above give counts, ids, digests, releases and Raft positions; app names, organization names, people’s names and app state never appear. The operator cannot sign in as anyone: sessions are minted only after a passkey ceremony, each signed by the attested node that ran it. Nor can the ops token alone make a decision the log accepts: that takes the operator key.
The operator does control availability. It can stop, freeze or restart nodes, withhold the network, or refuse to register nodes, and it sets release deadlines, which stop new work on old images but move no data and block no approval. An app stays available while a majority of its replicas run.
Backups
Section titled “Backups”With --backup-dir, every node writes, every 60 seconds (and on
POST /ops/backup), a copy of the current snapshot file of each app group it
leads, first asking for a fresh snapshot if the last one is older than what it
has applied. A backup is the log’s own snapshot file, sealed frame by frame
under the cluster’s storage key, stored as
{dir}/{app}/{index:020}.{group}.{snapshot}.snap. Only a node holding the
cluster’s K_r can open one (backup::latest). The directory stands in for
object storage.
Restoring a snapshot into a running cluster is not done yet.
Offline inspection
Section titled “Offline inspection”merod inspect reads a stopped node’s store: without the storage key, the
layout and sealed sizes only; with it, each group’s metadata (vote, applied
index, membership).