Skip to content

The system group

The system group is Raft group 1, hosted by every node of the cluster (crates/node/src/system.rs; each area’s commands in system/<area>.rs). Its state machine holds everything that is not one app’s own data:

Field of SystemState What it is
nodes, removed_nodes, node_epoch Every registered node (id, base URL, zone, role: replica, or job for a node that only runs compute, and a job node’s pool); the ids removed for good; a counter per id that moves with each registration or removal
apps, next_group The app directory (each instance’s bundle and release, owner, organization, grants, confirm methods, replicas, group, metadata, last upgrade) and the next app group number
bundles Bundle hash → size, uploader, and the release it ships in (the WASM itself is in the blob store)
packages, package_origins Installed packages (pinned signer, installers, and every release: .mpk hash and size, source, services, compute components, pipelines, declared outbound prefixes); the web origins each package registered
accounts, credentials, activity Accounts with their passkeys, labels, aliases and passkey times; credential id → account; each account’s recent sign-ins and step-ups
api_keys, refresh, rotated_refresh API-key hashes, live refresh tokens, and refresh tokens already exchanged
revoked_sessions, revoked_at Sessions that no longer verify, and when they were revoked
recovery_keys, account_recoveries Account recovery codes’ public keys, and recoveries under way
used_proofs, proof_expiry, operator_key The consents and operator signatures already used, and from the ExpireProofs mark on, when each sign-in’s proof is forgotten (the expiry of the session it opened, which its passkey signed; a sign-in that expired by the last one forgotten is refused); the operator’s verify key
orgs, invites, sign_up_invites Organizations (members, roles, capabilities, teams, approvals, recovery key, metadata, upgrade runs; personal marks an account’s own), org invites by token key, cluster sign-up invites
quotas, org_quotas The cluster’s limits and per-payer overrides
pools, price_table, compute, released_runs, run_served, … The compute queue: pools of job nodes, prices, each app’s run slots, and the fairness counters (Compute actions)
app_file_bytes, payer_file_bytes Sealed-file bytes charged per app and per payer
governance Who holds which decision, fixed at genesis (Deployment governance)
operator_releases, operator_outbound The operator’s approvals, where governance gives it those decisions
release_deadlines The operator’s end-of-new-work deadline per mero-tee release (Release deadlines)
publishers, unbound_publishers Package signing key → the organization it is bound to, and its binding generation, kept when the organization goes (Publishers)
deployers, cluster_owner, deployments, deployment_secrets Deployer grants, the cluster’s owner once named, owners’ lists of offered apps, and the secret values each deployment’s managers set for its instances (only the outbound gate reads a value; no answer carries one) (Deployers)
backup_keys Each backup holder’s X25519 public key for the recovery disk: the operator’s, an organization’s or an account’s (Encryption)
cluster_version, session_proofs_from, current_rules_from, attributes_by_release The logged cluster version, and the marks from which later rules hold (below)

A change is a SystemCmd, serialized as JSON and appended through the system leader. Every command that acts for someone names them: actor, account, installed_by, or a credential id. Clients never send commands: a node builds each one and fills the actor in from the request’s verified credential. apply checks the actor’s authority against the state the command applies to and answers { "ok": … } or { "err": … }:

DeleteApp { app_id, actor }
→ refused unless actor is the app's owner, or an admin of its organization
UpgradeApp { app_id, bundle, by_operator, actor, at, consent, … }
→ refused unless governance gives app upgrades to whoever this is, and
(an `owner` package) actor is the app's owner, or, for an org app, holds
MANAGE_APPLICATION on its team or org-wide (owner, admin, lead, granted),
or (a `publisher` package) holds MANAGE_APPLICATION in the publisher org;
and refused past a release deadline its approver has not met (at `at`)
SetReleaseApproval { scope, version, actor, … }
→ refused unless actor holds APPROVE_RELEASES in the org
BindPublisher { signer_id, org, cluster, proof, actor, … }
→ refused unless actor holds APPROVE_RELEASES in org and proof is the
key's signature over (cluster, org, key, the key's generation)
InstallPackage { package, signer_id, release }
→ refused if the package is pinned to another signer, or the same
version is installed with other contents, or the uploader's quota is spent

Because the log orders every command and every replica applies the same commands to the same state, there is one verdict. A refused command is still committed, as a no-op with its reason, and the node that proposed it turns the reason into the HTTP answer (400 REJECTED, or 429 QUOTA_EXCEEDED for a reason that starts with “quota exceeded”).

actor alone is a claim any node of the cluster could make, say SetReleaseApproval { actor: <the org's owner>, version: <its own release> }. So apply first runs system::authorize:

Command Must carry
SetReleaseApproval, TransferApp, DeleteOrg, UpgradeApp (an owner’s), SetOutboundApproval (an org’s or account’s), StartUpgradeRun (an account’s), BindPublisher, AddPasskey, RemovePasskey, CreateApiKey, SetOrgRole, RemoveOrgMember, SetTeamMember, RemoveTeamMember, SetTeamMembers, RemoveTeamMembers, SetCapabilities, SetDefaultCapabilities, CreateInvite The actor’s passkey consent: { subject, method, args, nonce, assertion }, where the assertion signs confirm_challenge(subject, method, args, nonce) with user verification, under a passkey the actor’s account holds in the state
SetOperatorRelease, UpgradeApp { by_operator }, SetOutboundApproval (cluster scope), StartUpgradeRun (the operator’s), SetReleaseDeadline, RegisterNode, RemoveNode, SetQuotas, SetOrgQuotas, SetPools, SetPriceTable, SetReplicas, RestoreApp, SetDeploymentsGranted, SetClusterOwner, deployer grants and the sign-up policy (until the cluster’s owner is named), SetBackupKey (the operator’s own key), ImportHolder The operator’s signature (Ed25519, under operator_key) over "calimero.operator.v1\0" ‖ cluster ‖ 0 ‖ subject ‖ 0 ‖ method ‖ 0 ‖ args ‖ 0 ‖ nonce ‖ 0 (Operator)
RedeemInvite A signature by the invite’s own key, derived from the token (seed = SHA-256("calimero.invite.v1\0" ‖ token)), over "calimero.invite-redeem.v1\0" ‖ invite key (hex) ‖ 0 ‖ account ‖ 0 ‖ now: the log keeps only the public key, so only a node the redeemer gave the token to can redeem it
Genesis Nothing: the first leader writes it once, before any account, with the configured peers (nodes); every later node is registered by the operator
UpgradeApp { run } Nothing of its own: the run it names must exist in the app’s organization, be started by the same party, not be finished, name the app, and the bundle must be a service of the run’s release

The statement must say what the command does (the subject, __approve_release or __withdraw_release as approved says, the version, and so on), and each proof is accepted once (used_proofs), so a consent cannot be logged again after its maker reversed the decision. Every replica reaches the same verdict from the same state, so a forged decision is refused on every node whoever proposed it, the system leader included. Origins, the RP id and the signature counter stay with the node that took the assertion (verify_assertion): they are each node’s configuration, not replicated state.

A new passkey is approved by one the account already has: the consent is checked against the passkeys as they stand, without the new one. An API key’s consent binds its record and the hash of the key, so the log cannot record another key under it. A replica move writes SetReplicas twice under one signature over its target: first the union of the current and the target set (which needs the signature unspent and leaves it so), then the target (which spends it); the union replayed after the move is refused.

Sessions carry their proofs too:

  • Sign-up (CreateAccount): a registration with attestation none carries no signature by the new key, so /auth/passkey/register/finish answers 412 CONFIRM_REQUIRED, and the new passkey signs account:<id> / __create_account / { account, refresh } (system::account_args: the whole account, its release among it, and the sign-up’s first refresh token). A node that saw someone’s new credential cannot register it first under an account and a release of its own. A sign-up asks the passkey twice.
  • Sign-in (CreateRefresh): the login challenge is the challenge of session / __sign_in / { refresh, key, session, expiresAt } (system::sign_in_args), made before the passkey is asked; the log keeps the refresh token only with that assertion, by the passkey the record names, once.
  • Refresh (UseRefresh): each refresh token stands for an Ed25519 key (proof::refresh_key, seed SHA-256("calimero.refresh.v1\0" ‖ token)); its record keeps the public half, and an exchange carries that key’s signature over the new token and the session (proof::refresh_use_message), which must stay the old token’s. A node that only saw the log, or the proposal, cannot exchange a token. Tokens from before keys are not exchanged: their holders sign in again. A token younger than api::REFRESH_ROTATE_AFTER (an hour) is not exchanged at all: a refresh reads its record at a read index and logs nothing (api::kept_refresh).
  • Counters (SetSignCount): the assertion that carries the counter; only the authenticator signs one, so no node can set a counter past it and lock a passkey out.
  • API key revocation (RevokeApiKey): the account’s passkey consent, account:<a> / __revoke_api_key / { keyId } (412 first, as for removing a passkey).

A cluster begun before these proofs replays what it logged then as it applied it: the first system leader on a release that makes them writes RequireSessionProofs, and they are required from that entry on (a new cluster requires them from genesis). The mark is written once and only tightens.

Commands also carry flags that name an earlier release’s rule when they are left false (owner_in_org, owner_in_team, apps_need_consent, teams_seen, free_when_unchanged, subtree, accounts_in_app_team, member_child_teams, revoke_api_keys, by_release, close_drops_below, all_apps_rows, name_fits), so a log that release wrote replays as it applied it. The first system leader on a release that requires the current rules writes RequireCurrentRules (the currentRules feature, so only once the cluster version brings it); from that entry on, every such flag is read as set whatever the entry says, and an organization’s upgrade run is derived from the state where it applies (Upgrades). Nodes keep setting the flags, so a replica on the release before applies these entries the same way.

Still without a proof: RefreshReused, RevokeSession, RevokeAllSessions, labels and aliases, and the rest of the account and app housekeeping; they are trusted from any node of the cluster. Splitting the system group into realms (docs/specs/gate-system-group.md, section 5) is the next phase.

The API checks most rules before it proposes, too, so a request that would be refused usually fails fast and records nothing. The check at apply is the one that counts: two requests that each passed the API’s check against slightly older state cannot both win.

The first node to lead the system group writes SystemCmd::Genesis with the governance it was configured with and its own mero-tee release. CreateAccount is refused until genesis has applied, and Genesis is refused once it has, or once any account exists. A node restarted with other --release-approvals, --app-upgrades or --outbound-approvals flags therefore changes nothing: the state already holds the answer, and it is the state every receipt key attests.

Genesis also carries the configured --peers as the first nodes, and records the operator’s verify key (--operator-key) when one is configured; every later node is registered by the operator’s signed POST /ops/nodes.

A sign-up (CreateAccount) makes no organization once the logged cluster version brings Feature::OrdinaryOrgs (1.0.0-rc.32; apply drops a seed an entry carries then). Below it, a sign-up made the account’s personal organization in the same entry (personal_org, named Personal, approving the release of the node it signed up through), and those entries replay so.

A cluster founded again from its backups (--restore-from) writes no genesis: its founder forms the system group alone and seeds it with the newest system backup, refused unless the governance and operator key it is configured with are the backup’s; the state then names --peers as the fleet and every app waits for its own restore (Found a cluster again).

A cluster brought back from a recovery disk (merod run --bootstrap --recover-from DIR) writes none either: its founder forms the system group alone and waits for the operator’s POST /ops/recover, which seeds it from the system file the operator opened on its own machine, under the same checks (Recover after every key service is gone). An organization’s or account’s slice comes onto another cluster through POST /recover/holder (ImportHolder, admitted by that cluster’s operator’s signature), refused where any of its organization, accounts or apps is there already.

POST /ops/nodes takes an optional role: replica (the default) or job. A job node runs compute for the cluster’s apps and never holds the cluster key, so it is never a Raft member, not even a learner: a member keeps its log sealed under that key. It is also never placed an app, never a target or a majority member of a blob or bundle push, and never sent the cluster key. /ops/placement lists it with its role and an app count of zero, and /ops/apps/{app}/replicas refuses a request that names one. DELETE /ops/nodes/{id} removes it at any time and changes no voter. A node’s role is fixed until it is removed, an address belongs to one node in any spelling, and --peers must list only replica nodes. Registering a job node is refused (409 CLUSTER_NOT_UPGRADED) until every replica node, voter and learner runs at least the registering node’s release, so no older node ever reads a job record.

Every node runs a reconcile loop (on each applied system entry, on a follow-up kick, and every 500 ms). On the system leader it also:

  • writes Genesis if nothing has (the configured --peers come with it, as Genesis { nodes }; reconcile registers nobody after that);
  • writes the marks a newer release brings, once each (RequireSessionProofs, AttributeByRelease, RequireCurrentRules, the last two only once the cluster version has the feature);
  • adds every registered replica node that is not in the system group as a learner (Replication), once it attests its transport key as a cluster node (a job node registered as a replica never does, so it is never admitted) and supports the logged cluster version;
  • raises the cluster version once every registered node supports a newer one (SetClusterVersion);
  • promotes caught-up learners to voters while there are fewer than three.

On every node it starts the app groups this node is a replica of, stops the ones it no longer is, runs a newly created app’s init where this node leads its group, and drives the follow-ups and timers of the groups it leads.

The system group’s machine keeps its whole state in memory (Machine::IN_MEMORY) and is checkpointed with each snapshot. Reads come in three strengths:

  • Local (Node::sys): what this replica has applied. Used for lookups that can tolerate a moment’s lag.
  • Until found (Node::sys_until): wait up to 2 seconds for a lookup to find something. A passkey registered through node 1 and used through node 2 a moment later is found instead of answering PASSKEY_UNKNOWN_HERE.
  • Fresh (Node::sys_fresh): a read index. The node asks the system leader for its applied index (internal/system/index, or locally if it leads) and waits up to 2 seconds to apply it. Every request that carries a credential, every gated request without one, and every client call on the app’s leader read the state this way, so a revocation or removal committed through any node holds before the next request anywhere.

A request makes one read-index round, where it is first judged (principal, or gate for an anonymous caller). Its other checks, on the same node or on the app’s leader, reuse that round: the request carries the index the round named (Principal::sys_index, CallRequest::sys_index on internal/call), and each check waits until its own replica has applied that index (Node::sys_fresh_from) instead of asking the system leader again. A client’s call or strong view costs one round, about one round trip to the system leader, wherever it enters and wherever the app’s leader is. calimero_system_read_rounds_total in /ops/metrics counts the rounds each node begins.

Why the carried index is as fresh as a round of the leader’s own:

  1. The round answers the system leader’s applied index after the leader confirmed, with a quorum, that it still leads. Every system write committed before the round began is at or below that index.
  2. A request takes only a round begun after it arrived (concurrent requests share the next round, never one already under way), so every write committed before the request arrived is at or below the index.
  3. Raft’s log matching makes the entries up to that index the same on every replica, and a replica’s applied index only grows. So a replica that has applied the index, on any node, holds each of those writes: a removal, revocation or demotion committed before the request is seen by every check made for it, the app leader’s included.
  4. A write committed after the request arrived is concurrent with it; ordering the request before it is linearizable, as a second round could have ordered it either way too.

If a replica cannot apply the index within 2 seconds, a round of its own would name an index at least as high, which it could not apply either; the check goes ahead best effort, as sys_fresh does. Calls nobody’s request started (timers, deliveries, actions) carry no index and run a round of their own.

Node::sys_write appends through the system leader, after refusing a command that uses a feature above the cluster version (409 FEATURE_NOT_ACTIVE): locally if this node leads, else by posting the command to the leader’s internal/system route over the peer sealed channel. It retries until its 10-second deadline (SYSTEM_WRITE_TIMEOUT) to ride out an election, then answers 503 UNAVAILABLE naming the group’s leader, voters and the nodes behind it. A follower that receives internal/system refuses rather than forwarding again.

Some system commands are not idempotent yet, so a forwarded system write keeps the long HTTP timeout instead of the short bounded forward app calls use.

Quotas live in the system state and are set by the operator (POST /ops/quotas, operator-signed); POST /ops/quotas with org sets one payer’s compute and file overrides (SetOrgQuotas, org_quotas). They are checked inside apply, so every node refuses the same command for the same reason:

Quota Default Checked by
maxAppsPerAccount 100 CreateApp
maxBundleBytesPerAccount 1 GiB PutBundle, InstallPackage (operator installs count against no account)
maxApiKeysPerAccount 50 CreateApiKey
maxOrgsPerAccount 20 (organizations owned; below cluster version 1.0.0-rc.32 the personal one does not count) CreateOrg
maxAccounts 100 000 CreateAccount (where sign-up is open, this bounds the cluster)
maxFileBytesPerOrg 50 000 000 000 ChargeFileBytes (plaintext bytes of the files of an organization’s apps; a personal organization is one like any other)
maxConcurrentRunsPerOrg 4 AcquireRun, ReleaseRun (compute runs of an organization’s apps granted to job nodes at once; the rest wait, the payer served longest ago first)
computeFuelPerOrg 10 000 000 000 000 AcquireRun, ReleaseRun (the declared fuel of an organization’s granted runs, summed)
maxFuelPerDeveloperPerDay "0" (a decimal string; no limit) not at apply: a client’s call or an xcall delivery to any app of the developer, where it runs, against the developer’s fuel today in its apps’ usage books, refreshed from the peers every 5 s (Q20, D76). The developer is usage::developer’s, the identity usage is billed to: a deployment’s owner for its instances, else the app’s organization or owner (D78); init, timer ticks and event handlers still run, and count. Written only once every node runs a release that knows it (409 UPGRADE_PENDING)

GET /auth/usage shows an account what it uses against them.