Upgrades & migrations

An upgrade moves an app instance to another bundle, usually another release of its package, and reshapes its state if the new code expects a different shape. Writing the migration itself is covered in Migrations; this page is about what the cluster does with it. How a release is parsed, verified and trusted when it is installed is in Packages & frontends.

Who may upgrade

New code reads the app’s data, so an upgrade is one of the three decisions deployment governance assigns:

• Owner side (default): each app’s package declares who, in its signed manifest (upgrades, App upgrades):

  • owner (the default): the app’s owner, or, for an organization’s app, whoever holds MANAGE_APPLICATION on its team or organization-wide;
  • publisher: whoever holds MANAGE_APPLICATION in the organization the package’s signing key is bound to, for every instance; nobody until the key is bound.

  Each upgrades with a passkey session and a step-up over the exact upgrade (POST /apps/{app}/upgrade { version | bundle, migrate? }, then the 412 challenge). The assertion is recorded as consent on the directory’s UpgradeApp command. A package may move from publisher to owner in a later release, never back, and an app is never upgraded from an owner release to a publisher one.

• Operator: only the operator (POST /ops/apps/{app}/upgrade), to a release it installed itself.

An owner-side upgrader may hand one app’s upgrades to CI with an upgrade key: made once with a passkey, it signs each upgrade of that app to a signed release of its package in place of the step-up, and the node refuses it everything else. Who may upgrade and the deadline below are judged for it exactly as for a passkey.

Past a release deadline its approver has not met, an upgrade is refused 409 RELEASE_DEADLINE before the migration runs.

A version alone means the app’s own package and service.

An instance runs a release, not only a WASM. A release’s frontend ships in its signed .mpk and works with that release’s backend only, so approving a release approves its frontend. The directory records the release each instance runs (AppRecord::release: package, version, service), set when it is created (the version named, or the release its deployment offers) and by every upgrade. Releases may share a WASM: one that changes only the frontend is still an upgrade, approved by whoever holds the app’s upgrades exactly as above, and afterwards the instance runs, is reported at, and is served the frontend of the new release. An instance recorded before releases were kept (or running a raw development bundle) is taken to run the release its bundle ships in.

What the passkey signs. The challenge’s arguments are { bundle, migrate, manifest }: the target WASM’s hash and the target release’s signed-manifest hash, so another manifest installed under the same version (other outbound prefixes or frontend origins, the same WASM) is not the upgrade that was approved. The node computes both; a target that changes between the challenge and the assertion no longer matches it, and the upgrade is refused. A raw development bundle has no manifest, and manifest is null.

From the app itself. An app’s frontend cannot upgrade its instance (its session is bound to its package). It asks the node for a consent request ({ kind: "upgrade", subject: { app, toVersion } }); the node’s own page shows the capability sheet (outbound, frontend origins, interfaces, methods, events, timers, migration, build) and takes the owner’s passkey there (The consent page).

Planning the migration

Unless the request names a migration (migrate), the leader plans one from the two bundles’ embedded ABIs (services::plan_upgrade):

Current → target state version	Plan
Same version	Code-only: switch the bundle, run nothing
One version up	Run the target’s #[app::migrate] edge from the current version
Down	Refused: no silent downgrade
More than one up	Refused: upgrade one version at a time
One up, no edge from the current version	Refused
One side embeds an ABI and the other does not	Refused
Neither embeds an ABI	Code-only

Running it

The upgrade is an entry in the app’s own log, so it is ordered against the app’s calls: every call before it ran on the old code, every call after it runs on the new.

1. The leader runs the migration export on the target bundle over the current state (Executor::migrate). The export returns the new root state (borsh).
2. If the target exports __calimero_migration_check, it runs over the migrated state and an optional witness the migration returned (a value the new schema no longer keeps). It must return true; anything else, or a trap, refuses the upgrade. Whatever the check writes is dropped, and the witness is never stored.
3. A migration that addresses the host’s key space is refused.
4. The host writes the returned state as the app’s root entry, stamped after whatever the root entry held so it wins (write_root_state), and the writes become the entry’s diff.
5. The leader appends AppCmd::Upgrade { bundle, base_index, outcome }. Every replica applies the diff and, if the outcome succeeded, switches to the new bundle. As with every entry, followers do not run the migration.
6. Only then does the node record the new bundle and release in the system directory (UpgradeApp, which checks the upgrader’s authority again, checks the release ships the bundle, and notes lastUpgrade). An upgrade to a release with the same bundle runs nothing on the app’s group in an organization-wide run; it is recorded the same way.

A migration runs once, on the leader, and its result is replicated like any other entry: no replica migrates on its own, so there is no window in which replicas hold different schemas.

Failure

A refused plan, a failing migration or a failing check leaves the app on its bundle with its state untouched. The node answers 422 UPGRADE_FAILED with the reason and records it on the instance (UpgradeFailed sets the app’s lastUpgrade with ok: false and the error), so a client can see why. The app’s members read the last upgrade, with the release it went from and to and the kind of actor, in GET /orgs/{org}/apps or GET /auth/apps; the operator reads it without the account or the error in GET /ops/apps (Deployment governance).

Organization-wide upgrade runs

An upgrade run moves every app of an organization that runs package — or only a team’s apps, and with cascade the apps of the teams below it — to version.

• Start. POST /orgs/{org}/upgrade { package, version, team?, cascade?, migrate? } by a MANAGE_APPLICATION holder there, with one passkey step-up for the whole run; POST /orgs/{org}/packages/{package}/upgrade { version, migrate? } by a publisher’s MANAGE_APPLICATION holder, over every instance of a publisher package, its customers’ included (the run is kept on the publisher); or POST /ops/orgs/{org}/upgrade where the operator holds app upgrades. An app the starter may not upgrade is recorded as failed without running anything. A release missing from the cluster is installed from the registry first (owner runs).
• Record. StartUpgradeRun stores an UpgradeRun on the organization (run id, package, version, team, cascade, who started it, its apps, which are done and which failed, and whether it finished); the organization keeps its newest 20 runs.
• Progress. A background task on the node that started it upgrades the apps one by one; each UpgradeApp or UpgradeFailed notes the app on the run and on the instance. An app already on the target bundle is recorded as done. FinishUpgradeRun marks the end.
• Follow and retry. GET /orgs/{org}/upgrades/{run} shows the run; POST …/retry starts a new run over the apps that failed.

The task lives in the memory of the node that started it: if that node stops mid-run, the run stays unfinished and the remaining apps are not upgraded until a retry or a new run.

Upgrading the cluster itself

Moving nodes to a new mero-tee release is not an app upgrade: it is a new node image. Nodes on the new release join the cluster, but receive an app’s data only once that app’s approver has approved the release (Release approvals). App frontends follow the package: {label}.{apps-domain}/ serves the newest installed release.

The cluster version

During a rollout, nodes on the old and the new release replicate one system log together, for as long as an app’s approver keeps old nodes in use. A node applies an entry it cannot decode as malformed command and skips it, so an entry only the new release understands (a new command, or a new field that changes what an old one does) would change the state of the new nodes alone: the old ones would keep answering reads and judging access from the state before it, and if the system leader were old, the caller would be refused an entry the new nodes applied.

So each such shape is a feature with the first core release that applies it (system::cluster_version::FEATURES), and the cluster has an active cluster version: the newest version every node supports.

• What a node supports. GET /health answers clusterVersion: the newest of the node’s core release and the features it knows (a build between two releases supports the coming one). A node from before this release answers only its core release, which counts in its place. A peer asks over the node’s sealed channel, so the answer comes from the attested image.
• Raising it. The system leader asks every registered node (and every member of the system group) at most every 10 seconds while the logged version is below its own, and logs SetClusterVersion { version, nodes } once every one of them answers and the oldest is newer than the logged version. Apply checks that nodes names every registered node and only those, that none is below version, and that version is newer than the logged one: it only moves forward. It is raised automatically, never by hand: raising it is safe only once every node supports it, which is what the leader checks, so an operator finishes a rollout (removes the old nodes, DELETE /ops/nodes/{id}) to raise it. A registered node that does not answer holds it where it is. GET /ops/placement shows the logged version (clusterVersion, null until the first).
• The gate. Every proposal goes through Node::sys_write, which refuses a command using a feature above the active version with 409 FEATURE_NOT_ACTIVE, naming the feature and the version it needs (details: { feature, needs, active }), before anything is logged. Until the logged version reaches it, the proposing node judges by what every node supports now, so a cluster rolling between two releases keeps every feature both have. Routes that ask for a passkey step-up check before asking.
• Older nodes stay out. Once a version is logged, the leader admits no node below it to the system group: a rollback means a new cluster, not an older node in this one.
• Apply never reads it. An entry applies as it did, so replaying a log is unchanged; the gate only decides what is proposed.

A node that still meets an entry it cannot decode logs an error and counts it in calimero_log_undecodable_entries_total{log} (system or app log): with every feature gated it stays at zero.

Feature	Command or field	Since
moveApp	MoveApp	1.0.0-rc.15
appTeamJoin	ScopeApps naming one app team’s join or keep	1.0.0-rc.15
orgQuotas	SetOrgQuotas (also 409 UPGRADE_PENDING first)	1.0.0-rc.15
computePools	SetPools (also 409 UPGRADE_PENDING first)	1.0.0-rc.15
teamCloseDropsBelow	UpdateTeam closing a team (join: invite), which takes the rows below it of whoever loses it; an earlier release keeps them	1.0.0-rc.16
moveAccountsAppTeam	MoveApp giving a moved DM’s people a row on its closed app team (accounts_in_app_team); an earlier release gives none	1.0.0-rc.16
scopeByRelease	ScopeApps judging an instance’s package by its release (by_release); an earlier release judges by the bundle	1.0.0-rc.16
allApps	SetAllApps, and an invite that gives every app added later (allApps); an earlier release cannot read the command and drops the invite’s flag	1.0.0-rc.16
restoreApp	RestoreApp and FinishRestore, an operator’s restore of an app from its sealed backup; an earlier release cannot read them	1.0.0-rc.16
renameOrg	RenameOrg, renaming an organization; an earlier release cannot read it	1.0.0-rc.16
teamMembersBatch	SetTeamMembers and RemoveTeamMembers, a member’s rows in several teams at once; an earlier release cannot read them	1.0.0-rc.16
appTeamChildTeams	ScopeApps naming one app team’s childTeams; an earlier ScopeApp refuses unknown fields	1.0.0-rc.16
clusterVersion	SetClusterVersion	1.0.0-rc.16

The rc.15 features were added before the gate: on a cluster whose every node runs rc.15 or newer they work as before, and they are refused only while a node on rc.14 or older replicates the log. App groups have no gated entry yet; a new app log entry gets a feature the same way, and the cluster version bounds every replica, since no node below it joins.