Observability

A node reports through three routes and its log. None of them carries app state, names or credentials: by design the operator sees counts, Raft status and, for billing, each app’s usage under its opaque id.

Source	Access	What
GET /health	public, in the clear	{ ok, node, systemLeader, release, clusterVersion }
GET /ops/status	ops token, sealed	Raft status of the system group and of every app group this node hosts, governance, the fleet directory, counts
GET /ops/metrics	ops token, sealed	Prometheus text (text/plain; version=0.0.4)
GET /metrics on --metrics-listen	plain HTTP, no credential, off by default	The same Prometheus text
stdout	the host	tracing log lines

Metrics

The _total metrics are counters, the _duration_seconds ones histograms, and the rest are gauges. “Here” means as this node sees it; directory counts are the same on every node once it has caught up.

No label carries a name or anything from a request: labels are route templates (/apps/{app}/intents), route groups, status classes, the node’s error codes, run kinds, call phases, group numbers and node ids. The one exception is the usage metering family calimero_app_*, labelled with an app’s id and its payer’s id (an organization, or a personal app’s owner account) so it can be billed; those are opaque random ids, never names. Whoever scrapes the metrics is the operator, who learns counts, and which app ids use how much, and nothing more.

A few series keep conventional names (http_requests_total, http_request_duration_seconds, merod_build_info, process_*, storage_disk_usage_bytes), so standard dashboards read them.

Metric	Labels	Meaning
calimero_cluster_nodes		Replica nodes in the fleet directory (job nodes are not counted).
calimero_cluster_job_nodes		Job nodes in the fleet directory.
calimero_apps		App instances in the directory.
calimero_accounts		Accounts.
calimero_orgs		Organizations.
calimero_packages		Installed packages.
calimero_bundle_bytes		Bytes of uploaded bundles.
calimero_hosted_groups		App groups this node holds a replica of.
calimero_leading_groups		App groups this node leads.
calimero_event_streams		Event and presence streams open on this node (the limit is 4096).
calimero_system_applied		The system group’s applied index here.
calimero_system_leader		1 when this node leads the system group.
calimero_system_voters		System group voters.
calimero_system_learners		System group learners.
calimero_build_conflicts_total		Counter: entries this node appended as leader that applied as conflicts (built on state that changed first) and ran again.
calimero_system_read_rounds_total		Counter: read-index rounds this node began on the system group. A client’s call or view makes one, wherever it enters and wherever the app’s leader is; requests waiting together share one.
calimero_raft_misaddressed_total	reason	Counter: Raft messages from peers whose addressee did not check out. not_addressee: addressed to another node, refused (421 NOT_ADDRESSEE; peers only); a steady rate means a peer reaches this node at an address the cluster has for another one. no_target: no addressee, from a peer on an earlier release; accepted this release (with a warning at most once a minute) and refused from the next, so it should fall to zero once every node runs this release.
calimero_log_undecodable_entries_total	log	Counter: log entries this node could not decode and skipped (system or app log); its state then differs from the nodes that applied them. Every new command and field is gated on the cluster version, so anything above zero is a bug: alert on it.
calimero_group_applied	group	An app group’s applied index here, per hosted group (the label is the group number, not the app’s name).
calimero_build_info	release	Always 1; release is this node’s release.
calimero_group_term	group	The group’s Raft term here; group is system or an app group’s number. A change is an election.
calimero_group_leader	group	1 when this node leads the group.
calimero_group_apply_lag	group	Entries in this node’s log of the group that it has not applied yet.
calimero_group_follower_lag	group, peer	On the group’s leader: how many entries the follower peer (a node id) is behind.
calimero_group_quorum_ack_age_seconds	group	On the group’s leader: seconds since a quorum last acknowledged it.
calimero_group_snapshot_index	group	The index this node’s latest snapshot of the group covers.
merod_build_info	version, peer_id	Always 1; version is the build’s release, peer_id this node’s numeric id.
storage_disk_usage_bytes	store	Disk a store takes, in allocated blocks, sampled at most every 30 s: datastore (the sealed store: log, state and blobs) and snapshots.
process_start_time_seconds		When this node’s process started, in Unix seconds (from /proc on Linux; elsewhere when it started serving).
process_resident_memory_bytes, process_virtual_memory_bytes		Memory, from /proc/self/status (Linux).
process_threads, process_open_fds		Threads and open file descriptors (Linux).
process_cpu_seconds_total		Counter: user and system CPU seconds (Linux).
http_requests_total	method, path, status, group	Counter: requests answered, by method, route template (path) and status class (2xx, 4xx, …; cancelled when the caller went away before the answer), including peers’ (internal, raft).
calimero_http_errors_total	group, code	Counter: error answers by the node’s error code (NOT_FOUND, QUOTA_EXCEEDED, TOO_LARGE, …; anything else, and a refusal without a code such as a malformed body’s, is OTHER).
http_request_duration_seconds	method, path, status, group	Histogram: time to a response’s head (buckets 1 ms doubling to 16 s); group is the route group (calls, views, sign_in, account, apps, blobs, streams, orgs, packages, ops, raft, internal, attestation, health, other).
calimero_wasm_run_duration_seconds	kind	Histogram: wall time of each WASM run this node made, whatever became of its result (buckets 250 µs doubling to 16 s). kind is call (a call through /intents), xcall (a delivered cross-app call), view, init, timer, action or migrate. Runs happen where usage is counted: calls, init, timers, xcalls and actions on the app’s leader, a view on the replica that answered it.
calimero_call_phase_duration_seconds	phase	Histogram: a logged call’s time on the app’s leader (same buckets), each call observed once in each phase. run: the WASM run that built its entry. commit: from proposing the entry until the log committed it and this node applied it (the Raft round trip to a majority, plus the apply). A call answered without an entry (a replay, a timeout, a refusal) is in neither.
calimero_http_requests_in_flight	group	Requests being handled now.
calimero_http_request_bytes_total	group	Counter: request body bytes, as each body’s length says, at most the sealed cap (64 MiB) per request.
calimero_http_response_bytes_total	group	Counter: response body bytes, where the length was known up front (streams count 0).

The request metrics count what the router matched: a request no route matches (404, 405), the sealed layer’s own refusals and handshakes, and app frontends (--apps-domain) are not counted. A call (/intents) of a method declared as a view counts under calls. The request and run metrics are kept per process: a test that runs several nodes in one process sees their sum on each node.

Where a slow call’s time goes: calimero_call_phase_duration_seconds splits it into the app’s own work (run) and replication (commit); the rest of http_request_duration_seconds{group="calls"} is the sealed channel, routing to the leader and waiting for the app’s proposal lock behind other calls.

# p99 of a call's run and commit over 5 minutes
histogram_quantile(0.99, sum by (phase, le) (rate(calimero_call_phase_duration_seconds_bucket[5m])))
# p99 WASM run time by kind
histogram_quantile(0.99, sum by (kind, le) (rate(calimero_wasm_run_duration_seconds_bucket[5m])))

Usage metering

The calimero_app_* metrics meter each app for request-based billing on shared clusters: calls, WASM CPU and memory while the app runs, state and blob bytes, and outbound requests. Instance-based billing on dedicated clusters needs none of it (it bills node lifetime, which the control plane knows from its VMs). Code: crates/node/src/usage.rs.

Metric	Type	Labels	Meaning
calimero_app_calls_total	counter	app, org, developer, kind	WASM calls this node ran for the app whose answer stood. kind is mutation (a call through /intents, the app’s init, an upgrade’s migration), view (a view, strong or stale, and a declared #[app::view] called through /intents), followup (an event handler, a timer, an xcall, counted on the app it called) or compute (a compute run that ended with a job node’s signed report).
calimero_app_wasm_cpu_seconds_total	counter	app, org, developer	CPU seconds the app’s WASM runs took on this node: the run’s thread CPU time (CLOCK_THREAD_CPUTIME_ID around the run, which runs on one thread) on Linux and other Unix systems; wall time elsewhere (the metric’s # HELP says which). Includes the host functions the run called (state reads, sealing), not the time an outbound request waits (that runs on a thread of its own).
calimero_app_wasm_memory_byte_seconds_total	counter	app, org, developer	Linear memory bytes × wall seconds of the app’s runs on this node. The bytes are the linear memory the module declares at instantiation, a floor of the run’s peak: the runtime does not report how far an instance grew yet.
calimero_app_compute_cpu_seconds_total	counter	app, org, developer	CPU seconds the compute runs this node ended for the app took on job nodes, as the job node’s signed report states. Counted by the node that committed the run’s final entry; a lost attempt is not charged. Calls that ended with a report also count in calimero_app_calls_total{kind="compute"}.
calimero_app_compute_gib_seconds_total	counter	app, org, developer	GiB-seconds of memory (peak memory times wall time) of those runs.
calimero_app_compute_fuel_total	counter	app, org, developer	Fuel those runs used.
calimero_app_outbound_requests_total	counter	app, org, developer	Outbound HTTPS requests the app’s runs on this node sent. A request the gate refused, or a malformed one, never left the node and is not counted; one that failed on the network is.
calimero_app_state_bytes	gauge	app, org, developer	Key and value bytes of the app’s state on this replica (blob marks excluded), sampled every 5 minutes. Logical bytes: the store’s sealing and indexes are not counted.
calimero_app_blob_bytes	gauge	app, org, developer	Bytes of the blobs the app holds, on this replica, sampled with the state.

Labels. app is the app’s id (as /apps lists it). org is the app’s payer, whose quotas it counts against: the organization’s id for an org app, account:{owner account id} for a personal app (: is not in an id’s alphabet, so the two never collide). developer is who is billed for its usage on a shared cluster: for an instance a customer made from a deployment (POST /apps { deployment }), the deployment’s owner, in the same form (an organization’s id, or account:{id}); for an app made from a release directly, the same value as org. A transfer to another payer starts a new series from zero. No other label exists.

Cardinality. org and developer each have one value per app at a time, so they add no series: a node exports at most 12 calimero_app_* series per app it runs or hosts (below), whatever the number of developers.

Where a run is counted. On the node that ran the WASM, once:

• an app’s calls, init, migrations, timers and follow-ups run on the app’s leader, which counts them; followers apply the leader’s diff and run nothing, so they count nothing for those entries;
• a strong view runs on the leader, a stale view on the replica that answered it;
• a run whose result was thrown away (a leader race’s entry that applied as a conflict and ran again, an entry a deposed leader built, a component’s timeout) adds its CPU and memory but no call: the call is counted by the run whose answer stood.

So the cluster’s total for an app is the sum over its nodes for the counters, and any one replica for the gauges (every replica holds the same state; do not sum them).

Series per app. A counter’s series appears once it counts something, and calimero_app_blob_bytes once the app holds a blob, so an app that takes mutations and views costs 5 series on each node that runs or hosts it (mutation, view, CPU, memory, state bytes) and up to 12 with blobs, follow-ups, outbound requests and compute runs. An app’s series leave the node’s output once it is deleted or its replica moves off the node.

Restarts. The counters are in memory and start from zero when merod starts; increase() and rate() treat the drop as a counter reset. Runs made after the last scrape before a crash are not counted.

Queries for each billing meter (MetricsQL; PromQL reads them the same way):

# Mutations and views, per app and developer, over the last hour
sum by (developer, app) (increase(calimero_app_calls_total{kind="mutation"}[1h]))
sum by (developer, app) (increase(calimero_app_calls_total{kind="view"}[1h]))
sum by (developer, app) (increase(calimero_app_calls_total{kind="followup"}[1h]))

# vCPU-seconds while the app's WASM ran
sum by (developer, app) (increase(calimero_app_wasm_cpu_seconds_total[1h]))

# Memory GB-seconds while the app's WASM ran
sum by (developer, app) (increase(calimero_app_wasm_memory_byte_seconds_total[1h])) / 1e9

# Outbound requests
sum by (developer, app) (increase(calimero_app_outbound_requests_total[1h]))

# State and blob GB-months over a 30-day month: the average size, one replica
max by (developer, app) (avg_over_time(calimero_app_state_bytes[30d])) / 1e9
max by (developer, app) (avg_over_time(calimero_app_blob_bytes[30d])) / 1e9

Group by org in place of developer for what a customer organization’s apps used, whoever developed them. For storage, max by picks one replica’s sample rather than summing the replicas; multiply by the replica count if storage is priced per stored copy.

Collecting metrics

/ops/metrics is inside the sealed channel, which Prometheus cannot open. Give the node --metrics-listen ADDR (metrics-listen in config.toml) and it serves the same text at http://ADDR/metrics, in plain HTTP, with no other route. It asks for no credential (sending the ops token in the clear would leak the Ops API’s credential), so bind it to an address only your collector reaches, such as the host’s private interface. The text holds counts, this node’s group indexes and the per-app usage under app and payer ids, never names or state.

Calimero’s own fleet keeps the collection outside this repository: the mero-tee node image starts merod with MEROD_METRICS_LISTEN=127.0.0.1:9464 and its vmagent scrapes that as job merod (mero-tee, roles/merotee); the Grafana dashboard (cluster, usage, latency, errors, Raft, disk) and the alert rules for the table below are in infrastructure, at k8s/grafana-dashboards/chart/dashboards/merod-release-v2.json and k8s/calimero-bases/victoria-metrics-k8s-stack/components/merod-rules.

Without it, collect through a client that seals: meroctl --api URL --tee-profiles PROFILE --ops-token TOKEN ops metrics prints the text, or mdma’s OpsClient (mdma_common.cloud_v2).

What to watch

Signal	Healthy	Look into
Exactly one node with calimero_system_leader 1	always	none for more than a few seconds: no system leader, so no sign-ins, installs or org changes
calimero_system_applied across nodes	within a few entries of each other	a node far behind or not moving
calimero_group_applied{group} across an app’s replicas	close together	a replica not moving: the app runs on the others, with less margin
calimero_system_voters	the founding set	fewer than expected
calimero_cluster_nodes vs voters + learners	equal once registered nodes joined	a registered node that never joined
calimero_build_conflicts_total rate	near zero	a steady rise: leaders keep losing their builds to leader changes or to concurrent writes
calimero_event_streams	well under 4096	near the limit: new streams answer 429
calimero_build_info{release}	the release you deployed, on every node	a node on a release you did not expect

ops status gives the same Raft facts with leaders and member ids per group, which is what you need before a replica move.

Logs

merod logs to stdout through tracing. The default filter is info,openraft=warn for run and warn,openraft=off for merod local; RUST_LOG replaces it:

RUST_LOG=debug merod --node n1 run
RUST_LOG=info,calimero_node=debug,openraft=info merod --node n1 run

Account ids and anything a user or an app wrote stay out of info! and above (scripts/check-no-user-data-at-info.py checks the code), because an operator’s log store receives the default level. App ids and group numbers do appear. Lines worth knowing:

Level	Message	Means
info	merod serving	The node is up, with its id and address.
info	hosting app group / stopped hosting app group	A replica of an app was placed on, or moved off, this node.
info	a registered node joined the system group	ops add-node took effect.
warn	a registered node could not join yet	The leader could not add it; it retries.
info	fetched and verified a release	A registry release was downloaded and checked.
info	migrated the log from the previous storage format	A data directory with log.redb was converted at start.
warn	could not start app group	A replica could not open an app; see the error.
warn	backup failed	A periodic backup could not be written.
warn	blob held by too few nodes, blob not stored, bundle not stored on peer	Blob replication fell short of a majority, or a peer refused.
warn	event handler failed, timer method failed, xcall refused, xcall failed	An app’s follow-up work failed; the app’s own logic is the usual cause.
error	state read failed, reading events failed	The store could not be read; check the disk and the storage key.

The Raft engine (openraft) is quiet by default; raise it with RUST_LOG=openraft=info when you debug elections or membership.