Observability

There are three ways to observe a Clustron Zaris store, from quickest to most integrated:

• The live console. Watch-ZrStoreMetrics renders a store's throughput and activity live, per node, with no setup. It is the fastest way to confirm a store is serving traffic or to spot an imbalance during an incident. See Monitoring and metrics.
• The metrics Zaris emits. Each node records a catalog of operation, replication, and memory metrics. Some are surfaced by the live console; a separate set of OpenTelemetry instruments is built for export.
• OTLP export to your own stack. With telemetry enabled, a node exports its OpenTelemetry instruments over OTLP (gRPC/HTTP) to any collector — Grafana/Tempo, Prometheus (via the collector), Jaeger, Datadog, Azure Monitor, and so on.

This article is a reference for the second and third: what metrics exist, and how to turn on export.

note

Telemetry export is off by default and opt-in. A node with no telemetry configuration initializes no exporter and pays no overhead — the metric instruments stay inert until an OpenTelemetry MeterProvider subscribes to them. Metrics, unlike traces, are never sampled: when the catalog is on, counters are recorded at full fidelity.

Two metric planes

Zaris emits metrics on two independent planes, and it helps to know which is which:

• The live-console plane. A set of counters and gauges (the short keys below — get, put, mem.*, repl.*, …) fed by the same internal accountant the node acts on, surfaced by Watch-ZrStoreMetrics and the web console. These are what you see in the console's Zaris metric category.
• The OpenTelemetry plane. A System.Diagnostics.Metrics meter named Clustron.Zaris that publishes the clustron.zaris.* instruments below — the node registers this meter with the OpenTelemetry SDK for OTLP export. As of 1.1.0 the node also bridges the internal live-console catalog onto this meter, so the short keys below can be exported over OTLP alongside the semantic instruments.

Metric keys were shortened in 1.1.0

The live-console keys dropped the zaris. / memory. prefixes — for example zaris.get is now get, and memory.active.bytes is now mem.active. If you built dashboards or alerts against the old names, update them to the keys shown below.

Metrics catalog

The names below are the real metric keys emitted by the code. Grouped tables follow.

Throughput and store state

Per-operation counters and the store-size gauge, surfaced in the live console.

Metric	What it measures
get / get.err / get.miss	Get operations served / failed / missed (key absent).
put / put.err	Put operations served / failed.
del / del.err	Delete operations served / failed.
clear	Clear operations.
reqs	Total client requests handled.
store.items	Items held by this node (active + replica); a gauge.
store.scan / store.scan.err	Scan queries / failures.
store.search / store.search.err	Search queries / failures.
store.agg / store.agg.err	Aggregate queries / failures.
store.groupby / store.groupby.err	Group-by queries / failures.
migr	Partition migrations.
migr.put	Puts applied as part of a migration.
ttl.expired	Items removed by TTL expiration.
ttl.sweeps	TTL sweep passes run.
ttl.due	Items found due for expiration in a sweep.

Replication

Per-node replication traffic and delivery health.

Metric	What it measures
repl.sent / repl.recv	Replication messages sent / received.
repl.err	Replication sends that failed.
repl.puts.sent / repl.puts.recv	Puts replicated outbound (primary → replica) / inbound.
repl.dels.sent / repl.dels.recv	Deletes replicated outbound / inbound.
repl.clears.sent	Clears replicated (as per-key deletes) to replicas.
repl.locks.sent / repl.locks.recv	Lock state replicated outbound / inbound.
repl.reverted	Replicated operations reverted.
repl.ack.sent / repl.ack.recv / repl.ack.err	Replication ACKs sent / received / failed.
repl.sync.timeout	Synchronous-replication waits that timed out.
repl.unacked	Batches sent but not yet ACKed by a replica (aggregate). Near zero in steady state; climbing means a replica is not acking (wedged / half-open); unbounded means delivery is failing silently.
sync.req	Full replica-sync operations requested.
sync.chunks	Snapshot chunks received during a replica sync.
sync.done / sync.err	Replica syncs that completed / failed.
sync.reads.blocked	Reads blocked on a replica because its sync is still pending.
sync.writes.rej	Writes rejected by a node acting as a replica.
sync.surplus	Syncs that finished with the replica ahead of the source's frozen snapshot (a live write landed mid-resync).

Memory and eviction

Footprint gauges published each reconcile tick, plus eviction counters. These are fed by the same accountant that enforces the ceiling, so the dashboard and the node never disagree. For the model behind these numbers, see Memory and eviction.

Metric	What it measures
mem.active	Accounted bytes of active-owned data (compared against the ceiling).
mem.replica	Accounted bytes of replica mirrors held (not charged to the ceiling).
mem.index	Bytes attributed to indexes and auxiliary structures.
mem.total	Total accounted footprint (active + replica + index).
mem.ceiling	The configured per-process ceiling on active-owned data.
mem.headroom	ceiling − total footprint on this node.
mem.usage	Total footprint ÷ ceiling, expressed per-mille (0–1000 = 0.0–1.0).
mem.over	1 while the node's total footprint is above the eviction trigger threshold, else 0.
mem.items	Count of active-owned items.
mem.evict	Items evicted to stay under the ceiling.
mem.evict.bytes	Bytes reclaimed by eviction.

The eviction metrics carry a policy attribute (lru, lfu, ttl-aware, no-eviction) and, on admission rejections, a reason attribute (ceiling_exceeded, max_item_size, tenant_ceiling, migration).

note

There is no dedicated client-connection-count metric in the emitted catalog. Connection counts and per-node age are surfaced in the management console's Nodes view, not as a store metric.

Instruments exported over OpenTelemetry

These are the instruments published on the Clustron.Zaris meter — the plane the node registers with the OpenTelemetry SDK for OTLP export.

Instrument	Kind	Unit	What it measures
clustron.zaris.operation.count	Counter	{operation}	Per-operation throughput (successes + failures).
clustron.zaris.operation.duration	Histogram	ms	Per-operation latency.
clustron.zaris.operation.errors	Counter	{error}	Per-operation error count.
clustron.zaris.topology.zero_segment_owned_partitions	Gauge	{partition}	Partitions with a live owner but owning 0 segments (zombie).
clustron.zaris.topology.wrong_owner_partitions	Gauge	{partition}	Partitions whose live owner is not a member of the partition group.
clustron.zaris.topology.under_replicated_partitions	Gauge	{partition}	Partitions with fewer live replicas than the map expects.
clustron.zaris.replication.replica_lag_batches	Gauge	{batch}	Batches sent to a present replica but not yet acked, tagged by replica.
clustron.zaris.write.during_ownership_transition	Counter	{write}	Acked writes admitted while a segment's ownership was transitioning (the write-fence did not hold them).
clustron.zaris.mapless_mint.decisions	Counter	{decision}	Mapless-leader mint decisions by class.
clustron.zaris.mapless_mint.fresh_peer_digests	Histogram	{peer}	Fresh peer map-digests observed at each mapless mint decision.

The operation instruments are tagged with metadata only — never a stored key, value, or payload:

Tag	Values
clustron.operation	The operation / command name.
clustron.origin	native (internal cluster control plane) or client (client-server request).
clustron.status	ok or error.
clustron.error_category	A short label for the failure class (present on errors).

Every exported metric also carries the resource attributes set at export time: service.name = clustron-zaris, service.instance.id = the node id, plus clustron.store and clustron.node.

Export via OpenTelemetry (OTLP)

Telemetry is configured in the node's Telemetry configuration section (for example, the Telemetry section of appsettings.json). The node's OpenTelemetry bootstrap reads these keys:

Key	Type	Default	Effect
Telemetry:Tracing:Enabled	bool	false	Turn trace export on.
Telemetry:Tracing:SampleRatio	double	1.0	Head-based trace sample ratio (1.0 = all). Metrics are never sampled.
Telemetry:Tracing:Console	bool	false	Also emit to the console exporter (local dev); applies to metrics too.
Telemetry:Metrics:Enabled	bool?	null	Turn the metrics catalog on. null follows the tracing decision — enabling telemetry gives you both — while an explicit true/false scopes metrics independently (metrics-only or traces-only).
Telemetry:Otlp:Endpoint	string	""	OTLP endpoint (for example http://collector:4317). Empty means console-only when telemetry is enabled.

The standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable also switches export on and is honored as the endpoint when Telemetry:Otlp:Endpoint is not set.

A node exports over OTLP when tracing is enabled in config or the OTEL_EXPORTER_OTLP_ENDPOINT env var is set; metrics follow the tracing decision unless scoped independently. If telemetry is on but no endpoint is resolved, the node falls back to the console exporter (the local-dev default).

Point a node at a collector

Enable metrics and send them to a collector on collector:4317:

{
  "Telemetry": {
    "Tracing": { "Enabled": true, "SampleRatio": 1.0 },
    "Metrics": { "Enabled": true },
    "Otlp": { "Endpoint": "http://collector:4317" }
  }
}

Or, equivalently, with the standard environment variable:

OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317

tip

For a metrics-only setup, set Telemetry:Metrics:Enabled to true and leave Telemetry:Tracing:Enabled at false. The metrics catalog exports without paying for trace spans. To scrape with Prometheus, point the node at an OpenTelemetry Collector and expose the collector's Prometheus endpoint.

The configuration reference summarizes the logging and telemetry sections alongside the rest of a store's settings — see Configuration reference.

What to watch

A short set of signals worth alerting on, tied to the real metrics above:

• Memory near the ceiling. mem.usage approaching 1000, or mem.over = 1, means the node is at or past its eviction trigger; expect mem.evict and mem.evict.bytes to climb. Falling mem.headroom is the leading indicator.
• Replication falling behind. A climbing or non-zero repl.unacked (or a sustained clustron.zaris.replication.replica_lag_batches for a specific replica) means a replica is not acking — a wedged or half-open connection dropping acks. repl.sync.timeout rising points the same way.
• Errors and unavailability. A rising clustron.zaris.operation.errors (broken down by clustron.error_category), or growth in the *.err operation counters (get.err, put.err, del.err), signals timeouts or unavailable partitions. Correlate with the client-side retry behavior in Client resilience patterns.
• Topology health. Non-zero clustron.zaris.topology.wrong_owner_partitions, ...zero_segment_owned_partitions, or ...under_replicated_partitions means the cluster's ownership or replication is not where the map expects it.
• Write-fence gaps. A non-zero clustron.zaris.write.during_ownership_transition reveals an acked write admitted during an ownership handoff — the window in which an acknowledged write can be lost at cutover. Correlate a spike with any lost keys.

Next steps

• Monitoring and metrics — the live Watch-ZrStoreMetrics console view.
• Client resilience patterns — what the client retries, and the statuses behind the error metrics.
• Configuration reference — the full store and client configuration, including telemetry.
• Memory and eviction — the model behind the memory.* gauges.