High availability and disaster recovery
Clustron Zaris keeps a store available while individual nodes come and go, and it keeps recent writes safe when you ask it to. This article explains what high availability (HA) means for a Zaris cluster, how failover and re-replication actually work, and — just as important — what disaster recovery (DR) can and cannot mean for an in-memory store that has no data on disk.
Two ideas run through everything below:
- High availability is about surviving the loss of a node without losing the ability to serve, and — with the right replication mode — without losing recent writes. It is a live-cluster property.
- Disaster recovery is about coming back from a larger loss. Because Zaris holds data only in memory, DR here is about replication and rebuild, not about restoring a data backup. There is no data backup to restore.
Zaris is an in-memory store. There is no disk persistence, write-ahead log, or data snapshot on any node — a node's records live in process memory only. Durability comes from replicas on other machines and from your system of record. Losing every copy of a partition, or the whole cluster, loses that in-memory data permanently. Plan for this rather than assuming a restore path exists. (DataDirectory holds config and certificates, not data.)
Replication for availability
Availability under node loss comes from the replication factor (RF) — the number of copies of each partition that exist across the cluster. Each partition has one primary that serves its reads and writes, plus RF − 1 replicas that stay close to the primary's state as hot standbys. Copies of the same partition are always placed on different machines, so losing one machine can never take out both a primary and its replica.
Set the replication factor when you create the store; New-ZrStore requires it explicitly, and the config property (replication.factor) defaults to 1.
- RF = 1 — one copy, no replica. A node loss means data loss. Use it only for caches you can rebuild.
- RF = 2 — a primary plus one replica. Recommended for production; survives one machine failure.
- RF = 3 — a primary plus two replicas. Survives two simultaneous machine failures.
For production the replication factor must not exceed the number of machines, so each copy lands on a distinct machine. Run at least RF = 2 for any store whose availability matters.
For the full model, see Replica and Partitioning and replication.
Failover
When a primary's node is lost, the cluster does not wait for it to come back. One of that partition's replicas is promoted to primary and takes over serving. What fails over is the partition's primary role — it moves from the lost node to a surviving replica — not the partition itself and not the node. The partition keeps its stable code-word name (Amber, Onyx, and so on) throughout, so you can follow it across the promotion in the console and in diagnostics.
Clients re-route automatically. They do not need to be told where a partition now lives — the client's built-in retry loop refreshes its routing map and lands the retry on the new primary. Concretely, an operation that reaches a node mid-transition comes back as a routing status the client handles for you:
Moved— routing is stale; the client refreshes its ownership view and retries.OwnershipChanging— ownership is mid-transition; the client refreshes and retries.
Both are in the client's automatic-retry set, so a well-behaved application sees a brief blip, not an error. See Client resilience patterns for exactly what the client retries and what it hands back to you.
The asynchronous data-loss window
Failover preserves availability, but whether it preserves the most recent writes depends on the store's replication mode.
With the default Async replication, the primary acknowledges a write as soon as it is applied locally and streams to replicas in the background. That is the lowest-latency path, but it means a replica can trail its primary. If the primary's node fails before the trailing writes reach a replica, those writes are lost on promotion — the newly promoted replica never received them.
Sync replication closes this window. The primary holds the acknowledgement until replicas confirm the write, according to the write-quorum policy (replication.quorum: All, Majority, or BestEffort), waiting up to syncAckTimeoutSeconds. You pay write latency for it, but a failover no longer drops acknowledged writes.
| Replication mode | Durability under a primary loss | Cost |
|---|---|---|
| Async (default) | Recent, not-yet-replicated writes can be lost on promotion | Lowest write latency |
Sync, quorum Majority | Acknowledged writes survive as long as a quorum of replicas held them | Higher write latency; a write waits for a majority of replicas |
Sync, quorum All | Acknowledged writes survive as long as any replica survives | Highest write latency; a slow or missing replica can delay or fail the write |
Under the default Async replication, a primary failure can silently drop the newest writes. Choose Sync for stores where a lost recent write is unacceptable, and size syncAckTimeoutSeconds accordingly. This is a durability-versus-latency decision, not a setting you can leave to a benchmark number.
For the consistency implications — including why enabling replica reads changes read-after-write behavior — see Consistency.
Node loss and re-replication
Losing a node does more than trigger a single failover. That node held a set of partition copies — some as primary, some as replica — and every one of those copies is now gone from the cluster. For each affected partition:
- If the lost copy was the primary, a replica is promoted (see Failover).
- The partition is now under-replicated: it has fewer copies than its replication factor calls for.
The cluster then rebuilds a fresh replica on a healthy machine to restore the configured replication factor. The promoted primary keeps serving throughout the rebuild; the partition simply shows as under-replicated until the new copy is in place.
There is a capacity implication you must plan for.
Note that this recovery only works while a partition has at least one surviving copy. If a single failure — or a rebuild that races another failure — takes out every node of a partition group, there is nothing left to promote or rebuild from.
When every copy is lost: the eviction window
If all members of a partition group go down at once, that partition's in-memory data is gone. The cluster does not repartition immediately, because most total outages are transient. Instead it marks the partition unavailable and starts a grace window — runtime.partitionGroupEvictionWindowSeconds, 30 seconds by default and configurable:
- If any member returns within the window, the countdown is cancelled and the partition recovers on that node.
- If the window expires with no member back, the cluster evicts the partition group and regenerates the map from the live members, folding that keyspace onto surviving nodes. New writes for those keys then succeed again on a live node — but the original data is not recovered.
Tune the window to your fault profile: raise it if whole groups tend to blip and return (to avoid folding a partition that would have come back); lower it to restore write availability for a lost keyspace sooner. The mechanism is described in full under Failure and recovery. The data and request load the lost node carried do not disappear — they are absorbed by the remaining nodes. After a node loss the surviving machines hold more partitions and more data each, and they also carry the extra work of streaming the rebuilt replicas. Size the cluster so that the remaining nodes have the memory and headroom to take on a failed node's share; a cluster running near its memory ceiling before a failure can hit eviction or capacity limits during the rebuild.
Disaster recovery for an in-memory store
This is where the in-memory model must be stated plainly. Zaris has no data backup and no restore. There is no on-disk copy of your records to recover from — not a snapshot, not a log. Replication protects you against the loss of some nodes; it does not protect you against the loss of all copies of a partition or of the whole cluster.
A total cluster loss — every node down at once — loses all in-memory data. There is nothing to restore it from inside Zaris. Recovery means restarting the cluster and rehydrating the data from your system of record. Do not treat Zaris as the durable copy of anything you cannot regenerate.
Design for this from the start:
- Keep the source of truth durable elsewhere. Zaris is a store and cache in front of a durable system — a database, an event log, an object store. Treat that system as the authority, and Zaris as a fast, distributed view of it that can be rebuilt.
- Rebuild by replaying from the source. After a full restart, repopulate the store from the durable system rather than expecting Zaris to have kept the data. A rebuild is a load operation, not a restore operation.
- Let TTLs repopulate naturally. For cache-shaped data, write entries with a time-to-live so that after a rebuild the working set refills on demand as keys are read and missed against the source of truth, rather than requiring a full bulk reload up front.
- Spread copies across failure domains. Because copies already land on distinct machines, place those machines in distinct racks, zones, or hosts so a single infrastructure fault does not take out every copy of a partition at once.
Operational actions
Day-to-day HA also depends on performing node and cluster operations without causing avoidable loss.
Inspect node and partition health
Before and after any operation, check what each node owns as primary and holds as replica, and whether any partition is under-replicated.
Get-ZrNode -StoreName orders
Get-ZrNode -StoreName orders -Name 10.0.0.11-n0
The detailed view reports a node's primary and replica partitions, its map version, and time since the last role change. See Get-ZrNode.
Safely restart a node
To restart a node — for a config change, a patch, or to clear a stuck process — use Restart-ZrNode. With a replicated store (RF ≥ 2), restarting one node triggers a failover of any partition it owned as primary, and clients re-route automatically; the node rejoins and takes its copies back afterward.
Restart-ZrNode -StoreName orders -Name 10.0.0.11-n0
Restart one node at a time and confirm the cluster is back to full replication (via Get-ZrNode) before restarting the next, so you never drop below the replication factor on any partition. On an RF = 1 store a restart loses that node's partitions outright — there is no replica to promote. See Restart-ZrNode.
Replace a node or machine
Replacing a machine is a scale-out followed by a scale-in. Add the replacement first, let partitions rebalance and replicas rebuild onto it, then remove the machine you are retiring. Add-ZrServer is lossless — partitions are handed to the new owner without dropping data.
Add-ZrServer -Address machine-e:7801
Remove-ZrServer -Address machine-d:7801 -Force
Scale-in is not yet drain-safe. Remove-ZrServer gracefully fails each leaving node's partitions over to a surviving replica, but shrinking the partition map evicts the resharded-away partitions rather than migrating their keys — so keys that lived only in an evicted partition can be lost. The cmdlet refuses to run without -Force for this reason (and -Force also overrides the replication-floor guard that keeps machines ≥ replication factor). Quiesce writes and make sure the data can be regenerated from your source of truth before removing a machine. Adding capacity is safe; removing it is not.
See Add-ZrServer, Remove-ZrServer, and Elastic scaling for the full scale-in caveat.