Upgrades and operational lifecycle
A Clustron Zaris cluster is a living system: you patch hosts, roll out new builds, grow and shrink capacity, and replace machines that fail — all while the store keeps serving. This article covers the lifecycle operations you perform on a running cluster and how to sequence them so you never lose availability or data.
The single fact that governs every operation here is that Zaris is an in-memory store. A node keeps its records in process memory only — there is no disk persistence, write-ahead log, or snapshot to fall back on. When a node stops, its in-memory data is gone with it. The only thing that makes stopping a node safe is a replica of its partitions living on another machine. So before you restart, remove, or replace anything, the first question is always: does every partition still have a surviving copy?
Before you touch a node
Every operation below assumes the store is replicated. Run any store whose availability matters at a replication factor of at least 2 (RF ≥ 2), so a single node can go down and each of its partitions still has a copy elsewhere. RF is set when you create the store and cannot be changed by these lifecycle operations — confirm it before you start. For the full model, see High availability and disaster recovery and Replica.
Check node and partition health with Get-ZrNode before and after each step. The list view shows every node's status, role, and partition-map version; the detailed view reports a single node's primary and replica partitions, its map version, and time since the last role change — which is how you confirm the cluster is back to full replication before moving on.
Get-ZrNode -StoreName orders
Get-ZrNode -StoreName orders -Name 10.0.0.11-n0
On an RF = 1 store, a node holds the only copy of its partitions. Stopping or restarting that node loses those partitions outright — there is no replica to promote and no disk copy to recover. Do not perform any of the operations below on an RF = 1 store unless you can regenerate the data from your system of record. Bring the store to RF ≥ 2 first, or accept the loss deliberately.
Restart a single node
To restart a node — for a configuration change, an OS patch, or to clear a stuck process — use Restart-ZrNode. It is implemented as a stop followed by a start (there is no dedicated restart route), and it reports a per-node result.
Restart-ZrNode -StoreName orders -Name 10.0.0.11-n0
On a replicated store, restarting one node triggers a failover of any partition it owned as primary — a replica is promoted, and clients re-route automatically after a map refresh. When the node comes back it rejoins the cluster and takes its copies back.
Do this one node at a time. After each restart, use Get-ZrNode to confirm the affected partitions are back to full replication before you touch the next node. Restarting a second node while the first is still rebuilding its replicas can drop a partition below its replication factor — or below its last surviving copy. This one-at-a-time, wait-for-readiness sequence is the rolling-restart pattern you reuse for the rolling upgrade below.
To stop or start a node as separate steps rather than in one call, use Stop-ZrNode and Start-ZrNode. A single-node stop defaults to a graceful drain and cluster-leave so surviving peers get a clean handoff; -Force hard-kills the process instead.
Add capacity
To grow the cluster, add a Server — a whole machine, which is the unit of scaling in supervisor mode — with Add-ZrServer. The new machine takes on the same share of every store, the partition map expands, and partitions redistribute onto it.
Add-ZrServer -Address machine-d:7801
Scale-out is lossless: partitions are handed to the new owner without dropping data, and clients transparently pick up the new partition map. Adding a Server increases both capacity (more memory across the cluster) and throughput (more nodes serving in parallel). The new machine must be blank, reachable, and not already claimed by another workspace. See Elastic scaling for how rebalancing works, and note that in attach mode you scale with your orchestrator instead.
Remove a node
Removing a Server with Remove-ZrServer gracefully stops the leaving machine's nodes — so each partition fails over to a surviving replica — drops the machine from every store's definition, and shrinks the partition map.
Remove-ZrServer -Address machine-d:7801 -Force
Scale-in is not yet drain-safe, and it is lossy by default. Shrinking the partition map evicts the partitions that are resharded away rather than migrating their keys to the remaining nodes, so keys that lived only in an evicted partition can be lost. For this reason Remove-ZrServer refuses to run without -Force. -Force also overrides the replication-floor guard that keeps the number of machines at or above a store's replication factor, so it acknowledges reduced fault tolerance as well. Quiesce writes and make sure the affected stores can be regenerated from your source of truth before you remove a Server. Adding capacity is safe; removing it is not.
Replace a failed node
Replacing a machine is a scale-out followed by a scale-in. Add the replacement first, let partitions rebalance and replicas rebuild onto it, then remove the machine you are retiring. Because Add-ZrServer is lossless, doing it in this order means the cluster re-replicates onto the new machine before anything leaves — so you are not relying on the lossy scale-in path to preserve data.
Add-ZrServer -Address machine-e:7801
# wait for rebalancing and re-replication to settle (Get-ZrNode)
Remove-ZrServer -Address machine-d:7801 -Force
If the machine you are replacing has already failed outright, its partitions have already failed over to their surviving replicas and the cluster has begun rebuilding fresh replicas on healthy machines (see High availability and disaster recovery). Add the replacement to restore full capacity, confirm re-replication with Get-ZrNode, then remove the dead machine's roster entry so the workspace no longer expects it.
Rolling upgrade
To roll out a new Zaris build across the cluster, upgrade one node at a time behind RF ≥ 2, reusing the rolling-restart discipline from above:
- Confirm the store is at full replication with
Get-ZrNode. - Take one node down —
Restart-ZrNodeif the new binaries are already staged on that host, orStop-ZrNode/ patch /Start-ZrNodeif you are swapping the build out of band. - Wait for the node to rejoin and for its partitions to return to full replication before moving to the next node.
- Repeat until every node runs the new build.
Because only one node is ever down at a time and every partition keeps a copy on another machine, the store stays available throughout, and any primary that was on the upgrading node simply fails over and back.
Mind wire and version compatibility while a rolling upgrade is in progress: the cluster is temporarily running mixed versions, and not every version pair is guaranteed to interoperate on the wire. Check the compatibility matrix in the Versioning and compatibility reference before you start, so you know whether the old and new builds can coexist in one cluster. Keep the mixed-version window as short as practical — complete the roll rather than parking the cluster on two versions — and do not run mixed versions longer than the upgrade itself requires.