Skip to main content

16 posts tagged with "Operations"

Deploying, scaling, and running Zaris in production.

View All Tags

An Owner That Restarts Has No Right to Its Own Emptiness

· 13 min read
Clustron Team
Distributed Systems Engineering

Rebuilding a restarted owner from its replicas

A failover covers the moment an owner dies: the cluster picks a survivor and keeps serving. A planned migration covers moving a healthy owner's partition on purpose. This post is about the third act, the one that runs after the sirens stop — the original owner process restarts, rejoins the cluster, and wants its partition back. Its on-disk-nothing, in-memory-everything store came back empty. The data it is responsible for is still alive, but it is alive on the replicas that kept serving while it was down.

The whole correctness problem is contained in one sentence: the restarted owner cannot be allowed to trust its own emptiness. An empty store looks exactly like a store someone deliberately cleared. If the node assumes "I'm the owner and I have no data, therefore this partition is empty," it will happily serve NotFound for millions of live keys and, worse, let its empty state flow back onto the replicas and delete them. This post walks the five small, pure policies Zaris uses to make sure that never happens.

Two Clusters, One Change Feed: Replicating Across a WAN Without Stretching Zaris

· 10 min read
Clustron Team
Distributed Systems Engineering

Two independent clusters linked by a resumable change feed across a WAN

Sooner or later someone asks the question: we have a cluster in one region and users in another — can we just add a few nodes over there and let one cluster span both? It is a reasonable instinct. One cluster, one namespace, one connection string, data everywhere. The instinct is also a trap, and it is worth being precise about why.

A Zaris cluster is engineered for a datacenter network. Membership is tracked with heartbeats on short timers. Ownership of a partition moves between nodes through a handshake that assumes the other side answers quickly. Failover fences a suspected-dead owner and promotes a replica on the order of seconds. Every one of those mechanisms is calibrated for a link where the round trip is measured in microseconds and a "slow" node is genuinely sick — not merely eighty milliseconds and an ocean away. Put half the nodes across a WAN and you haven't built a geo-distributed cluster; you've built one cluster that now mistakes normal WAN latency for failure, flaps ownership, and risks split-brain the first time the link between regions hiccups.

So the right shape is not one stretched cluster. It is two independent clusters — each self-contained, each with its own ownership and failover staying entirely on its own LAN — joined by an asynchronous, resumable change feed. This post is about that pattern: why independence is the load-bearing decision, and how Zaris's operation log already hands you the one thing a WAN replicator actually needs — a cursor you can resume from.

The Memory Ceiling: Why a Zaris Node Never Grows Without Bound

· 9 min read
Clustron Team
Distributed Systems Engineering

A Zaris node holding memory usage in a quiet band below its ceiling, evicting at 80% down to 70% so writes never cross the limit

Here is a question that sounds simple and isn't: what happens when an in-memory store runs out of memory?

The naive answer is "it crashes" — the process climbs until the operating system kills it. The slightly-less-naive answer is "it refuses writes." Zaris does neither by default. A Zaris node is an in-memory store, but it is deliberately not an unbounded one: each node is a bounded buffer with a hard ceiling, and when the data it owns approaches that ceiling the node evicts its coldest data to make room. Writes keep succeeding; the limit holds; the process doesn't balloon. The node behaves like a cache, because underneath, it is one.

The Worker-Pool Ceiling: When a Node Plateaus at 25% CPU

· 8 min read
Clustron Team
Distributed Systems Engineering

A node pinned near 25% CPU with flat throughput because the dispatch worker pool was the ceiling

Here is a benchmark result that looks like a contradiction. A Zaris node serving a read-heavy workload tops out at a little under half a million ops/sec and refuses to climb — add more client connections, more load generators, and the number doesn't move. The obvious diagnosis is "it's saturated." Except the CPU graph says otherwise: the node is sitting at roughly a quarter of its available CPU. Three cores out of four are idle, and throughput is flat.

That is not a core-bound node, and — if you've ruled it out carefully — it's not a client-bound benchmark either. It's a third thing, and it's a classic: the request-dispatch worker pool was narrower than the work wanted to be, so requests queued while cores sat on their hands.

The Control Plane in Your Shell: Running a Zaris Cluster from PowerShell

· 11 min read
Clustron Team
Distributed Systems Engineering

Provisioning and running a Zaris cluster from PowerShell

Most of what you read about a data store is about the data path — reads, writes, the wire protocol, how fast it goes. But before any of that matters, somebody has to stand the thing up: decide how many partitions it has, how many copies of each, where the ports live, start it, confirm it came up healthy, and watch it while it runs. That's the control plane, and in Zaris it lives in PowerShell.

This is a day-one operator walkthrough. We'll create a workspace, provision a store, start it, and then read its live state back — all from the AdminShell module. No web console required, nothing hand-edited. And because the whole point of an operator story is trust, we'll be explicit about the choices New-ZrStore puts in front of you and what each one actually costs.

No Auth Server: Offline-Verifiable Tokens, Roles, and Scopes in Zaris

· 11 min read
Clustron Team
Distributed Systems Engineering

Offline-verifiable tokens, roles, and scopes in Zaris

Most ways of putting auth in front of a distributed data store end up adding a second distributed system to run it: an auth service to call, a session database to replicate, a token-introspection endpoint every node has to reach on the hot path. You secured the store and bought yourself a new availability dependency and a new source of latency.

Zaris takes a different route, and it is a deliberately boring one: one signing key per cluster, and tokens that any node can verify by itself. A token is a JWT signed with the cluster's private key; every manager and every node holds the public half and checks the signature offline — no call to an auth server, no quorum, no database lookup. Security is off by default, and a cluster with it off behaves exactly like an unsecured one. This post is about what turns on when you switch it on: the token's shape, the role-and-scope model, the two planes where enforcement actually happens, and — because pretending otherwise is how people get a false sense of security — the operational edges that are easy to get wrong.

Replication Factor Is Not Fault Tolerance: Sizing a Zaris Cluster for Durability on Kubernetes

· 8 min read
Clustron Team
Distributed Systems Engineering

Sizing a Zaris cluster for durability on Kubernetes

There is a tempting line of reasoning when you deploy a replicated store on Kubernetes: I set replicationFactor: 2, so every partition has a backup, so I survive a node loss. It feels airtight. It is also, for a surprisingly common cluster size, wrong — and wrong in the quiet way that only shows up the first time a pod actually dies.

Zaris on Kubernetes stores data in memory and makes it durable by replication, not disk: the node data volume is an emptyDir, and a rescheduled pod rebuilds from the copy held on another pod. That design is sound — but it leans entirely on there being a copy on another pod. Whether that copy exists is not decided by replicationFactor alone. It is decided by the arithmetic between your pod count and your replication factor, and if you get that arithmetic wrong the store will run perfectly, serve every read and write, and silently hold one partition that has nowhere to replicate to.

Fail Closed, Never Destructive: Inside the Zaris License Lifecycle

· 10 min read
Clustron Team
Distributed Systems Engineering

The Zaris license lifecycle

Licensing is the part of a product most engineers never want to think about, and it shows. The usual pattern is a time bomb: a date is checked, and when it passes, the software punishes you — it throws on startup, it corrupts or wipes local state to stop you "stealing" it, or it simply crashes and takes your data with it. That design treats your own operators as adversaries, and it fails in exactly the moment you can least afford it: in production, after the clock quietly rolled over.

Zaris takes the opposite stance. The license check is a distributed data store's most safety-critical switch, so it's built like one: a single pure, deterministic policy function, a graceful and fully reversible enforcement path that never destroys a byte, and one clock per machine that resumes the instant you activate a key. This post walks through how that lifecycle actually works in the shipping code — because the interesting engineering here is in what it refuses to do to you.

Retries You Don't Write: Surviving Ownership Churn in the Zaris Client

· 10 min read
Clustron Team
Distributed Systems Engineering

Transparent retry and reroute under ownership churn

Here is a fact about any partitioned key-value store that the glossy diagrams tend to skip: the node that owns your key will move. A node fails and its partitions fail over to their replicas. You scale out and partitions migrate to the new member. A preferred primary comes back and ownership fails back to it. Every one of those events means that the node your client was happily talking to a millisecond ago is, right now, the wrong node for some set of keys.

The naive client experience of that moment is ugly: a connection reset, a generic "server error," or — worse — a write that lands on a node that is no longer the active owner and quietly goes nowhere. The Zaris .NET client is built so that none of that reaches your code. An ownership change becomes a typed status, the client reroutes to the new owner, and the retry is bounded by both an attempt budget and a wall-clock deadline. In the common case your PutAsync just takes a few milliseconds longer and returns Ok. This post is about the machinery that makes that true, and — just as importantly — about the cases it deliberately does not paper over.

Clustron Zaris 2.0.1: Durable by Default, Licensed in One Line

· 5 min read
Clustron Team
Distributed Systems Engineering

Clustron Zaris 2.0.1 — durable by default, licensed in one line

Zaris 2.0.1 is out. It's a drop-in upgrade from 2.0.0 — same wire protocol, same client API, same on-disk and in-memory formats — and it does two things worth upgrading for: it makes the cluster's ownership and recovery path meaningfully more durable under failure, and it gives Zaris a clean, built-in trial-and-license model so you can evaluate it and run it in production without friction.