Skip to main content

Retries You Don't Write: Surviving Ownership Churn in the Zaris Client

· 10 min read
Clustron Team
Distributed Systems Engineering

Transparent retry and reroute under ownership churn

Here is a fact about any partitioned key-value store that the glossy diagrams tend to skip: the node that owns your key will move. A node fails and its partitions fail over to their replicas. You scale out and partitions migrate to the new member. A preferred primary comes back and ownership fails back to it. Every one of those events means that the node your client was happily talking to a millisecond ago is, right now, the wrong node for some set of keys.

The naive client experience of that moment is ugly: a connection reset, a generic "server error," or — worse — a write that lands on a node that is no longer the active owner and quietly goes nowhere. The Zaris .NET client is built so that none of that reaches your code. An ownership change becomes a typed status, the client reroutes to the new owner, and the retry is bounded by both an attempt budget and a wall-clock deadline. In the common case your PutAsync just takes a few milliseconds longer and returns Ok. This post is about the machinery that makes that true, and — just as importantly — about the cases it deliberately does not paper over.

Clustron Zaris 2.0.1: Durable by Default, Licensed in One Line

· 5 min read
Clustron Team
Distributed Systems Engineering

Clustron Zaris 2.0.1 — durable by default, licensed in one line

Zaris 2.0.1 is out. It's a drop-in upgrade from 2.0.0 — same wire protocol, same client API, same on-disk and in-memory formats — and it does two things worth upgrading for: it makes the cluster's ownership and recovery path meaningfully more durable under failure, and it gives Zaris a clean, built-in trial-and-license model so you can evaluate it and run it in production without friction.

No Acknowledged Write Left Behind: The Ownership & Recovery Backbone Under 12 Hours of Chaos

· 10 min read
Clustron Team
Distributed Systems Engineering

Zero acknowledged-write loss, proven over 12 hours of chaos

The happy path of a distributed key-value store is easy to get right. A write lands on the primary, streams to the replicas, everyone agrees, you move on. If that were the whole story, durability would be a solved problem and this post wouldn't exist.

The hard problems live in the seams — the brief, awkward moments when the cluster is mid-transition. A node restarts in the middle of a partition migration. A replica falls far behind the active it's copying from, and then that active fails over to someone else. Ownership of a partition moves to a new node while writes are still in flight to the old one. Each of these is a window: a stretch of a few seconds where, if the handoff is not exactly right, an acknowledged write — one the client was told succeeded — can quietly disappear. Not because the storage engine is buggy, but because two parts of the system disagreed, for a moment, about who owned the truth.

This post is about how Zaris closes those windows, and — the part that actually matters — how we convinced ourselves it works, by running the cluster under continuous, deliberate chaos for twelve hours straight.

Alloc-Free Hot Paths: How the Zaris .NET Client Uses ValueTask

· 6 min read
Clustron Team
Distributed Systems Engineering

The Zaris .NET client uses ValueTask on its hot-path operations

Most of the time, Task<T> is exactly the right return type for an async method, and you should not think twice about it. But there's a specific place where it quietly costs you: a hot path that completes synchronously, called millions of times.

Here's the mechanism. Task<T> is a reference type. Every call that returns one allocates a Task object on the heap — even when the operation completes synchronously. A cache hit served straight from a local buffer, an awaitable that's already done: none of that requires real asynchrony, but you still pay for a heap object to carry the result. One allocation is nothing. At the call rate of a high-throughput client — a tight loop doing millions of small Gets and Puts — that's a steady stream of small heap allocations, and that stream feeds the garbage collector. More gen-0 collections, more CPU spent on GC instead of your work, more jitter in your tail latencies.

The Zaris .NET client attacks this exactly where it matters and nowhere else. This post is about that decision: which methods changed, why, the rules you have to follow to use them correctly, and — deliberately — which methods we left alone.

Big Values, Small Pauses: Large-Object Chunking in Zaris

· 6 min read
Clustron Team
Distributed Systems Engineering

Large-object chunking keeps big values off the LOH

There's a number every .NET performance engineer eventually memorizes: 85,000 bytes. Any single allocation at or above roughly that size doesn't go on the normal small-object heap — it goes on the Large Object Heap (LOH). The LOH is a different animal. It's only collected during a gen-2 (full) garbage collection, and by default it isn't compacted. That combination is fine for a big buffer you allocate once and keep. It is not fine for a low-latency in-memory store that repeatedly writes and overwrites large values.

Picture the workload: big serialized blobs, large JSON documents, images, protocol payloads — stored, updated, replaced, over and over. Every one of those large writes lands a fresh object on the LOH. Because the LOH isn't compacted, freed slots don't merge back into contiguous space; they fragment. And because reclaiming any of it means a full gen-2 collection, the pauses get longer and land more often exactly as the working set grows. For a store whose entire pitch is low latency, that's the wrong kind of surprise.

Zaris ships an opt-in feature to sidestep the LOH entirely for these values. It's off by default, and this post is about what it does, how it works, and — just as important — when you should and shouldn't turn it on.

Elastic Scaling: Adding and Removing Zaris Servers Without Downtime

· 5 min read
Clustron Team
Distributed Systems Engineering

Growing and shrinking a Zaris store live under a supervisor

Capacity is never right for long. Traffic grows, a launch lands, a season ends — and the store you sized last quarter is either straining or half-idle. The question that decides whether that's a chore or a crisis is simple: can you change the size of a running store without taking it down?

For Zaris the answer is yes, with an important asymmetry that's worth being honest about up front. Scaling out — adding servers — is the easy, safe, lossless direction, and it happens live while the store keeps serving. Scaling in — removing servers — is also supported, but it's a deliberate, monitored operation because the data on a departing server has to go somewhere first.

This post walks both directions: growing a store with Add-ZrServer, shrinking it with Remove-ZrServer, why scale-in needs a drain, and where the web console fits.

One Connection String, Any Platform: Connecting to Zaris

· 6 min read
Clustron Team
Distributed Systems Engineering

One connection string across .NET config, PowerShell, and the web console

Half of the "works in dev, breaks in prod" connection bugs come down to the same thing: the connection details were spelled three different ways in three different places. The appsettings.json had one host list, the ops runbook pasted another into a PowerShell session, and the string someone copied off a wiki had a stale token baked right into it.

Zaris takes that whole class of problem off the table with a single, platform-agnostic connection-string format. The exact same string works when your .NET app reads it from configuration, when an operator hands it to a PowerShell cmdlet, and when someone copies it straight out of the web console. One canonical spelling, everywhere.

This post breaks the string down part by part, shows all three ways to use it, and makes the case for the one rule that matters most in production: never put a raw token in the string.

Pluggable TLS Trust: Four CA Modes for Locking Down Zaris

· 7 min read
Clustron Team
Distributed Systems Engineering

Four CA modes for pluggable TLS trust in Zaris

The fastest way to make sure TLS gets skipped is to force one trust model on everyone. A dev running a store on a laptop doesn't want to stand up a CA. A bank with an offline root in an HSM doesn't want your tool minting certificates at all. Ship a store that only knows one of those, and half your users turn security off to get their work done.

So Zaris doesn't pick for you. Its TLS trust is pluggable across four certificate-authority modes, from zero-config self-signed all the way to certificates you provision entirely out of band. You choose the mode that matches the PKI you already run, and Zaris slots into it instead of asking you to rebuild around it. TLS isn't one-size-fits-all, and treating it that way is exactly why security gets deferred.

Deploying Zaris on Kubernetes With Helm: A Secure End-to-End Walkthrough

· 7 min read
Clustron Team
Distributed Systems Engineering

Deploying Zaris on Kubernetes with Helm, secured end to end

Getting a distributed store onto Kubernetes is easy. Getting it there with TLS on, reachable from outside the cluster, and a client that actually connects is where most walkthroughs quietly stop. Zaris ships a Helm chart, and we've validated the whole path — including the secured, externally-reachable configuration — end to end on a local cluster.

This post is that walkthrough. We run it on kind (Kubernetes-in-Docker) so you can reproduce every step on a laptop, and we take the harder road on purpose: TLS enabled, certificates with the right subject alternative names, external access on, and a client connecting over the secured endpoint. We call this the Model B configuration — externally-reachable and secured, versus an internal-only cluster.

One correctness rule snags almost every first deploy, and it has nothing to do with certificates: the store name in your connection string must equal the deployment's clusterId. We'll flag exactly where that bites. Everything else is identical on a production cluster — kind just gives us a clean, disposable place to prove it.

Reaching Zaris From Outside the Cluster: External Clients Without a Native SDK

· 6 min read
Clustron Team
Distributed Systems Engineering

Reaching Zaris from outside the cluster

Run Zaris in Kubernetes and the happy path is easy: in-cluster clients reach the pods by their internal addresses — pod IPs and cluster-DNS names like zaris-0.zaris.svc — and everything routes cleanly. The moment a client lives outside the cluster, that breaks. An external application cannot dial zaris-0.zaris.svc or a pod IP; those names and addresses only resolve and route inside the cluster network.

This is not a Zaris quirk — it is the fundamental split between the cluster's internal network and the outside world. It bites hardest with topology-aware clients. A cluster-mode Redis client, or any client that discovers nodes and routes per-key, fetches a topology map from the server and connects to the addresses in it. If those addresses are internal, an external client gets a map full of endpoints it can never reach.

Zaris solves this with advertised addresses, handled as a per-listener "boundary view."