Skip to main content

Two Clusters, One Change Feed: Replicating Across a WAN Without Stretching Zaris

· 10 min read
Clustron Team
Distributed Systems Engineering

Two independent clusters linked by a resumable change feed across a WAN

Sooner or later someone asks the question: we have a cluster in one region and users in another — can we just add a few nodes over there and let one cluster span both? It is a reasonable instinct. One cluster, one namespace, one connection string, data everywhere. The instinct is also a trap, and it is worth being precise about why.

A Zaris cluster is engineered for a datacenter network. Membership is tracked with heartbeats on short timers. Ownership of a partition moves between nodes through a handshake that assumes the other side answers quickly. Failover fences a suspected-dead owner and promotes a replica on the order of seconds. Every one of those mechanisms is calibrated for a link where the round trip is measured in microseconds and a "slow" node is genuinely sick — not merely eighty milliseconds and an ocean away. Put half the nodes across a WAN and you haven't built a geo-distributed cluster; you've built one cluster that now mistakes normal WAN latency for failure, flaps ownership, and risks split-brain the first time the link between regions hiccups.

So the right shape is not one stretched cluster. It is two independent clusters — each self-contained, each with its own ownership and failover staying entirely on its own LAN — joined by an asynchronous, resumable change feed. This post is about that pattern: why independence is the load-bearing decision, and how Zaris's operation log already hands you the one thing a WAN replicator actually needs — a cursor you can resume from.

Migrating a Partition With Zero Data Loss: The Cut-Over Protocol

· 13 min read
Clustron Team
Distributed Systems Engineering

The Zaris partition cut-over protocol

A failover is what happens when an owner dies — the cluster has seconds to pick a survivor and the partition map never moves. Migration is the calmer, more dangerous cousin: the owner is perfectly healthy, and you want to move a partition off it on purpose — to grow the cluster, to rebalance, to drain a node you're about to retire. Nothing has failed, which is exactly why the bar is higher. A failover is allowed a few honestly-lost tail writes in the worst case; a planned migration is not. If moving a partition from node A to node B can lose a single acknowledged write, migration isn't a feature, it's a liability.

This post walks the cut-over protocol Zaris actually runs, in order, and spends most of its time on the one step everyone gets wrong: the moment the source is finally "free" to drop its copy. That step is a loaded gun, and the interesting engineering is the safety on it.

The Latency Floor: One Microsecond of Work Inside Ten of Plumbing

· 11 min read
Clustron Team
Distributed Systems Engineering

One microsecond of work inside ten microseconds of plumbing

Here is a number that reframes how you think about a key/value store. When we isolate the Zaris in-memory store and feed it operations with nothing else in the way — no socket, no serialization, no routing — a single thread serves reads at over a million per second, and scales cleanly to more than eight million across eight threads. The store, in other words, does the actual work of a GET in under two microseconds, and it is emphatically not the bottleneck.

And yet a full GET over a real TCP connection, on the same machine, plateaus at a few hundred thousand per second. The store can do the work five times faster than the server can deliver it. So the interesting performance question about Zaris is not "how fast is the store?" It's the opposite: where does all the time between the socket and the store go? This post is the answer, traced operation by operation, because the gap between 1.7 microseconds of logic and 10-to-13 microseconds of server CPU per operation is not noise — it's the latency floor, and almost all of it is plumbing.

Scanning a Keyspace That Won't Hold Still: How SCAN Iterates a Cluster

· 10 min read
Clustron Team
Distributed Systems Engineering

Many sorted segments merged into one ordered, paged stream

A single-key GET is easy to reason about: hash the key, find its owner, ask that one node. SCAN is the opposite kind of operation. It has no key — it asks a question about the whole keyspace: give me every key in this range, a page at a time. And in a partitioned, replicated store that keyspace isn't one thing in one place. It's split across 256 segments spread over however many nodes you run, each node holding only the slices it currently owns, and the whole set is being written to while you iterate. There is no global sorted index to walk. So where does the ordering come from, what exactly does a resume token point at, and what happens to your scan when a key is written — or a partition moves — halfway through it?

This post follows a SCAN down through the three layers that actually serve it: a single segment, a single node, and the client that stitches the nodes together. Each layer solves one piece of the problem, and the honest limits of the operation fall out of how those pieces fit.

The Memory Ceiling: Why a Zaris Node Never Grows Without Bound

· 9 min read
Clustron Team
Distributed Systems Engineering

A Zaris node holding memory usage in a quiet band below its ceiling, evicting at 80% down to 70% so writes never cross the limit

Here is a question that sounds simple and isn't: what happens when an in-memory store runs out of memory?

The naive answer is "it crashes" — the process climbs until the operating system kills it. The slightly-less-naive answer is "it refuses writes." Zaris does neither by default. A Zaris node is an in-memory store, but it is deliberately not an unbounded one: each node is a bounded buffer with a hard ceiling, and when the data it owns approaches that ceiling the node evicts its coldest data to make room. Writes keep succeeding; the limit holds; the process doesn't balloon. The node behaves like a cache, because underneath, it is one.

What Crosses the Wire: The Anatomy of a Zaris Record

· 12 min read
Clustron Team
Distributed Systems Engineering

The anatomy of a Zaris stored record

Most of what Zaris does well — replicating a write to a backup, catching up a lagging replica, cutting a partition over to a new owner — comes down to one quiet question: what, exactly, is the unit of data that moves between nodes? If that unit is right, convergence is almost boring. If it drops a field or carries the wrong one, you get a resurrected deleted key, a replica that can't tell stale from fresh, or one node's bookkeeping leaking onto another.

In Zaris that unit is a single type — StoredRecord — and it is worth taking apart field by field, because every decision in it is load-bearing. This post is the anatomy: what's in the record, how it's framed on the wire, why a delete travels as a record instead of as an absence, why there are two different version numbers, and — just as important — what is deliberately left behind on each node and never crosses the wire at all.

The Worker-Pool Ceiling: When a Node Plateaus at 25% CPU

· 8 min read
Clustron Team
Distributed Systems Engineering

A node pinned near 25% CPU with flat throughput because the dispatch worker pool was the ceiling

Here is a benchmark result that looks like a contradiction. A Zaris node serving a read-heavy workload tops out at a little under half a million ops/sec and refuses to climb — add more client connections, more load generators, and the number doesn't move. The obvious diagnosis is "it's saturated." Except the CPU graph says otherwise: the node is sitting at roughly a quarter of its available CPU. Three cores out of four are idle, and throughput is flat.

That is not a core-bound node, and — if you've ruled it out carefully — it's not a client-bound benchmark either. It's a third thing, and it's a classic: the request-dispatch worker pool was narrower than the work wanted to be, so requests queued while cores sat on their hands.

Drop-In Cache for ASP.NET Core: IDistributedCache and HybridCache on Zaris

· 10 min read
Clustron Team
Distributed Systems Engineering

Drop-in distributed cache for ASP.NET Core on Zaris

Most of the reasons to reach for a distributed cache in a .NET app never mention a cache by name. You add output caching and it wants an IDistributedCache. You enable sessions and the middleware wants a backing store. You turn on data-protection key sharing across instances and, again, it wants a place to put bytes that every node can see. The application code is written against an abstraction; the question is only which implementation you register at startup.

Zaris answers that question with three small NuGet packages, each targeting a specific ASP.NET Core seam: a plain IDistributedCache, a two-tier HybridCache, and a session store. You register one line at startup and the rest of your app — including third-party libraries that only know the abstraction — caches on a Zaris cluster without a single change to the code that uses the cache. This post walks through all three, and is deliberate about where each one honestly stops.

Anatomy of a Failover: What Happens to an In-Flight Write When the Owner Dies

· 12 min read
Clustron Team
Distributed Systems Engineering

Anatomy of a failover in Zaris

A failover is the moment a distributed store is most honest about what it really is. In steady state every store looks the same: a write lands on an owner, gets acknowledged, you move on. The design only shows itself when the owner disappears mid-sentence — a node is holding the active copy of one of your partitions, writes are landing on it right now, and then it's gone. Hard-killed. Network-partitioned. OOM'd. The cluster has a few seconds to notice, agree on a new owner, and get traffic flowing again — and whether it does that correctly decides whether your acknowledged write survives or quietly evaporates.

This post follows that sequence end to end, in the order Zaris actually runs it, and names the cases where the answer is genuinely "the write is lost" — because a failover post that only describes the happy path isn't worth reading.

The Same Code, With or Without a Network: Zaris Embedded Mode

· 9 min read
Clustron Team
Distributed Systems Engineering

Zaris embedded in-process mode

Most distributed data stores make you choose, early and permanently, between two worlds. There is the "real" store — a cluster you deploy, point a client at over the network, and talk to through a wire protocol. And there is the "fake" store you reach for in unit tests: an in-memory dictionary, a mock, a hand-rolled stub that implements just enough of the interface to make the test compile. The two share nothing. Your test exercises the stub's behaviour, not the store's, and the gap between them is exactly where bugs hide.

Zaris collapses that choice. The same typed .NET client that talks to a distributed cluster over TLS also runs the store embedded inside your process, with no socket, no partition map, and no deployment — and it is the same store engine, not a stand-in. You select which one you get with a single connection string. This post is about how that works, the cases it's genuinely good for, and the hard line where embedded mode stops and a cluster begins.