Skip to main content

31 posts tagged with "Distributed Systems"

Articles about distributed systems architecture and design.

View All Tags

Two Clusters, One Change Feed: Replicating Across a WAN Without Stretching Zaris

· 10 min read
Clustron Team
Distributed Systems Engineering

Two independent clusters linked by a resumable change feed across a WAN

Sooner or later someone asks the question: we have a cluster in one region and users in another — can we just add a few nodes over there and let one cluster span both? It is a reasonable instinct. One cluster, one namespace, one connection string, data everywhere. The instinct is also a trap, and it is worth being precise about why.

A Zaris cluster is engineered for a datacenter network. Membership is tracked with heartbeats on short timers. Ownership of a partition moves between nodes through a handshake that assumes the other side answers quickly. Failover fences a suspected-dead owner and promotes a replica on the order of seconds. Every one of those mechanisms is calibrated for a link where the round trip is measured in microseconds and a "slow" node is genuinely sick — not merely eighty milliseconds and an ocean away. Put half the nodes across a WAN and you haven't built a geo-distributed cluster; you've built one cluster that now mistakes normal WAN latency for failure, flaps ownership, and risks split-brain the first time the link between regions hiccups.

So the right shape is not one stretched cluster. It is two independent clusters — each self-contained, each with its own ownership and failover staying entirely on its own LAN — joined by an asynchronous, resumable change feed. This post is about that pattern: why independence is the load-bearing decision, and how Zaris's operation log already hands you the one thing a WAN replicator actually needs — a cursor you can resume from.

Migrating a Partition With Zero Data Loss: The Cut-Over Protocol

· 13 min read
Clustron Team
Distributed Systems Engineering

The Zaris partition cut-over protocol

A failover is what happens when an owner dies — the cluster has seconds to pick a survivor and the partition map never moves. Migration is the calmer, more dangerous cousin: the owner is perfectly healthy, and you want to move a partition off it on purpose — to grow the cluster, to rebalance, to drain a node you're about to retire. Nothing has failed, which is exactly why the bar is higher. A failover is allowed a few honestly-lost tail writes in the worst case; a planned migration is not. If moving a partition from node A to node B can lose a single acknowledged write, migration isn't a feature, it's a liability.

This post walks the cut-over protocol Zaris actually runs, in order, and spends most of its time on the one step everyone gets wrong: the moment the source is finally "free" to drop its copy. That step is a loaded gun, and the interesting engineering is the safety on it.

Scanning a Keyspace That Won't Hold Still: How SCAN Iterates a Cluster

· 10 min read
Clustron Team
Distributed Systems Engineering

Many sorted segments merged into one ordered, paged stream

A single-key GET is easy to reason about: hash the key, find its owner, ask that one node. SCAN is the opposite kind of operation. It has no key — it asks a question about the whole keyspace: give me every key in this range, a page at a time. And in a partitioned, replicated store that keyspace isn't one thing in one place. It's split across 256 segments spread over however many nodes you run, each node holding only the slices it currently owns, and the whole set is being written to while you iterate. There is no global sorted index to walk. So where does the ordering come from, what exactly does a resume token point at, and what happens to your scan when a key is written — or a partition moves — halfway through it?

This post follows a SCAN down through the three layers that actually serve it: a single segment, a single node, and the client that stitches the nodes together. Each layer solves one piece of the problem, and the honest limits of the operation fall out of how those pieces fit.

What Crosses the Wire: The Anatomy of a Zaris Record

· 12 min read
Clustron Team
Distributed Systems Engineering

The anatomy of a Zaris stored record

Most of what Zaris does well — replicating a write to a backup, catching up a lagging replica, cutting a partition over to a new owner — comes down to one quiet question: what, exactly, is the unit of data that moves between nodes? If that unit is right, convergence is almost boring. If it drops a field or carries the wrong one, you get a resurrected deleted key, a replica that can't tell stale from fresh, or one node's bookkeeping leaking onto another.

In Zaris that unit is a single type — StoredRecord — and it is worth taking apart field by field, because every decision in it is load-bearing. This post is the anatomy: what's in the record, how it's framed on the wire, why a delete travels as a record instead of as an absence, why there are two different version numbers, and — just as important — what is deliberately left behind on each node and never crosses the wire at all.

Anatomy of a Failover: What Happens to an In-Flight Write When the Owner Dies

· 12 min read
Clustron Team
Distributed Systems Engineering

Anatomy of a failover in Zaris

A failover is the moment a distributed store is most honest about what it really is. In steady state every store looks the same: a write lands on an owner, gets acknowledged, you move on. The design only shows itself when the owner disappears mid-sentence — a node is holding the active copy of one of your partitions, writes are landing on it right now, and then it's gone. Hard-killed. Network-partitioned. OOM'd. The cluster has a few seconds to notice, agree on a new owner, and get traffic flowing again — and whether it does that correctly decides whether your acknowledged write survives or quietly evaporates.

This post follows that sequence end to end, in the order Zaris actually runs it, and names the cases where the answer is genuinely "the write is lost" — because a failover post that only describes the happy path isn't worth reading.

Reading Your Own Writes: The Zaris Consistency Model, Stated Honestly

· 12 min read
Clustron Team
Distributed Systems Engineering

Reading your own writes — one owner per partition

There's a question every developer eventually asks a distributed store, usually right after a confusing bug report: if I write a value and then immediately read it back, am I guaranteed to see what I wrote? It sounds like it should have a one-word answer. It doesn't — not for any partitioned, replicated store, and not for Zaris. The honest answer is "yes, within a partition, because reads go to the same single owner your write did" — and the interesting part is everything packed into that qualifier.

This post is the consistency model for Zaris, assembled in one place. We've written about the pieces separately — optimistic compare-and-swap, transparent retry under ownership churn, the replayable op-log that feeds replicas, multi-key transactions. Here we tie them together from the reader's side and answer the only question that matters at a keyboard: when I read, what am I actually promised? Including, especially, where the promise stops.

No Auth Server: Offline-Verifiable Tokens, Roles, and Scopes in Zaris

· 11 min read
Clustron Team
Distributed Systems Engineering

Offline-verifiable tokens, roles, and scopes in Zaris

Most ways of putting auth in front of a distributed data store end up adding a second distributed system to run it: an auth service to call, a session database to replicate, a token-introspection endpoint every node has to reach on the hot path. You secured the store and bought yourself a new availability dependency and a new source of latency.

Zaris takes a different route, and it is a deliberately boring one: one signing key per cluster, and tokens that any node can verify by itself. A token is a JWT signed with the cluster's private key; every manager and every node holds the public half and checks the signature offline — no call to an auth server, no quorum, no database lookup. Security is off by default, and a cluster with it off behaves exactly like an unsecured one. This post is about what turns on when you switch it on: the token's shape, the role-and-scope model, the two planes where enforcement actually happens, and — because pretending otherwise is how people get a false sense of security — the operational edges that are easy to get wrong.

One Log, Two Jobs: How a Replica and a New Owner Catch Up the Same Way

· 12 min read
Clustron Team
Distributed Systems Engineering

Catch up by replaying the log — one revision-ordered op-log feeds replicas and new owners alike

Two very different-looking events in a distributed store turn out to be the same problem underneath. The first: a replica falls behind its primary — a GC pause, a slow disk, a brief network partition — and now holds a stale copy of a partition it must eventually match. The second: a partition is reassigned to a node that has never held it, and that node must acquire the entire dataset from the current owner. One is replication lag; the other is migration. They have different triggers, different operators, different failure stories.

But the question each one has to answer is identical: how do I converge to the current truth of this partition without freezing writes while I do it? Zaris answers both with a single mechanism — a revision-ordered operation log kept per segment, replayed from a known point. This post is about that log: how it's built on the write path, how it's bounded so it can't grow without limit, how it's replayed, and why the exact same substrate carries both a lagging replica and a brand-new owner to convergence.

Clustron Zaris 2.0.1: Durable by Default, Licensed in One Line

· 5 min read
Clustron Team
Distributed Systems Engineering

Clustron Zaris 2.0.1 — durable by default, licensed in one line

Zaris 2.0.1 is out. It's a drop-in upgrade from 2.0.0 — same wire protocol, same client API, same on-disk and in-memory formats — and it does two things worth upgrading for: it makes the cluster's ownership and recovery path meaningfully more durable under failure, and it gives Zaris a clean, built-in trial-and-license model so you can evaluate it and run it in production without friction.

No Acknowledged Write Left Behind: The Ownership & Recovery Backbone Under 12 Hours of Chaos

· 10 min read
Clustron Team
Distributed Systems Engineering

Zero acknowledged-write loss, proven over 12 hours of chaos

The happy path of a distributed key-value store is easy to get right. A write lands on the primary, streams to the replicas, everyone agrees, you move on. If that were the whole story, durability would be a solved problem and this post wouldn't exist.

The hard problems live in the seams — the brief, awkward moments when the cluster is mid-transition. A node restarts in the middle of a partition migration. A replica falls far behind the active it's copying from, and then that active fails over to someone else. Ownership of a partition moves to a new node while writes are still in flight to the old one. Each of these is a window: a stretch of a few seconds where, if the handoff is not exactly right, an acknowledged write — one the client was told succeeded — can quietly disappear. Not because the storage engine is buggy, but because two parts of the system disagreed, for a moment, about who owned the truth.

This post is about how Zaris closes those windows, and — the part that actually matters — how we convinced ourselves it works, by running the cluster under continuous, deliberate chaos for twelve hours straight.