Skip to main content

26 posts tagged with "Architecture"

System architecture patterns and infrastructure design.

View All Tags

Scanning a Keyspace That Won't Hold Still: How SCAN Iterates a Cluster

· 10 min read
Clustron Team
Distributed Systems Engineering

Many sorted segments merged into one ordered, paged stream

A single-key GET is easy to reason about: hash the key, find its owner, ask that one node. SCAN is the opposite kind of operation. It has no key — it asks a question about the whole keyspace: give me every key in this range, a page at a time. And in a partitioned, replicated store that keyspace isn't one thing in one place. It's split across 256 segments spread over however many nodes you run, each node holding only the slices it currently owns, and the whole set is being written to while you iterate. There is no global sorted index to walk. So where does the ordering come from, what exactly does a resume token point at, and what happens to your scan when a key is written — or a partition moves — halfway through it?

This post follows a SCAN down through the three layers that actually serve it: a single segment, a single node, and the client that stitches the nodes together. Each layer solves one piece of the problem, and the honest limits of the operation fall out of how those pieces fit.

The Memory Ceiling: Why a Zaris Node Never Grows Without Bound

· 9 min read
Clustron Team
Distributed Systems Engineering

A Zaris node holding memory usage in a quiet band below its ceiling, evicting at 80% down to 70% so writes never cross the limit

Here is a question that sounds simple and isn't: what happens when an in-memory store runs out of memory?

The naive answer is "it crashes" — the process climbs until the operating system kills it. The slightly-less-naive answer is "it refuses writes." Zaris does neither by default. A Zaris node is an in-memory store, but it is deliberately not an unbounded one: each node is a bounded buffer with a hard ceiling, and when the data it owns approaches that ceiling the node evicts its coldest data to make room. Writes keep succeeding; the limit holds; the process doesn't balloon. The node behaves like a cache, because underneath, it is one.

What Crosses the Wire: The Anatomy of a Zaris Record

· 12 min read
Clustron Team
Distributed Systems Engineering

The anatomy of a Zaris stored record

Most of what Zaris does well — replicating a write to a backup, catching up a lagging replica, cutting a partition over to a new owner — comes down to one quiet question: what, exactly, is the unit of data that moves between nodes? If that unit is right, convergence is almost boring. If it drops a field or carries the wrong one, you get a resurrected deleted key, a replica that can't tell stale from fresh, or one node's bookkeeping leaking onto another.

In Zaris that unit is a single type — StoredRecord — and it is worth taking apart field by field, because every decision in it is load-bearing. This post is the anatomy: what's in the record, how it's framed on the wire, why a delete travels as a record instead of as an absence, why there are two different version numbers, and — just as important — what is deliberately left behind on each node and never crosses the wire at all.

Drop-In Cache for ASP.NET Core: IDistributedCache and HybridCache on Zaris

· 10 min read
Clustron Team
Distributed Systems Engineering

Drop-in distributed cache for ASP.NET Core on Zaris

Most of the reasons to reach for a distributed cache in a .NET app never mention a cache by name. You add output caching and it wants an IDistributedCache. You enable sessions and the middleware wants a backing store. You turn on data-protection key sharing across instances and, again, it wants a place to put bytes that every node can see. The application code is written against an abstraction; the question is only which implementation you register at startup.

Zaris answers that question with three small NuGet packages, each targeting a specific ASP.NET Core seam: a plain IDistributedCache, a two-tier HybridCache, and a session store. You register one line at startup and the rest of your app — including third-party libraries that only know the abstraction — caches on a Zaris cluster without a single change to the code that uses the cache. This post walks through all three, and is deliberate about where each one honestly stops.

Anatomy of a Failover: What Happens to an In-Flight Write When the Owner Dies

· 12 min read
Clustron Team
Distributed Systems Engineering

Anatomy of a failover in Zaris

A failover is the moment a distributed store is most honest about what it really is. In steady state every store looks the same: a write lands on an owner, gets acknowledged, you move on. The design only shows itself when the owner disappears mid-sentence — a node is holding the active copy of one of your partitions, writes are landing on it right now, and then it's gone. Hard-killed. Network-partitioned. OOM'd. The cluster has a few seconds to notice, agree on a new owner, and get traffic flowing again — and whether it does that correctly decides whether your acknowledged write survives or quietly evaporates.

This post follows that sequence end to end, in the order Zaris actually runs it, and names the cases where the answer is genuinely "the write is lost" — because a failover post that only describes the happy path isn't worth reading.

The Same Code, With or Without a Network: Zaris Embedded Mode

· 9 min read
Clustron Team
Distributed Systems Engineering

Zaris embedded in-process mode

Most distributed data stores make you choose, early and permanently, between two worlds. There is the "real" store — a cluster you deploy, point a client at over the network, and talk to through a wire protocol. And there is the "fake" store you reach for in unit tests: an in-memory dictionary, a mock, a hand-rolled stub that implements just enough of the interface to make the test compile. The two share nothing. Your test exercises the stub's behaviour, not the store's, and the gap between them is exactly where bugs hide.

Zaris collapses that choice. The same typed .NET client that talks to a distributed cluster over TLS also runs the store embedded inside your process, with no socket, no partition map, and no deployment — and it is the same store engine, not a stand-in. You select which one you get with a single connection string. This post is about how that works, the cases it's genuinely good for, and the hard line where embedded mode stops and a cluster begins.

Which Primitive, Which Problem: A Field Guide to the Zaris Data Model

· 14 min read
Clustron Team
Distributed Systems Engineering

Choosing the right Zaris primitive for the problem

Most of this blog has gone deep on one thing at a time — compare-and-swap, multi-key transactions, native data structures, streams, watch, the op-log underneath. Each post answers "how does this work." This one answers a different, more practical question: when you have a problem in front of you, which one do you reach for?

That question matters because the wrong choice is rarely a crash. It's a system that works in the demo and then loses an update under load, or funnels a thousand writers through one key, or uses a fire-and-forget notification where it needed a durable log. The primitives don't stop you from using them wrong. So this is a field guide: the mental model that ties them together, a decision table, a walk down each branch with its honest fit and honest edge, and a flowchart at the end.

Reading Your Own Writes: The Zaris Consistency Model, Stated Honestly

· 12 min read
Clustron Team
Distributed Systems Engineering

Reading your own writes — one owner per partition

There's a question every developer eventually asks a distributed store, usually right after a confusing bug report: if I write a value and then immediately read it back, am I guaranteed to see what I wrote? It sounds like it should have a one-word answer. It doesn't — not for any partitioned, replicated store, and not for Zaris. The honest answer is "yes, within a partition, because reads go to the same single owner your write did" — and the interesting part is everything packed into that qualifier.

This post is the consistency model for Zaris, assembled in one place. We've written about the pieces separately — optimistic compare-and-swap, transparent retry under ownership churn, the replayable op-log that feeds replicas, multi-key transactions. Here we tie them together from the reader's side and answer the only question that matters at a keyboard: when I read, what am I actually promised? Including, especially, where the promise stops.

Fail Closed, Never Destructive: Inside the Zaris License Lifecycle

· 10 min read
Clustron Team
Distributed Systems Engineering

The Zaris license lifecycle

Licensing is the part of a product most engineers never want to think about, and it shows. The usual pattern is a time bomb: a date is checked, and when it passes, the software punishes you — it throws on startup, it corrupts or wipes local state to stop you "stealing" it, or it simply crashes and takes your data with it. That design treats your own operators as adversaries, and it fails in exactly the moment you can least afford it: in production, after the clock quietly rolled over.

Zaris takes the opposite stance. The license check is a distributed data store's most safety-critical switch, so it's built like one: a single pure, deterministic policy function, a graceful and fully reversible enforcement path that never destroys a byte, and one clock per machine that resumes the instant you activate a key. This post walks through how that lifecycle actually works in the shipping code — because the interesting engineering here is in what it refuses to do to you.

One Log, Two Jobs: How a Replica and a New Owner Catch Up the Same Way

· 12 min read
Clustron Team
Distributed Systems Engineering

Catch up by replaying the log — one revision-ordered op-log feeds replicas and new owners alike

Two very different-looking events in a distributed store turn out to be the same problem underneath. The first: a replica falls behind its primary — a GC pause, a slow disk, a brief network partition — and now holds a stale copy of a partition it must eventually match. The second: a partition is reassigned to a node that has never held it, and that node must acquire the entire dataset from the current owner. One is replication lag; the other is migration. They have different triggers, different operators, different failure stories.

But the question each one has to answer is identical: how do I converge to the current truth of this partition without freezing writes while I do it? Zaris answers both with a single mechanism — a revision-ordered operation log kept per segment, replayed from a known point. This post is about that log: how it's built on the write path, how it's bounded so it can't grow without limit, how it's replayed, and why the exact same substrate carries both a lagging replica and a brand-new owner to convergence.