26 posts tagged with "Architecture"

Live features look simple on a laptop. A price moves, a game scores, an order ships — you raise an event, your WebSocket handler pushes it to the connected clients, and the screen updates. It keeps working right up until the moment there is more than one copy of your app. Then it quietly breaks, and it breaks in a way that is easy to miss in testing because a single instance always looks fine.

The reason is that an in-process notification — a C# event, an IObservable, a Channel<T> — only reaches code running inside the same process. Put four instances of your API behind a load balancer and the browser sessions spread across all four. A price tick that arrives on instance A raises an event on instance A, which pushes to the sockets instance A is holding. The three-quarters of your users connected to B, C, and D see nothing. The update is real, it just never leaves the process that heard about it.

The fix is a broker that carries the event across instances: one component publishes, every instance that holds interested clients receives, and each forwards to its own sockets. That is exactly what publish/subscribe is for, and Zaris ships it as a first-class native .NET API — IPubSubClient — with one property that makes it right for this job: a message published on any node in the cluster is delivered to subscribers connected to any node, exactly once. This post builds a live fan-out on it, verified against the shipping API.

A leaderboard looks trivial until it has to be live. Players earn points constantly, and at any instant you need to answer three questions fast: what are the top ten, what's this player's rank, and who is just above and below them. On a single process with a few hundred players you can keep a list, sort it, and move on. The moment the board is big, busy, and spread across more than one app instance, the naive shapes fall apart in two different ways.

Keep the board in each instance's memory and you have the same problem every stateful-in-process design has: four instances, four different ideas of who's in first. A score earned on instance A is invisible to instance B until something reconciles them, and nothing does. Push the board into a SQL table instead and the correctness problem goes away, but now every "top ten" is an ORDER BY score DESC LIMIT 10 over a table that's being written to on every single point scored, and every "what's my rank" is a COUNT(*) WHERE score > mine. Those queries re-sort or re-scan on each call. Under real traffic that is the most expensive thing in your system, and you end up bolting a cache in front of it that is immediately stale.

The data structure that fixes this is a sorted set: a collection of unique members, each with a score, kept permanently in score order so that rank is an index, not a computation. Redis made it famous. Zaris ships it as a first-class native .NET type — ISortedSetClient — with the ordering maintained on the server as scores change, so a point scored is one atomic increment and the top ten is one range read. This post builds a working leaderboard on it, verified against the shipping API.

Session state is the thing you forget is stateful. You call HttpContext.Session.SetString(...) to stash a cart, a wizard step, an anti-forgery nonce, a "you just added 3 items" flash message — and on a single process it just works, because the default session store is a dictionary in that process's memory. Then you add a second instance behind a load balancer, and it stops working in a way that is maddening to debug: most requests are fine, but every so often a user's cart is empty, their multi-step form resets, or they're bounced to the login page. The reason is simple once you see it — their session lives in instance A's memory, and that request landed on instance B, which has never heard of them.

The usual patch is sticky sessions: configure the load balancer to pin each client to the instance that first served them. It makes the symptom go away, but it doesn't make the state shared — it just hides the fact that it isn't. Sticky routing means your load balancing is now lopsided, a rolling deploy logs out everyone pinned to the pod being replaced, autoscaling-in drops whoever was pinned to the terminated instance, and a single crash takes every session on that node with it. You haven't made sessions durable across instances; you've made each instance a single point of failure for its own users.

The real fix is to put the session bytes somewhere every instance can see, so routing stops mattering. Zaris ships exactly that as a first-class ASP.NET Core integration — Clustron.Zaris.AspNetCore.Session — and it is one call at startup. This post shows the whole wiring, verified against the shipping package, and the one thing it does that a plain distributed cache does not: slide the idle timeout on every request.

Every API eventually needs to say no. Too many password attempts, too many writes from one tenant, too many calls against a metered upstream — at some point you have to count requests inside a time window and reject the ones over the line. On a single process that's a dictionary and a lock. The moment you run two instances behind a load balancer, it stops being that.

The naive move is to keep the limiter in each instance's memory. It's fast and it's wrong: with four instances and a limit of 100 requests a minute, a client that spreads its traffic evenly gets 400 through. You haven't built a 100/min limiter, you've built four of them.

So the state has to be shared. And the instant it's shared, the tempting shape — read the current count, add one, write it back — becomes a classic read-modify-write race. Two instances read 99, both compute 100, both write 100, both allow the request. The count is now wrong and the limit leaked. Redis users reach for INCR or a Lua script to make the step atomic. This post shows the same guarantee on Clustron Zaris with its native counter API: a server-side atomic increment, with the window's TTL applied in the same call, so the whole decision is one round trip and there is no window where two callers can disagree.

Sooner or later a service needs to react to a change rather than poll for it. A cache wants to invalidate the instant a key is overwritten. A pipeline wants to ship every mutation into a warehouse. A dashboard wants to redraw the moment state moves. Zaris gives you two mechanisms for that, and the most important thing to understand up front is that they are not two flavors of the same feature. They make opposite trade-offs, and choosing wrong is a correctness bug waiting to happen.

The Change Feed — OpenChangeFeed — is change-data-capture: it tails a durable, revision-ordered operation log, hands a consumer a resumable cursor, and guarantees at-least-once delivery of every write, including a replay from the beginning for a fresh consumer. Watch — WatchPrefixAsync — is a live keyspace notification: the owner node pushes each change to subscribers as the write lands, best-effort, live-from-now, with no backfill and no durable cursor.

One is built so you never miss an event. The other is built so you hear about events as fast as physically possible. This post walks through how each actually works under the hood, then puts both through the same 1,000,000-item workload on a four-pod AKS cluster so the trade-offs are numbers, not adjectives.

A failover covers the moment an owner dies: the cluster picks a survivor and keeps serving. A planned migration covers moving a healthy owner's partition on purpose. This post is about the third act, the one that runs after the sirens stop — the original owner process restarts, rejoins the cluster, and wants its partition back. Its on-disk-nothing, in-memory-everything store came back empty. The data it is responsible for is still alive, but it is alive on the replicas that kept serving while it was down.

The whole correctness problem is contained in one sentence: the restarted owner cannot be allowed to trust its own emptiness. An empty store looks exactly like a store someone deliberately cleared. If the node assumes "I'm the owner and I have no data, therefore this partition is empty," it will happily serve NotFound for millions of live keys and, worse, let its empty state flow back onto the replicas and delete them. This post walks the five small, pure policies Zaris uses to make sure that never happens.

Most writing about distributed stores is about steady state — reads, writes, replication — or about the dramatic failures: a failover when an owner dies, a cut-over when a partition moves. But there's a quieter moment that decides whether any of that works: the moment the cluster comes up. Before a single key can be routed, the nodes have to agree on who is in the cluster, who owns which slice of the keyspace, and — the step most systems get wrong — whether it is safe to start serving at all.

This post follows that boot sequence in the order Zaris actually runs it: membership, the partition map, and the readiness gates that stand between "the process is up" and "the node is taking traffic." It also draws a line, honestly, around what Zaris itself implements versus what it inherits from the underlying cluster runtime it's built on.

Here is a number that reframes how you think about a key/value store. When we isolate the Zaris in-memory store and feed it operations with nothing else in the way — no socket, no serialization, no routing — a single thread serves reads at over a million per second, and scales cleanly to more than eight million across eight threads. The store, in other words, does the actual work of a GET in under two microseconds, and it is emphatically not the bottleneck.

And yet a full GET over a real TCP connection, on the same machine, plateaus at a few hundred thousand per second. The store can do the work five times faster than the server can deliver it. So the interesting performance question about Zaris is not "how fast is the store?" It's the opposite: where does all the time between the socket and the store go? This post is the answer, traced operation by operation, because the gap between 1.7 microseconds of logic and 10-to-13 microseconds of server CPU per operation is not noise — it's the latency floor, and almost all of it is plumbing.

Sooner or later someone asks the question: we have a cluster in one region and users in another — can we just add a few nodes over there and let one cluster span both? It is a reasonable instinct. One cluster, one namespace, one connection string, data everywhere. The instinct is also a trap, and it is worth being precise about why.

A Zaris cluster is engineered for a datacenter network. Membership is tracked with heartbeats on short timers. Ownership of a partition moves between nodes through a handshake that assumes the other side answers quickly. Failover fences a suspected-dead owner and promotes a replica on the order of seconds. Every one of those mechanisms is calibrated for a link where the round trip is measured in microseconds and a "slow" node is genuinely sick — not merely eighty milliseconds and an ocean away. Put half the nodes across a WAN and you haven't built a geo-distributed cluster; you've built one cluster that now mistakes normal WAN latency for failure, flaps ownership, and risks split-brain the first time the link between regions hiccups.

So the right shape is not one stretched cluster. It is two independent clusters — each self-contained, each with its own ownership and failover staying entirely on its own LAN — joined by an asynchronous, resumable change feed. This post is about that pattern: why independence is the load-bearing decision, and how Zaris's operation log already hands you the one thing a WAN replicator actually needs — a cursor you can resume from.

A single-key GET is easy to reason about: hash the key, find its owner, ask that one node. SCAN is the opposite kind of operation. It has no key — it asks a question about the whole keyspace: give me every key in this range, a page at a time. And in a partitioned, replicated store that keyspace isn't one thing in one place. It's split across 256 segments spread over however many nodes you run, each node holding only the slices it currently owns, and the whole set is being written to while you iterate. There is no global sorted index to walk. So where does the ordering come from, what exactly does a resume token point at, and what happens to your scan when a key is written — or a partition moves — halfway through it?

This post follows a SCAN down through the three layers that actually serve it: a single segment, a single node, and the client that stitches the nodes together. Each layer solves one piece of the problem, and the honest limits of the operation fall out of how those pieces fit.