31 posts tagged with "Distributed Systems"

Almost every service grows a background queue. A request comes in, you do the fast part inline — validate, write a row, return 202 Accepted — and you hand the slow part to a worker: render the PDF, charge the card, send the email, re-index the document. Enqueuing is the easy half. The hard half is the promise you have quietly made to the caller: this job will actually run, even if a worker falls over in the middle of it.

That promise is where naive queues break. The usual first cut is a list: push a job on one end, have workers pop from the other. It works beautifully until a worker pops a job, begins the work, and then the process is killed — a deploy, an OOM, a node reboot. The job was removed from the list the instant it was popped, so it exists nowhere now except in the memory of a process that no longer exists. No error, no retry, no trace. The caller got their 202 and the work silently never happened.

A durable work queue needs three properties a pop-from-a-list queue does not have: a job is delivered to exactly one worker at a time, it stays owned-but-unfinished until that worker explicitly confirms completion, and if the worker never confirms, the job becomes claimable by another worker. Zaris streams with consumer groups give you all three as first-class server-side behaviour. This post builds the queue on them, verified against the shipping native .NET API.

Live features look simple on a laptop. A price moves, a game scores, an order ships — you raise an event, your WebSocket handler pushes it to the connected clients, and the screen updates. It keeps working right up until the moment there is more than one copy of your app. Then it quietly breaks, and it breaks in a way that is easy to miss in testing because a single instance always looks fine.

The reason is that an in-process notification — a C# event, an IObservable, a Channel<T> — only reaches code running inside the same process. Put four instances of your API behind a load balancer and the browser sessions spread across all four. A price tick that arrives on instance A raises an event on instance A, which pushes to the sockets instance A is holding. The three-quarters of your users connected to B, C, and D see nothing. The update is real, it just never leaves the process that heard about it.

The fix is a broker that carries the event across instances: one component publishes, every instance that holds interested clients receives, and each forwards to its own sockets. That is exactly what publish/subscribe is for, and Zaris ships it as a first-class native .NET API — IPubSubClient — with one property that makes it right for this job: a message published on any node in the cluster is delivered to subscribers connected to any node, exactly once. This post builds a live fan-out on it, verified against the shipping API.

A leaderboard looks trivial until it has to be live. Players earn points constantly, and at any instant you need to answer three questions fast: what are the top ten, what's this player's rank, and who is just above and below them. On a single process with a few hundred players you can keep a list, sort it, and move on. The moment the board is big, busy, and spread across more than one app instance, the naive shapes fall apart in two different ways.

Keep the board in each instance's memory and you have the same problem every stateful-in-process design has: four instances, four different ideas of who's in first. A score earned on instance A is invisible to instance B until something reconciles them, and nothing does. Push the board into a SQL table instead and the correctness problem goes away, but now every "top ten" is an ORDER BY score DESC LIMIT 10 over a table that's being written to on every single point scored, and every "what's my rank" is a COUNT(*) WHERE score > mine. Those queries re-sort or re-scan on each call. Under real traffic that is the most expensive thing in your system, and you end up bolting a cache in front of it that is immediately stale.

The data structure that fixes this is a sorted set: a collection of unique members, each with a score, kept permanently in score order so that rank is an index, not a computation. Redis made it famous. Zaris ships it as a first-class native .NET type — ISortedSetClient — with the ordering maintained on the server as scores change, so a point scored is one atomic increment and the top ten is one range read. This post builds a working leaderboard on it, verified against the shipping API.

Session state is the thing you forget is stateful. You call HttpContext.Session.SetString(...) to stash a cart, a wizard step, an anti-forgery nonce, a "you just added 3 items" flash message — and on a single process it just works, because the default session store is a dictionary in that process's memory. Then you add a second instance behind a load balancer, and it stops working in a way that is maddening to debug: most requests are fine, but every so often a user's cart is empty, their multi-step form resets, or they're bounced to the login page. The reason is simple once you see it — their session lives in instance A's memory, and that request landed on instance B, which has never heard of them.

The usual patch is sticky sessions: configure the load balancer to pin each client to the instance that first served them. It makes the symptom go away, but it doesn't make the state shared — it just hides the fact that it isn't. Sticky routing means your load balancing is now lopsided, a rolling deploy logs out everyone pinned to the pod being replaced, autoscaling-in drops whoever was pinned to the terminated instance, and a single crash takes every session on that node with it. You haven't made sessions durable across instances; you've made each instance a single point of failure for its own users.

The real fix is to put the session bytes somewhere every instance can see, so routing stops mattering. Zaris ships exactly that as a first-class ASP.NET Core integration — Clustron.Zaris.AspNetCore.Session — and it is one call at startup. This post shows the whole wiring, verified against the shipping package, and the one thing it does that a plain distributed cache does not: slide the idle timeout on every request.

Every API eventually needs to say no. Too many password attempts, too many writes from one tenant, too many calls against a metered upstream — at some point you have to count requests inside a time window and reject the ones over the line. On a single process that's a dictionary and a lock. The moment you run two instances behind a load balancer, it stops being that.

The naive move is to keep the limiter in each instance's memory. It's fast and it's wrong: with four instances and a limit of 100 requests a minute, a client that spreads its traffic evenly gets 400 through. You haven't built a 100/min limiter, you've built four of them.

So the state has to be shared. And the instant it's shared, the tempting shape — read the current count, add one, write it back — becomes a classic read-modify-write race. Two instances read 99, both compute 100, both write 100, both allow the request. The count is now wrong and the limit leaked. Redis users reach for INCR or a Lua script to make the step atomic. This post shows the same guarantee on Clustron Zaris with its native counter API: a server-side atomic increment, with the window's TTL applied in the same call, so the whole decision is one round trip and there is no window where two callers can disagree.

Sooner or later a service needs to react to a change rather than poll for it. A cache wants to invalidate the instant a key is overwritten. A pipeline wants to ship every mutation into a warehouse. A dashboard wants to redraw the moment state moves. Zaris gives you two mechanisms for that, and the most important thing to understand up front is that they are not two flavors of the same feature. They make opposite trade-offs, and choosing wrong is a correctness bug waiting to happen.

The Change Feed — OpenChangeFeed — is change-data-capture: it tails a durable, revision-ordered operation log, hands a consumer a resumable cursor, and guarantees at-least-once delivery of every write, including a replay from the beginning for a fresh consumer. Watch — WatchPrefixAsync — is a live keyspace notification: the owner node pushes each change to subscribers as the write lands, best-effort, live-from-now, with no backfill and no durable cursor.

One is built so you never miss an event. The other is built so you hear about events as fast as physically possible. This post walks through how each actually works under the hood, then puts both through the same 1,000,000-item workload on a four-pod AKS cluster so the trade-offs are numbers, not adjectives.

There are two honest ways to build a distributed cache client. The first is to pick any node, send it the request, and let the cluster forward it internally to whoever owns the key — simple on the client, but every request that lands on the wrong node pays for an extra network hop inside the cluster. The second is to make the client smart enough to know which node owns the key before it sends anything, so the first socket it touches is the one that can actually answer. Zaris takes the second path. A GET leaves your process already addressed to the node that holds the key — no proxy, no forwarding hop, no coordinator in the middle.

This post is the inside view of how that works: the function that turns a key into an owner, the lookup table the client keeps in memory, and — the part that's easy to get wrong — how the client stays correct when ownership moves while your traffic is in flight.

Ask a distributed store for one key and the cost is dominated by a single number: the round-trip to the node that owns it. The lookup itself is nanoseconds; the network is microseconds. Now ask for a thousand keys. If you loop and call Get a thousand times, you pay that round-trip a thousand times, serially, and the store spends almost all of the wall-clock sitting idle waiting for packets. The keys might be spread across four nodes that could have answered in parallel — but a naive loop never gives them the chance.

Zaris has three bulk APIs — GetManyAsync, PutManyAsync, DeleteManyAsync — and a mixed ExecuteBatchAsync underneath them, and their whole reason for existing is to turn "a thousand keys" into "one request per owning node, issued in parallel." This post is the fan-out path end to end: how the client buckets keys by owner, why all four APIs collapse onto a single engine, how an Index field keeps the answers in order through two layers of concurrency, and why — unlike a multi-key transaction — a bulk operation is per-item and partial success is normal.

A failover covers the moment an owner dies: the cluster picks a survivor and keeps serving. A planned migration covers moving a healthy owner's partition on purpose. This post is about the third act, the one that runs after the sirens stop — the original owner process restarts, rejoins the cluster, and wants its partition back. Its on-disk-nothing, in-memory-everything store came back empty. The data it is responsible for is still alive, but it is alive on the replicas that kept serving while it was down.

The whole correctness problem is contained in one sentence: the restarted owner cannot be allowed to trust its own emptiness. An empty store looks exactly like a store someone deliberately cleared. If the node assumes "I'm the owner and I have no data, therefore this partition is empty," it will happily serve NotFound for millions of live keys and, worse, let its empty state flow back onto the replicas and delete them. This post walks the five small, pure policies Zaris uses to make sure that never happens.

Most writing about distributed stores is about steady state — reads, writes, replication — or about the dramatic failures: a failover when an owner dies, a cut-over when a partition moves. But there's a quieter moment that decides whether any of that works: the moment the cluster comes up. Before a single key can be routed, the nodes have to agree on who is in the cluster, who owns which slice of the keyspace, and — the step most systems get wrong — whether it is safe to start serving at all.

This post follows that boot sequence in the order Zaris actually runs it: membership, the partition map, and the readiness gates that stand between "the process is up" and "the node is taking traffic." It also draws a line, honestly, around what Zaris itself implements versus what it inherits from the underlying cluster runtime it's built on.