24 posts tagged with ".NET"

Almost every service grows a background queue. A request comes in, you do the fast part inline — validate, write a row, return 202 Accepted — and you hand the slow part to a worker: render the PDF, charge the card, send the email, re-index the document. Enqueuing is the easy half. The hard half is the promise you have quietly made to the caller: this job will actually run, even if a worker falls over in the middle of it.

That promise is where naive queues break. The usual first cut is a list: push a job on one end, have workers pop from the other. It works beautifully until a worker pops a job, begins the work, and then the process is killed — a deploy, an OOM, a node reboot. The job was removed from the list the instant it was popped, so it exists nowhere now except in the memory of a process that no longer exists. No error, no retry, no trace. The caller got their 202 and the work silently never happened.

A durable work queue needs three properties a pop-from-a-list queue does not have: a job is delivered to exactly one worker at a time, it stays owned-but-unfinished until that worker explicitly confirms completion, and if the worker never confirms, the job becomes claimable by another worker. Zaris streams with consumer groups give you all three as first-class server-side behaviour. This post builds the queue on them, verified against the shipping native .NET API.

Live features look simple on a laptop. A price moves, a game scores, an order ships — you raise an event, your WebSocket handler pushes it to the connected clients, and the screen updates. It keeps working right up until the moment there is more than one copy of your app. Then it quietly breaks, and it breaks in a way that is easy to miss in testing because a single instance always looks fine.

The reason is that an in-process notification — a C# event, an IObservable, a Channel<T> — only reaches code running inside the same process. Put four instances of your API behind a load balancer and the browser sessions spread across all four. A price tick that arrives on instance A raises an event on instance A, which pushes to the sockets instance A is holding. The three-quarters of your users connected to B, C, and D see nothing. The update is real, it just never leaves the process that heard about it.

The fix is a broker that carries the event across instances: one component publishes, every instance that holds interested clients receives, and each forwards to its own sockets. That is exactly what publish/subscribe is for, and Zaris ships it as a first-class native .NET API — IPubSubClient — with one property that makes it right for this job: a message published on any node in the cluster is delivered to subscribers connected to any node, exactly once. This post builds a live fan-out on it, verified against the shipping API.

A leaderboard looks trivial until it has to be live. Players earn points constantly, and at any instant you need to answer three questions fast: what are the top ten, what's this player's rank, and who is just above and below them. On a single process with a few hundred players you can keep a list, sort it, and move on. The moment the board is big, busy, and spread across more than one app instance, the naive shapes fall apart in two different ways.

Keep the board in each instance's memory and you have the same problem every stateful-in-process design has: four instances, four different ideas of who's in first. A score earned on instance A is invisible to instance B until something reconciles them, and nothing does. Push the board into a SQL table instead and the correctness problem goes away, but now every "top ten" is an ORDER BY score DESC LIMIT 10 over a table that's being written to on every single point scored, and every "what's my rank" is a COUNT(*) WHERE score > mine. Those queries re-sort or re-scan on each call. Under real traffic that is the most expensive thing in your system, and you end up bolting a cache in front of it that is immediately stale.

The data structure that fixes this is a sorted set: a collection of unique members, each with a score, kept permanently in score order so that rank is an index, not a computation. Redis made it famous. Zaris ships it as a first-class native .NET type — ISortedSetClient — with the ordering maintained on the server as scores change, so a point scored is one atomic increment and the top ten is one range read. This post builds a working leaderboard on it, verified against the shipping API.

Session state is the thing you forget is stateful. You call HttpContext.Session.SetString(...) to stash a cart, a wizard step, an anti-forgery nonce, a "you just added 3 items" flash message — and on a single process it just works, because the default session store is a dictionary in that process's memory. Then you add a second instance behind a load balancer, and it stops working in a way that is maddening to debug: most requests are fine, but every so often a user's cart is empty, their multi-step form resets, or they're bounced to the login page. The reason is simple once you see it — their session lives in instance A's memory, and that request landed on instance B, which has never heard of them.

The usual patch is sticky sessions: configure the load balancer to pin each client to the instance that first served them. It makes the symptom go away, but it doesn't make the state shared — it just hides the fact that it isn't. Sticky routing means your load balancing is now lopsided, a rolling deploy logs out everyone pinned to the pod being replaced, autoscaling-in drops whoever was pinned to the terminated instance, and a single crash takes every session on that node with it. You haven't made sessions durable across instances; you've made each instance a single point of failure for its own users.

The real fix is to put the session bytes somewhere every instance can see, so routing stops mattering. Zaris ships exactly that as a first-class ASP.NET Core integration — Clustron.Zaris.AspNetCore.Session — and it is one call at startup. This post shows the whole wiring, verified against the shipping package, and the one thing it does that a plain distributed cache does not: slide the idle timeout on every request.

Every API eventually needs to say no. Too many password attempts, too many writes from one tenant, too many calls against a metered upstream — at some point you have to count requests inside a time window and reject the ones over the line. On a single process that's a dictionary and a lock. The moment you run two instances behind a load balancer, it stops being that.

The naive move is to keep the limiter in each instance's memory. It's fast and it's wrong: with four instances and a limit of 100 requests a minute, a client that spreads its traffic evenly gets 400 through. You haven't built a 100/min limiter, you've built four of them.

So the state has to be shared. And the instant it's shared, the tempting shape — read the current count, add one, write it back — becomes a classic read-modify-write race. Two instances read 99, both compute 100, both write 100, both allow the request. The count is now wrong and the limit leaked. Redis users reach for INCR or a Lua script to make the step atomic. This post shows the same guarantee on Clustron Zaris with its native counter API: a server-side atomic increment, with the window's TTL applied in the same call, so the whole decision is one round trip and there is no window where two callers can disagree.

There are two honest ways to build a distributed cache client. The first is to pick any node, send it the request, and let the cluster forward it internally to whoever owns the key — simple on the client, but every request that lands on the wrong node pays for an extra network hop inside the cluster. The second is to make the client smart enough to know which node owns the key before it sends anything, so the first socket it touches is the one that can actually answer. Zaris takes the second path. A GET leaves your process already addressed to the node that holds the key — no proxy, no forwarding hop, no coordinator in the middle.

This post is the inside view of how that works: the function that turns a key into an owner, the lookup table the client keeps in memory, and — the part that's easy to get wrong — how the client stays correct when ownership moves while your traffic is in flight.

Here is a number that reframes how you think about a key/value store. When we isolate the Zaris in-memory store and feed it operations with nothing else in the way — no socket, no serialization, no routing — a single thread serves reads at over a million per second, and scales cleanly to more than eight million across eight threads. The store, in other words, does the actual work of a GET in under two microseconds, and it is emphatically not the bottleneck.

And yet a full GET over a real TCP connection, on the same machine, plateaus at a few hundred thousand per second. The store can do the work five times faster than the server can deliver it. So the interesting performance question about Zaris is not "how fast is the store?" It's the opposite: where does all the time between the socket and the store go? This post is the answer, traced operation by operation, because the gap between 1.7 microseconds of logic and 10-to-13 microseconds of server CPU per operation is not noise — it's the latency floor, and almost all of it is plumbing.

Here is a question that sounds simple and isn't: what happens when an in-memory store runs out of memory?

The naive answer is "it crashes" — the process climbs until the operating system kills it. The slightly-less-naive answer is "it refuses writes." Zaris does neither by default. A Zaris node is an in-memory store, but it is deliberately not an unbounded one: each node is a bounded buffer with a hard ceiling, and when the data it owns approaches that ceiling the node evicts its coldest data to make room. Writes keep succeeding; the limit holds; the process doesn't balloon. The node behaves like a cache, because underneath, it is one.

Here is a benchmark result that looks like a contradiction. A Zaris node serving a read-heavy workload tops out at a little under half a million ops/sec and refuses to climb — add more client connections, more load generators, and the number doesn't move. The obvious diagnosis is "it's saturated." Except the CPU graph says otherwise: the node is sitting at roughly a quarter of its available CPU. Three cores out of four are idle, and throughput is flat.

That is not a core-bound node, and — if you've ruled it out carefully — it's not a client-bound benchmark either. It's a third thing, and it's a classic: the request-dispatch worker pool was narrower than the work wanted to be, so requests queued while cores sat on their hands.

Most of the reasons to reach for a distributed cache in a .NET app never mention a cache by name. You add output caching and it wants an IDistributedCache. You enable sessions and the middleware wants a backing store. You turn on data-protection key sharing across instances and, again, it wants a place to put bytes that every node can see. The application code is written against an abstraction; the question is only which implementation you register at startup.

Zaris answers that question with three small NuGet packages, each targeting a specific ASP.NET Core seam: a plain IDistributedCache, a two-tier HybridCache, and a session store. You register one line at startup and the rest of your app — including third-party libraries that only know the abstraction — caches on a Zaris cluster without a single change to the code that uses the cache. This post walks through all three, and is deliberate about where each one honestly stops.