Retries You Don't Write: Surviving Ownership Churn in the Zaris Client
Here is a fact about any partitioned key-value store that the glossy diagrams tend to skip: the node that owns your key will move. A node fails and its partitions fail over to their replicas. You scale out and partitions migrate to the new member. A preferred primary comes back and ownership fails back to it. Every one of those events means that the node your client was happily talking to a millisecond ago is, right now, the wrong node for some set of keys.
The naive client experience of that moment is ugly: a connection reset, a generic "server error," or — worse — a write that lands on a node that is no longer the active owner and quietly goes nowhere. The Zaris .NET client is built so that none of that reaches your code. An ownership change becomes a typed status, the client reroutes to the new owner, and the retry is bounded by both an attempt budget and a wall-clock deadline. In the common case your PutAsync just takes a few milliseconds longer and returns Ok. This post is about the machinery that makes that true, and — just as importantly — about the cases it deliberately does not paper over.