No Acknowledged Write Left Behind: The Ownership & Recovery Backbone Under 12 Hours of Chaos
The happy path of a distributed key-value store is easy to get right. A write lands on the primary, streams to the replicas, everyone agrees, you move on. If that were the whole story, durability would be a solved problem and this post wouldn't exist.
The hard problems live in the seams — the brief, awkward moments when the cluster is mid-transition. A node restarts in the middle of a partition migration. A replica falls far behind the active it's copying from, and then that active fails over to someone else. Ownership of a partition moves to a new node while writes are still in flight to the old one. Each of these is a window: a stretch of a few seconds where, if the handoff is not exactly right, an acknowledged write — one the client was told succeeded — can quietly disappear. Not because the storage engine is buggy, but because two parts of the system disagreed, for a moment, about who owned the truth.
This post is about how Zaris closes those windows, and — the part that actually matters — how we convinced ourselves it works, by running the cluster under continuous, deliberate chaos for twelve hours straight.