Replication Factor Is Not Fault Tolerance: Sizing a Zaris Cluster for Durability on Kubernetes
There is a tempting line of reasoning when you deploy a replicated store on Kubernetes: I set replicationFactor: 2, so every partition has a backup, so I survive a node loss. It feels airtight. It is also, for a surprisingly common cluster size, wrong — and wrong in the quiet way that only shows up the first time a pod actually dies.
Zaris on Kubernetes stores data in memory and makes it durable by replication, not disk: the node data volume is an emptyDir, and a rescheduled pod rebuilds from the copy held on another pod. That design is sound — but it leans entirely on there being a copy on another pod. Whether that copy exists is not decided by replicationFactor alone. It is decided by the arithmetic between your pod count and your replication factor, and if you get that arithmetic wrong the store will run perfectly, serve every read and write, and silently hold one partition that has nowhere to replicate to.