Honest membership under failure
Detecting failure well is the hard part of clustering. Nodus uses heartbeats, per-peer reachability and a suspicion window to tell a slow node from a gone one — then re-elects deterministically so the cluster heals without flapping and without a leadership gap.
Telling "slow" from "gone"
Evict too eagerly and a GC pause looks like a crash; evict too late and clients hang on a dead node. Nodus is built around that trade-off.
Heartbeats
Members exchange heartbeats over the persistent transport. A peer that keeps answering is healthy; one that goes quiet is a candidate for suspicion — no separate health-check service required.
Per-peer reachability
Reachability is tracked per peer, so a partition that isolates one link does not condemn the whole cluster. Each member judges each peer on its own evidence.
A suspicion window
Before a quiet peer is evicted it is held "suspect" for a window (about 20 seconds by default). That window tells "slow" from "gone" and stops a brief hiccup from flapping the membership.
Graceful leave vs hard kill
A clean shutdown broadcasts a leave so peers evict it immediately and failover is quick. A hard kill has no such signal, so the suspicion window is what eventually removes it.
What happens when a node dies
Failover is automatic. The survivors detect the loss, re-elect if needed, and re-admit the node when it returns — all without operator action.
Detect
Heartbeats stop from a peer; after the suspicion window it is declared gone and removed from the roster.
Re-elect
If the lost peer was the leader, every survivor runs the same deterministic election and the epoch advances — so there is never a gap with no coordinator or two claiming the role.
Re-admit
When the node comes back it rejoins from its peer set, a join event fires, and the roster and leadership settle again.
No two leaders, no leaderless gap
Because leadership is carried on a monotonically increasing epoch and every survivor runs the same election over the same reachable peers, failover converges on exactly one new leader at a higher epoch. A message from a deposed leader is recognizably stale, so split-brain claims are rejected rather than acted on.
Fast where it counts
- Graceful leaves evict immediately — no waiting on a timeout
- The suspicion window (~20s default) absorbs transient blips
- Re-election is a local computation, not a multi-round vote
- Disposal leaves the cluster cleanly so peers recover quickly
Resilience you don't have to build
Heartbeats, suspicion and failover — embedded, so your service just gets them.