Skip to main content

WAN geo-replication architecture

This page documents how Clustron Zaris replicates a store across datacenters, end to end. For the model at a glance see core concepts; for configuration and operations see the developer guide. Geo-replication is new in 2.1.0, off by default, and additive.

Two independent sites, coupled by a feed​

A site is a fully independent Zaris cluster — its own membership, partition map, ownership backbone, and local quorum. Sites are coupled only by a thin, resumable, pull-based change feed that ships durable key mutations between them. This is deliberate: a stretched cluster across a WAN would inherit the WAN's latency and partition behavior into every quorum decision, which is unacceptable for an in-memory store. Independence means each site runs at full local speed, a WAN partition degrades replication lag rather than availability, and each site fails over and recovers on its own. The price is eventual consistency across sites, bounded by replication lag — which the design makes observable and correct, never pretends away.

The change feed substrate​

The feed is not a second source of mutations — it is the existing op-log, served as a resumable, gated stream.

OWNER SITEPEER SITEClient writeat the owning siteOp-logrevision-orderedwatermark-gatedChangeFeedServiceowner-only pullserver-side filterSiteReplicatorapply via write path(SiteApply)Durable cursor(generation,revision)pullresume from durable cursor
Solid = data flow between stages. Dashed = the peer resuming its pull from the durable cursor.

Op-log as the source of truth​

Every segment owner maintains a revision-ordered operation log; each entry carries the key, the operation (Put/Delete), the record, the Revision, and the OwnershipGeneration. The feed reads that same log. (Op-logging is normally skipped for RF1 / no-replica segments as an optimization; when the feed is enabled that optimization is suppressed so the op-log is armed on every segment — missing this was historically the single biggest convergence bug.)

The cursor — (generation, revision)​

A consumer's position is a FeedCursor { Generation, Revision }. Generation-first ordering means an ownership-era change outranks any revision, so a consumer that crosses a peer failover orders events correctly instead of being confused by revision numbers that reset across eras. 0/0 means "from the beginning," and the position is persisted durably so a reconnect resumes rather than re-snapshots.

The replicated watermark — the phantom-write guard​

The feed must never serve a write that is not yet durable on a surviving replica; otherwise an owner crash could leave a peer holding a phantom write that no longer exists anywhere. The guard is the replicated watermark: per segment, the highest revision R such that every revision up to R is acked by at least one synced replica. It is a contiguous high-water mark, not the max acked revision — because replication delivery is reliable but not reorder-free, taking the max acked could advertise a high revision as durable while a lower one is still missing. The contiguous high never advances past the lowest still-unacked revision, so it is always safe. It is fail-closed: a segment configured with replicas but with none currently synced holds at the replicated watermark (it does not fall back to the local head). RF1 / no-replica segments serve the local head (an explicitly accepted single-copy exposure).

Generation fencing​

Each stored record and op-log entry carries an OwnershipGeneration, giving the apply side a total (generation, revision) order that stays correct across a site's own ownership changes. It is default-on.

The pull protocol and the replicator​

The owner serves one request/response pair — ChangeFeedPull { SegmentId, After (cursor), MaxOps } → { Events, Next, OutOfRange, Watermark, CaughtUp } — and it is owner-only: a node that is not the segment's active owner returns a not-owner reply so the puller re-routes. Key-prefix and op-kind filters are applied server-side, before events cross the wire.

The peer side is the SiteReplicator, built from three interfaces (so it is fully unit-testable without a live cluster):

InterfaceRole
IChangeFeedSourcepull the feed + read a segment snapshot
IChangeFeedApplierapply Put/Delete to the local site
ISiteCursorStoredurable per-segment resume cursor

Steady state: for each segment, read the durable cursor, pull from it, apply the events in feed order, and advance the cursor only after a non-empty batch — so a crash mid-batch re-pulls from the last durable position and re-applies idempotently. Passes run segments in bounded parallel with batched, per-key-ordered apply; a segment that faults is isolated, counted, and retried next pass; and the SiteReplicationLoop drives this continuously with geometric backoff when idle.

Recovery — snapshot-then-tail​

If a site has been offline long enough that its cursor falls below the owner's op-log trim floor, the next pull returns OutOfRange. The replicator then re-syncs from a consistent, owner-served single-revision snapshot of the segment's live keys, performs a DIFF-delete (remove any local key the snapshot no longer has, so the resync converges rather than merely adds), and resumes tailing from the snapshot revision. Durable cursors live under a reserved key prefix that the feed excludes, so they never cross-replicate. The op-log keeps a larger resumable tail when the feed is enabled (retention is op-count and byte budgeted; it drops-and-signals, never blocks writes).

Active-active​

Active-active builds on the same substrate, adding conflict versioning and an origin-aware apply.

Site A · originwrite Kstamp conflict versionHLC + origin AChange feedown-origin only(no echo back)Site BSiteApplyversion-gated last-writer-winswins → applyno echo backloses → dropolder version
A feed serves only own-origin changes (no echo); the peer applies version-gated, so sites converge without a coordinator.
  • Conflict version — a hybrid logical clock (physicalMs, logical) plus the origin site id, stamped once on the original write. Resolution compares physical, then logical, then site id, so every site computes the same winner for the same pair of writes.
  • Origin tagging + no-echo — a feed serves only its own-origin changes, and the privileged in-cluster apply (ChangeFeedApply → SiteApply) is version-gated, so a change applied from a peer is never re-emitted back to that peer. Resync is per origin.
  • Counters use a PN-counter: each site writes only its own component and the value is the sum, so concurrent increments at different sites are never lost. Collections are whole-value last-writer-wins in this release. Deletes are versioned tombstones.
  • Home-site prefixes can optionally pin keys to a home site (writes to keys homed elsewhere are rejected).
  • A clock-skew guard quarantines a change stamped too far into the future and exposes skew as a metric; keep site clocks on NTP, since sustained skew distorts last-writer-wins and TTL outcomes.

Site registry, epoch fence, and switch-over​

Each site carries a SiteOptions (role, peers, and a monotonic epoch). The epoch fence accepts state at or above the current epoch, so a deposed old-active is fenced out. A passive site rejects application writes with a read-only status. Switch-over is a pure decision over lag:

  • Planned switch-over proceeds with zero loss only when the target's replication lag is 0.
  • Forced switch-over proceeds immediately and reports the un-replicated gap as the accepted loss (the tail of a crashed owner that never reached a replica).

The orchestrated switch-over and evacuate operations (Switch-ZrActiveSite, Suspend-ZrSite, Resume-ZrSite) are a management-plane roadmap item; the lag tracking and the go/no-go decision they build on are implemented.

What stays site-local​

Geo-replication converges data, not coordination. CAS, transactions, locks, and leases are site-local and must not be relied on for cross-site mutual exclusion. TTL crosses as an absolute expiry (same wall-clock instant everywhere, subject to clock skew) and does not restart on apply or resync. A site-apply fires a local watch event only when the incoming change wins its conflict resolution.

Observability​

  • site.* counters, both directions — applied, skipped, conflicts, and bytes, with live rates. Conflict counts are approximate (they count "an incoming change was based on a different version than the one held") and never influence resolution.
  • RPO is reported as the age of the oldest un-applied op — about zero when caught up, climbing to real backlog-seconds when behind.
  • Per-peer, per-segment lag (revisions and seconds) is exposed and gates the planned-switch-over zero-loss check.
  • The web console renders an animated, live replication topology (active vs passive sites, one-way vs bidirectional lanes, per-site item counts) with a health banner (healthy / falling-behind / stalled).

Validation and status​

Active → passive replication is delivered and live-validated on two independent AKS clusters (write 300 keys and delete 40 on the active, the passive converges to 300 puts and 40 deletes via the feed). Active-active and the N-site surface are built on the same substrate (conflict versions, version-gated apply, per-origin resync and versioned tombstones, PN-counter, clock-skew guard, site registry, lag tracker, and switch-over decision), with the hosted per-peer puller service and the management-plane cmdlet surface being the integration layer. Everything is off by default; a store with no site config is unchanged.