Skip to main content

Geo-replication and multi-site concepts

Geo-replication lets a Clustron Zaris store span more than one datacenter. Two (or more) independent clusters — one per region — each take writes locally and replicate to each other over a resumable, pull-based change feed. This is the conceptual model; the architecture page covers how it is built, and the developer guide covers how you turn it on and operate it.

Geo-replication is new in Zaris 2.1.0 and off by default — a store with no site configuration behaves exactly as a normal single-site store.

A site is an independent cluster​

A site is one complete Zaris cluster — its own membership, partition map, ownership, and local quorum — identified by a site id. Geo-replication couples sites only through the change feed; it is deliberately not a stretched cluster.

SITE A · REGION WESTindependent clusterCluster An1n2n3partitioned · RF ≥ 2op-log → change feedrevision-ordered · watermark-gatedSITE B · REGION EASTindependent clusterCluster Bn1n2n3partitioned · RF ≥ 2op-log → change feedrevision-ordered · watermark-gatedchange feed →← change feedWAN linkpull · resumable
Two independent clusters, coupled only by a resumable, pull-based change feed — not a stretched cluster.

Keeping sites independent is the whole point of an in-memory store: the WAN's latency never enters a local write path, and a WAN partition degrades replication lag, never local availability. The cost is eventual consistency across sites, bounded by that lag — made observable rather than hidden.

The change feed couples the sites​

Every segment owner already keeps a revision-ordered op-log of its mutations. The change feed is exactly that op-log tail, served as a resumable stream and gated at what is durable on a replica. A peer pulls the feed and applies it locally. Because the feed is pull-based, each site owns its own position and flow control.

  • Cursor — a site's durable position in a peer's feed, a (generation, revision) pair. Generation-first ordering keeps a consumer correct across the peer's own failovers. After a restart or a WAN cut, a site resumes from its cursor instead of re-syncing from scratch.
  • Origin — the site that first wrote a value. A feed serves only its own-origin changes, so a write never echoes back around a two-way topology.

Active-passive and active-active​

  • Active → passive — one site is the active primary; the peer is a read-only mirror that trails it over the feed and refuses application writes (reads and watches still work).
  • Active-active — every site is active and lists every other site as a peer. Each site takes writes locally, pulls every peer's feed, and all sites converge.

How concurrent writes converge​

When two sites write the same key at about the same time, the outcome is deterministic and symmetric — every site computes the same winner, so all sites end up identical without a coordinator arbitrating between them.

Concurrent writes to key K at two sitesresolve by value kind ↓SCALARSHLC last-writer-winsphysical · logical · site idCOUNTERSPN-countersum of per-site parts · nothing lostDELETESVersioned tombstonenewer delete beats older putEvery site converges to the same value
Resolution is deterministic and symmetric — every site computes the same winner, so all sites converge.
  • Conflict version — a hybrid logical clock (physicalMs, logical) plus the origin site id, stamped once on the original write. Last-writer-wins compares these in order: physical time, then logical counter, then site id.
  • Scalars use last-writer-wins: the losing concurrent write is discarded. Use counters where every update must survive.
  • Counters use a PN-counter: each site contributes its own component and the value is the sum, so concurrent increments at different sites are never lost.
  • Collections (hash, list, set, sorted-set, stream) are whole-value last-writer-wins in this release — a concurrent change replaces the whole value by version, with no field-level merge.
  • Deletes are versioned tombstones decided by the same rule; a clear is geo-aware and replicates as versioned tombstones rather than dropping only local data.

What stays site-local​

Geo-replication converges data, not coordination. CAS, transactions, locks and leases are site-local — do not use them for cross-site mutual exclusion. TTL crosses as an absolute expiry, so an item expires at the same wall-clock instant on every site (keep clocks synchronized with NTP). A local watch fires only when an incoming change wins its conflict resolution.

Consistency model​

Geo-replication is eventually consistent across sites: a read reflects the local site's state, and sites converge as the feed propagates. There is no cross-site read-your-write or linearizability — that is the deliberate trade for independent, low-latency regional availability.