Skip to main content

WAN active-active geo-replication

Clustron Zaris can replicate a store between independent clusters in different datacenters over a resumable, pull‑based change feed. Each site is a complete, self‑coordinating Zaris cluster — not a stretched cluster — so every site runs at full local speed, serves in‑region reads and writes, and a WAN partition degrades replication lag, never local availability. When the link returns, each site resumes from its durable cursor.

Geo‑replication is new in Zaris 2.1.0, off by default, and additive: a store with no site configuration behaves exactly as a normal single‑site store.

The model​

  • Site — one independent Zaris cluster, with its own membership, partition map, and store, identified by a site id.
  • Change feed — the per‑segment, revision‑ordered stream of a store's mutations. A peer pulls this feed and applies it locally. The feed is gated at the replicated watermark, so a consumer never sees a write that an owner crash could still lose.
  • Cursor — (generation, revision) per segment: a site's durable position in a peer's feed. After a restart or a WAN cut, the puller resumes from here — no full resync.
  • Conflict version — a hybrid logical clock (physicalMs, logical) plus the origin site id, stamped once on the original write. Last‑writer‑wins compares these (physical, then logical, then site id).
  • Origin — the site that first wrote a value. A feed serves only its own‑origin changes, so nothing echoes back around a two‑way topology.

Active-passive vs active-active​

Active → passive — one site is the active primary; the peer is a read‑only mirror that trails it over the feed. The passive refuses application writes (a non‑retryable "write elsewhere" error) but serves reads and watches from its local apply. A WAN cut just raises lag; on restore the mirror catches up.

Active-active — every site is active and lists every other site as a peer. Each site pulls every other site's feed, and because a feed serves only own‑origin changes, writes never ping‑pong. Every site takes writes locally and all sites converge.

Conflict resolution​

When two sites write the same key concurrently, resolution is deterministic and symmetric — every site computes the same winner for the same pair of writes:

  • Scalar values use hybrid‑logical‑clock last‑writer‑wins. The higher conflict version wins; the losing write is discarded. Use counters where every update must survive.
  • Counters use a PN‑counter: each site writes only its own component and the value is the sum, so concurrent increments at different sites are never lost.
  • Collections (hash / list / set / sorted‑set / stream) are whole‑value last‑writer‑wins in this release — a concurrent change replaces the whole value by version, with no field‑level merge.
  • A delete is a versioned tombstone decided by the same rule; a newer delete beats an older put and vice‑versa. A clear is geo‑aware: it replicates to peers as versioned tombstones rather than silently dropping only local data.

A site‑apply fires a local watch event only when the incoming change wins — a losing or re‑delivered change raises no event.

What stays site-local​

CAS, transactions, locks and leases are site‑local: they are not coordinated across sites, so do not rely on them for cross‑site mutual exclusion. TTL crosses as an absolute expiry — an item expires at the same wall‑clock instant on every site and does not restart on apply or resync.

Setup​

A store becomes a site by adding a site section to its configuration (additive; absent = a normal single‑site store):

site {
  id:    "dc1"                       # this cluster's site id
  role:  Active | Passive            # Passive refuses app writes (reads/watches allowed)
  epoch: <long>                      # monotonic site epoch, minted by the management plane
  peers: [                           # every peer this site replicates with
    { id: "dc2", store: "orders", seeds: "dc2-a:7861,dc2-b:7861" }
  ]
  mergePolicy:      LastWriterWins   # the only policy in this release
  homeSitePrefixes: [ "us/" ]        # optional: keys under these prefixes are homed here
}

Environment overrides (primarily for tests/ops): ZARIS_WAN_SYNC_FEED=1 enables the feed on a cluster; ZARIS_WAN_SYNC_FEED_RETENTION_OPS / ZARIS_WAN_SYNC_FEED_RETENTION_BYTES size the resumable tail.

With the PowerShell cmdlets​

The supported, operator‑facing path configures both sides in one call with the Zr AdminShell cmdlets — it POSTs the site/change‑feed config to the local and the peer managers and enables the feed:

# Link two stores active-active (bidirectional, last-writer-wins)
New-ZrSiteLink -Store orders-dc1 `
    -PeerWorkspace prod-dc2 -PeerStore orders-dc2 `
    -LocalSeeds 20.59.9.10:7863 -PeerSeeds 20.59.9.241:7863 `
    -Mode ActiveActive -MergePolicy LastWriterWins -Restart

Get-ZrSiteStatus -Store orders-dc1        # configured state + per-peer, per-segment lag
Remove-ZrSiteLink -Store orders-dc1 -PeerStore orders-dc2 -PeerWorkspace prod-dc2 -Restart

For an active‑passive link, use -Mode ActivePassive -Primary <store>. The WanReplication.Setup sample is a fully‑commented script that does this end to end.

Switch-over / failover. The epoch fence and role are implemented and persisted, but the orchestrated switch‑over (Switch‑ZrActiveSite) and active‑active evacuate (Suspend‑ZrSite / Resume‑ZrSite) are a documented roadmap item — not shipped cmdlets in this release. A planned switch‑over proceeds with zero loss only when the target's lag is 0; a forced switch‑over accepts and reports a bounded tail‑loss.

Observability​

  • site.* counters, both directions — applied, skipped, conflicts, and bytes, with live rates. Conflict counts are approximate (they count "an incoming change was based on a different version than the one held") and never influence resolution.
  • RPO is reported as the age of the oldest un‑applied op — about zero when caught up, climbing to real backlog‑seconds when a site falls behind.
  • Per‑peer, per‑segment lag (revisions and seconds) is available from Get-ZrSiteStatus and gates a planned switch‑over's zero‑loss check.
  • The web console renders an animated, live replication topology (active vs passive sites, one‑way vs bidirectional lanes, per‑site item counts) with a health banner (healthy / falling‑behind / stalled) and a dashboard callout.

Clock requirements​

Last‑writer‑wins and TTL use wall‑clock‑seeded hybrid logical clocks, so keep site clocks synchronized (NTP). A change stamped too far into the future is rejected/quarantined by the clock‑skew guard and the skew is exposed as a metric — investigate skew alerts promptly, as sustained skew distorts LWW outcomes and TTL.

Guarantees and limits​

  • Eventual consistency. A read reflects the local site's state; sites converge as the feed propagates. There is no cross‑site read‑your‑write or linearizability.
  • Last‑writer‑wins loses the losing write for a scalar key. Use counters (PN‑counter) where every update must survive.
  • Collections are whole‑value LWW in this release (no field‑level merge).
  • Forced switch‑over accepts and reports a bounded tail‑loss (writes not yet replicated when the old active was lost).

Operational notes​

  • Off by default and additive — enable it per store; a store with no site config is unchanged.
  • Validated on a two‑region AKS deployment over private network peering, including a three‑store isolation run.
  • Geo‑replication is built directly on Change Data Capture — the same durable, resumable change feed is the transport.