Skip to main content

Operating WAN geo-replication

This is the operational guide for running WAN-replicated sites — configuring, linking, monitoring, switching over, and troubleshooting. For the model see geo-replication concepts, for the design see the architecture page, and for the client/SDK view see the developer guide.

Geo-replication is new in 2.1.0 and off by default. A store with no site configuration is a normal single-site store.

Enabling the change feed and configuring sites​

WAN replication rides on the change feed, so the feed must be enabled on each cluster, and each store gains a site section. You can do this two ways.

Config file — add changeFeed and site blocks to the store config:

{
  "clusterId": "orders",
  "changeFeed": { "enabled": true },
  "site": {
    "id": "dc1",
    "role": "Active",
    "epoch": 1,
    "peers": [
      { "id": "dc2", "store": "orders", "seeds": "dc2-a:7861,dc2-b:7861" }
    ],
    "mergePolicy": "LastWriterWins"
  }
}

PowerShell — New-ZrSiteLink configures both sides in one call (it posts the site/change-feed config to the local and the peer managers and enables the feed), which is the recommended path:

# Active-active link between two stores in two workspaces
New-ZrSiteLink -Store orders-dc1 `
    -PeerWorkspace prod-dc2 -PeerStore orders-dc2 `
    -LocalSeeds 20.59.9.10:7863 -PeerSeeds 20.59.9.241:7863 `
    -Mode ActiveActive -MergePolicy LastWriterWins -Restart

# Or an active -> passive mirror
New-ZrSiteLink -Store orders-dc1 -PeerWorkspace prod-dc2 -PeerStore orders-dc2 `
    -LocalSeeds 20.59.9.10:7863 -PeerSeeds 20.59.9.241:7863 `
    -Mode ActivePassive -Primary orders-dc1 -Restart

-Restart rolling-restarts each node so it arms the link now; omit it to restart on your own schedule. The relevant environment knobs (ZARIS_WAN_SYNC_FEED, retention, fencing, and the watermark gate) are in the configuration reference.

Cmdlets​

CmdletPurpose
New-ZrSiteLinkConfigure an active-active or active-passive link across two workspaces (both sides).
Get-ZrSiteStatusShow role, epoch, each peer, and per-peer / per-segment lag (revisions and seconds).
Remove-ZrSiteLinkUnlink two sites (clears both sides; the feed stays on for other CDC consumers).
Switch-ZrActiveSitePlanned or forced active → passive switch-over (role change).
Suspend-ZrSite / Resume-ZrSiteActive-active evacuate — quiesce one site to read-only for maintenance, then resume.

Status. New-ZrSiteLink, Get-ZrSiteStatus, and Remove-ZrSiteLink are the configuration path. The orchestrated Switch-ZrActiveSite and the Suspend-ZrSite / Resume-ZrSite evacuate are a documented management-plane roadmap item in this release; the lag tracking and the zero-loss go/no-go decision they build on are implemented.

The web console Replication tab​

The web console adds a top-level Replication tab to the store page (alongside Overview, Nodes, Partitions, Data, Metrics, Settings). It shows the live replication topology — active vs passive sites, one-way vs bidirectional lanes, and per-site item counts — a health banner, and a wizard that calls New-ZrSiteLink under the hood (it can reach multiple managers to pair both sides).

Monitoring​

  • site.* counters, both directions — applied, skipped, conflicts, and bytes, with live rates, plus the per-pass stats applied / deleted / snapshots / failed / notOwner.
  • RPO is reported as the age of the oldest un-applied op — about zero when caught up, climbing to real backlog-seconds when a site falls behind.
  • Per-peer, per-segment lag in revisions and seconds from Get-ZrSiteStatus, plus a single "caught up?" per peer.
  • Health states: healthy (lag near zero, no failures), falling-behind (lag growing), and stalled (no progress). Alert on sustained lagMs growth or a non-zero failed / notOwner streak. A brief lag spike after a write burst is normal — the feed only serves up to the replicated watermark.

Switch-over​

Planned (zero loss). Fence the active's writes (clients get a retryable OwnershipChanging), wait until the target's cursors reach the old active's head (lag 0 — the command blocks on this), then the epoch bumps and roles flip; the old active becomes passive and starts pulling from the new active. Verify with Get-ZrSiteStatus.

Forced (bounded gap). Promotes immediately and reports the un-replicated gap as the accepted loss (A1). When the old active returns it sees the higher epoch and flips to passive; its stale-epoch writes are fenced. Investigate the reported gap if it is larger than expected.

Active-active has no switch-over (both sides already take writes); the maintenance analog is evacuate — Suspend-ZrSite to quiesce one site to read-only, Resume-ZrSite after. This does not change roles or bump the epoch.

Adding and removing a site​

  • Add: register the new peer in every existing site's peers and set the new site's role/peers. The new site bootstraps from a per-origin snapshot from each peer, then tails each feed — it converges without deleting any existing site's data.
  • Remove: drop the peer from every other site's peers, so its cursors and retained feed tail can be released. Confirm no site still lists it.

Geo-aware clear​

A clear on a replicated store is geo-aware: it replicates to peers as versioned tombstones rather than dropping only local data, so a cleared keyspace converges across sites instead of resurrecting from a peer.

Clock requirements​

Last-writer-wins and TTL use wall-clock-seeded hybrid logical clocks, so keep every site's clock on NTP. A change stamped too far into the future is quarantined by the clock-skew guard and the forward skew is exposed as a metric. On a skew alert, correct NTP promptly — sustained skew distorts conflict outcomes (a fast-clock site wins conflicts it should not) and TTL (early or late expiry).

Troubleshooting​

A replicator that stops making progress (stalled). Check Get-ZrSiteStatus for a non-zero failed / notOwner streak and growing lagMs. notOwner means the segment owner moved — the replicator re-routes on the next pass. A failed segment is isolated and retried next pass. Confirm the peer is reachable at its seeds and that the source side still has the feed enabled.

OutOfRange storms. A consumer whose cursor fell below the op-log trim floor re-syncs from a server-side snapshot — a few after a long cut are normal. Many segments repeatedly going OutOfRange means the retained tail is too small for the lag: raise ZARIS_WAN_SYNC_FEED_RETENTION_OPS / _BYTES, or reduce lag (network/capacity). A snapshot that keeps replying "retry" means the segment's replicated watermark has not caught up (writes in flight or a replica down) — check replica health.

Kubernetes: re-apply site config after a pod reschedule. When a node's site configuration is written to an emptyDir (ephemeral) volume, a pod reschedule starts the node with no site config and it comes up as a plain single-site store. Re-apply the geo configuration (New-ZrSiteLink, or a config mounted from a ConfigMap/PVC) after any reschedule, or mount the site config from durable storage so it survives.