Change Feed or Watch? Two Ways Zaris Tells You a Key Changed

Sooner or later a service needs to react to a change rather than poll for it. A cache wants to invalidate the instant a key is overwritten. A pipeline wants to ship every mutation into a warehouse. A dashboard wants to redraw the moment state moves. Zaris gives you two mechanisms for that, and the most important thing to understand up front is that they are not two flavors of the same feature. They make opposite trade-offs, and choosing wrong is a correctness bug waiting to happen.

The Change Feed — OpenChangeFeed — is change-data-capture: it tails a durable, revision-ordered operation log, hands a consumer a resumable cursor, and guarantees at-least-once delivery of every write, including a replay from the beginning for a fresh consumer. Watch — WatchPrefixAsync — is a live keyspace notification: the owner node pushes each change to subscribers as the write lands, best-effort, live-from-now, with no backfill and no durable cursor.

One is built so you never miss an event. The other is built so you hear about events as fast as physically possible. This post walks through how each actually works under the hood, then puts both through the same 1,000,000-item workload on a four-pod AKS cluster so the trade-offs are numbers, not adjectives.

Change Feed (CDC): a durable log you replay

The Change Feed is built on the same substrate Zaris uses for replication and migration: a revision-ordered operation log kept per segment on the owner node. Every accepted write advances a monotonic revision and appends an entry to that segment's log before the consumer ever gets involved — the write path only appends, it never fans out. A consumer is just a long-lived reader of those logs.

Three properties make it CDC rather than a tail:

• Durable consumer name. A feed is opened under a named consumer (cdc-a, warehouse-sync, …). The server keeps that name's committed offset per segment in an offset registry. The cursor is server-side state, not something your client has to persist.
• Parallel, filtered pull across all 256 segments. The client pump opens per-segment windows and pulls them in parallel across every partition owner, not through one funnel. Each pull is filtered server-side by key-prefix and op-kind, so a feed scoped to orders: on Put never ships you anything else.
• Replay from Start, resume from offset. A brand-new consumer name starts from the beginning of the retained log — it sees the seed write of every key, then every update, in revision order. An existing consumer resumes from its committed offset. Delivery is at-least-once: a reconnect may re-deliver the tail, so the consumer de-duplicates by key + version.

Writes append to a durable, ordered log; the consumer reads at its own pace while the server tracks its position — so after a restart or failover it resumes where it left off (or replays from the start). Nothing is missed.

The payoff is in that dashed line. When a consumer process dies and restarts under the same name, it does not start over and it does not lose the gap — it reconnects, asks each segment for everything after its committed offset, and the server serves it from the retained log. A source-side failover (a partition changes owners) is just another reconnect: the new owner continues the same revision sequence, and the consumer resumes from the same offset. Nothing is lost as long as the op-log retention window covers the consumer's lag.

The costs you accept are real: the Change Feed requires the op-log to be enabled on the server (ZARIS_WAN_SYNC_FEED=1, plus retention sizing), and because it is pull-based catch-up, a large burst of writes shows up as a queue the consumer drains after the writes land — seconds of catch-up latency for a million-item burst, and a heavier client buffer while it drains.

Watch (keyspace notifications): a live push you might miss

Watch is the opposite shape. There is no log and no cursor. A subscription registers interest in a key-prefix with the partition owners, and from that moment on, every write to a matching key is pushed live to the subscriber on the write path — the owner fans the notification out to all registered subscribers as part of committing the write.

That "on the write path" detail is the whole character of Watch:

• Live-from-now. A subscription sees writes that happen after it registered. There is no backfill of history. (IncludeInitialSnapshot can seed the current state of matching keys at subscribe time, but that is a one-shot snapshot, not a replayable cursor.)
• Best-effort, no durable position. There is no committed offset. If a subscriber disconnects — a slow consumer, a dropped TCP connection — or if the partition's ownership moves mid-stream, the subscription re-registers live-from-now again, and every event in that gap is lost silently. Nothing resumes it, because there is nothing that remembers where it was.
• Fan-out is a write-path cost. The work of delivering the notification lands on the node committing the write, in line with the write. More subscribers, or slower ones, means more work per write.

Each write is pushed live to subscribers — lowest latency, nothing stored. But there is no log and no saved position, so a subscriber that is offline when a write happens simply misses it, with no way to go back.

Watch is cheap on the client — there is almost nothing to buffer, because the subscriber is paced by the live write rate — and it is as fast as the store can be, because the notification leaves the owner the moment the write commits. What it does not have is a guarantee. The dashed "resume?" path is a dead end on purpose: Watch has no durable cursor, so a gap is unrecoverable by design.

Pros and cons, side by side

Change Feed (CDC)

Dimension	Change Feed
Durability	Durable op-log; no silent drops within retention
Ordering	Revision order per segment
Resumability	Server-tracked committed offset per consumer — survives restart & failover
Backfill / replay	Replays from Start for a fresh consumer (incl. the seed write)
Latency	Pull-based catch-up — seconds on a large burst, low on steady streams
Write-path cost	Low — the write only appends to the log
Client footprint	Heavier — buffers the backlog it is draining
Operational complexity	Requires ZARIS_WAN_SYNC_FEED=1 + op-log retention sizing

Watch (keyspace notifications)

Dimension	Watch
Durability	Best-effort — no guarantee
Ordering	Live arrival order; no replay to re-establish it
Resumability	None — no durable cursor
Backfill / replay	None (IncludeInitialSnapshot seeds current state only, one-shot)
Latency	Real-time — pushed on commit
Write-path cost	Higher — synchronous live fan-out per subscriber
Client footprint	Light — paced by the live write rate
Operational complexity	None beyond a running cluster

What it costs: a 1M-item benchmark on AKS

Adjectives are easy. Here are the measurements. The setup: a four-pod RF2 Zaris store on AKS (4× Standard_D8s_v7), a converged two-partition / 256-segment map. The workload: 1,000,000 items, each seeded at version 1 then updated to version 2 by a single updater, with two independent listeners that must each receive the full stream. We ran it twice on the same cluster — once with OpenChangeFeed, once with WatchPrefixAsync — each run on its own key prefix so neither replays the other's history.

On this stable, converged cluster, both paths delivered 100%: each listener reached 1,000,000 distinct final values with zero drops and zero duplicates, and the updater's independent read-back confirmed missing = 0. That is the honest starting point — and read the caveat below before you conclude Watch is "also reliable."

Metric	Change Feed (CDC)	Watch
Completeness (both listeners)	1,000,000 / 0 drops / 0 dups	1,000,000 / 0 drops / 0 dups
End-to-end catch-up lag	~1.95–2.39 s (burst drain)	≈ 0 (real-time)
Delivery latency p99	~5.8–8.0 s	~49–72 ms
Delivery latency max	~6.1–8.4 s	~104–326 ms
Updater update-phase throughput	263,682 items/s	135,480 items/s
Listener RSS (peak)	~343–369 MiB	~25 MiB
Store CPU, busiest owner (peak)	~326 m	~729 m

Four things fall out of that table, and they are exactly the trade-offs the mechanisms predict:

1. Latency — Watch wins decisively. Watch pushes each change live, so it kept up in real time (p99 ≤ 72 ms). CDC is pull-based catch-up: the consumer drains the op-log after the writes land, so a 1M burst shows seconds of catch-up. For steady, low-rate streams CDC latency is far lower — the seconds here are the cost of draining a million-item burst, not a fixed tax.

2. Write throughput — CDC wins. The updater pushed ~264k updates/s with CDC but only ~135k/s with Watch. Watch's synchronous live fan-out to the two subscribers adds work on the write path; CDC only appends to the log and lets consumers pull asynchronously. Roughly, live fan-out halved write throughput here.

3. Store CPU follows the fan-out. The busiest owner peaked ~729 m under Watch vs ~326 m under CDC — the fan-out cost lands on the write path, right where the throughput hit shows up.

4. Client memory — Watch is far lighter. Watch listeners peaked ~25 MiB; CDC listeners peaked ~343–369 MiB, because CDC pulls the op-log as fast as it can and briefly queues a large in-client backlog. Watch is paced by the live write rate and barely buffers.

The caveat you must not skip

Watch dropped nothing in this run only because the cluster was stable — converged, with no reconnect, no ownership move, and no backpressure during the 1M burst, so Watch never had to re-register and therefore never had a live-from-now gap. This is an honest best-effort success, not a guarantee. The moment a Watch subscription loses its owner binding mid-stream — node churn, a map rebuild, a slow consumer — the events in that gap are lost silently with no resume. CDC would not lose them: it would resume from its committed offset and re-deliver the gap. Do not read "0 drops" as "durable." It means "0 drops on a quiet cluster."

Where each one is the right tool

Reach for the Change Feed when a missed event is a correctness bug. Concrete cases:

• Cross-system data sync / ETL — streaming every mutation into a warehouse or search index.
• Outbox + CDC pipelines — the feed is the reliable delivery of your domain events.
• Audit logs — a dropped event is a compliance hole, not a cosmetic one.
• Cache rebuild after downtime — a consumer that was offline resumes from its offset and catches up the gap; a fresh consumer replays from Start.
• Anything that needs replay or resume — if you ever have to answer "did we process every change?", you need the durable cursor.

Reach for Watch when latency matters and a missed event only costs brief staleness. Concrete cases:

• Live dashboards / UI that reflect the latest state — a dropped update is corrected by the next write.
• Cache freshness / invalidation — worst case is a brief stale read, self-healed on the next change.
• Presence — "user is online" is inherently latest-wins; stale state expires anyway.
• Config hot-reload — pick up the new value fast; the next write re-asserts truth.

The common thread on the Watch side is self-correcting latest-state: every write re-establishes the correct state, so a gap costs you milliseconds of staleness, not a permanent hole. The common thread on the CDC side is the event itself is the record — losing one changes the outcome.

The decision rule

If losing an event would be a correctness bug — a missed row in the warehouse, a gap in the audit trail, an un-replicated write — use the Change Feed. You pay for it in burst latency, a heavier client, and a little server configuration, and in return you get durability, ordering, replay-from-Start, and a resumable cursor that survives restarts and failovers.

If losing an event only costs you a few milliseconds of staleness that the next write will fix — a dashboard tile, a cache entry, a presence flag — use Watch, and enjoy the real-time latency and the tiny client footprint. Just size your write path for the fan-out cost, and never mistake a clean run on a stable cluster for a delivery guarantee.

Two mechanisms, one question — what does it cost you to miss one? Answer that, and the choice makes itself.