Change Data Capture (CDC)

Clustron Zaris exposes a change feed: a durable, ordered, resumable stream of every Put and Delete applied to the keyspace, captured automatically. Subscribe to it and you can keep a cache, a search index, a materialized view, an audit log, or a downstream system in lock‑step with the store — without dual‑writes or polling.

CDC is new in Zaris 2.1.0, off by default, and fully additive: a store with the feed disabled behaves exactly as before.

How it differs from pub/sub and Streams

	What it is	Who writes it
Pub/Sub	Fire‑and‑forget fan‑out of messages to live subscribers	Your app publishes explicitly; ephemeral
Streams	An append‑only log you write records to, with consumer groups	Your app appends explicitly; durable
Change feed (CDC)	A record of what changed in the keyspace — every Put/Delete	Captured automatically by the store; durable

Pub/Sub and Streams are things you write to. The change feed is a faithful, implicit record of what actually happened to your data — you only read it.

Enabling the feed

The feed must be enabled on the cluster. In clustron.json:

changeFeed {
  enabled: true
}

or, primarily for tests and ops, start the node(s) with the environment override:

ZARIS_WAN_SYNC_FEED=1

The resumable tail is sized with ZARIS_WAN_SYNC_FEED_RETENTION_OPS / ZARIS_WAN_SYNC_FEED_RETENTION_BYTES. A consumer that falls behind the retention window transparently re‑snapshots (see Falling behind below).

The consumer API

The client implements IZarisChangeFeedClient. Get a client as usual, then open a feed:

using Clustron.Zaris.Abstractions.ChangeFeed;
using Clustron.Zaris.Client;
using Clustron.Zaris.Client.ChangeFeed;

// A Zaris client is also a change-feed consumer.
var feed = (IZarisChangeFeedClient)client;

var options = new ChangeFeedOptions
{
    ConsumerName = "search-indexer",                      // durable → server-tracked offset + lag
    KeyPrefix    = "product:",                            // server-side filter (optional)
    Kinds        = new[] { FeedEventKind.Put, FeedEventKind.Delete }, // optional (default: both)
};

await foreach (ChangeEvent e in feed.OpenChangeFeed(options, ct))
{
    if (e.IsDelete)
        await index.RemoveAsync(e.Key);
    else if (e.Value is { } bytes)                        // raw bytes of a plain-value write
        await index.UpsertAsync(e.Key, bytes, e.ExpiresAtUnixMs);
    // No manual commit — the named consumer's offset is tracked server-side.
}

OpenChangeFeed returns an IAsyncEnumerable<ChangeEvent> that yields events as they occur across all segments and runs until the CancellationToken is cancelled. Per‑segment cursors and snapshot recovery are managed internally. Backpressure is natural: a slow consumer throttles the reader.

ChangeFeedOptions

Property	Type	Default	Meaning
ConsumerName	string?	null	A durable consumer name. When set, the server tracks this consumer's committed offset (for resume and lag). null = anonymous: a fresh, session‑only stream.
KeyPrefix	string?	null	Only surface changes to keys starting with this prefix. null/empty = all keys. Pushed server‑side.
Kinds	IReadOnlyCollection<FeedEventKind>?	null	Only surface these op kinds. null/empty = both Put and Delete. Pushed server‑side.
MaxOpsPerPull	int	1000	Max events per pull window, per segment.
PollInterval	TimeSpan	200 ms	How often to poll when caught up; backs off geometrically toward MaxPollInterval.
MaxPollInterval	TimeSpan	5 s	Ceiling on the idle poll interval.

ChangeEvent

Member	Type	Meaning
Key	string	The key that changed.
Kind	FeedEventKind	Put or Delete.
IsDelete	bool	Convenience: Kind == Delete.
Value	byte[]?	Convenience: the raw value of a plain‑value write. null for a delete, a counter, or a framed collection — use Entry for those.
Entry	ZarisEntry?	The full entry for a write — value (any type) plus metadata (labels, entity, content‑type, TTL). null for a delete.
ExpiresAtUnixMs	long	Absolute expiry (unix ms) when the written value has a TTL; 0 = no TTL. Absolute, so it does not restart on the consumer.
WriteUnixMs	long	When the mutation happened at the source (unix ms).
Revision	long	Per‑partition monotonic source revision — for ordering within a partition; not a global sequence.

FeedEventKind is a two‑value enum: Put and Delete.

Delivery semantics

• Ordering is per‑key / per‑partition, not a global total order across keys.
• Delivery is at‑least‑once. A reconnect or snapshot recovery can re‑deliver an event. Make your handler idempotent — key your writes by ChangeEvent.Key so upsert‑on‑Put / remove‑on‑Delete is a safe no‑op on re‑delivery.
• A Put whose Value is null is a counter or a collection change; drop/re‑read the key (or inspect Entry) rather than treating it as a byte value.

Named consumers, offsets and resume

Set ConsumerName to make a named consumer. The server tracks that consumer's committed per‑segment offset, so:

• On restart, OpenChangeFeed resumes automatically from the last committed position — there is no manual cursor‑commit call.
• The consumer's lag is visible to operators.

An anonymous consumer (ConsumerName = null) starts fresh and is live only for the current session.

For lag monitoring or diagnostics you can read a named consumer's committed per‑segment offsets:

// On the concrete client (ZarisClusterClient):
Dictionary<ushort, FeedCursor> offsets =
    await client.GetChangeFeedOffsetsAsync("search-indexer", ct);

Each FeedCursor is (Generation, Revision) — the durable position in one segment's feed.

Filtering

KeyPrefix and Kinds are applied on the owner node, before the events cross the wire, so filtered‑out ops never reach the client. The cursor still advances past filtered ops, so the tail never stalls.

Security

When security is enabled on the store, opening the feed is read‑authorized — the identity needs read permission, the same as reading the keyspace.

Falling behind and recovery

If a consumer falls behind the retention window, the feed transparently re‑snapshots (OutOfRange → snapshot‑then‑tail): the current state is re‑emitted as Put events and the tail resumes from there. Because handlers are idempotent, this is seamless — you simply observe the current value of every (matching) key again.

Use cases

• Cache / CDN invalidation — evict or refresh exactly the entry that changed, the moment it changes, instead of guessing with blanket TTLs.
• Search index sync — stream Puts and Deletes into Elasticsearch/OpenSearch or a vector store so the index trails the data by milliseconds.
• Materialized views / projections — maintain denormalized read models (CQRS) by folding the feed forward; rebuild from the start when the shape changes.
• Audit logs / event sourcing — persist an ordered, durable trail of every mutation for audit, replay, and point‑in‑time reconstruction.
• Outbox / ETL — ship changes to a warehouse, a message bus, or a partner system without dual‑writes or a brittle polling job.

Runnable samples

Two console samples on GitHub consume the feed end to end:

• Cdc.CacheInvalidation — keep a local cache coherent.
• Cdc.SearchIndexer — project the feed into an inverted index.

Related

• Watch — react to changes on a single key or prefix within one cluster (live, in‑memory).
• Streams — an explicit, durable append‑only log with consumer groups.
• Geo-replication — CDC is the transport that active‑active WAN replication is built on.