Redis Streams, Natively: Append-Only Logs, Consumer Groups, and Blocking Reads

Zaris already speaks in collections — hashes, lists, sets, and sorted sets that live on the server and replicate like any other key. Streams are the fifth kind, and they're the one that turns Zaris from a store into a piece of messaging infrastructure.

A Zaris stream is a real append-only log: entries get monotonic IDs, you can read ranges, and you can attach consumer groups so a pool of workers divides the log between them, tracks per-consumer pending entries, and acknowledges what it has processed. You can also do a blocking read — a consumer parks and waits until new entries arrive instead of hot-looping.

And because it's a native Zaris collection, the log is partitioned and replicated exactly like everything else. A stream survives a node failover, which is the whole reason to reach for durable messaging in the first place.

The log, from .NET

Streams have their own typed surface: client.Streams.Stream(key) returns a handle to one server-side log. A producer appends entries as field sets; a consumer group reads and acknowledges:

var orders = client.Streams.Stream("orders");

// Producer — append an entry (returns its generated ID)
await orders.AppendAsync(new { id = 1001, total = 59.90, status = "placed" });

// Set up a consumer group once
await orders.CreateGroupAsync("fulfilment");

// Worker — read as part of the group, process, then acknowledge
var batch = await orders.ReadGroupAsync("fulfilment", "worker-1", count: 10);
foreach (var entry in batch)
{
    Process(entry);
    await orders.AckAsync("fulfilment", entry.Id);
}

ReadGroupAsync hands each entry to exactly one consumer in the group and moves it to that consumer's pending list until it's acked. If worker-1 crashes before it calls AckAsync, the entry stays pending and can be reclaimed — that's the mechanism behind at-least-once delivery. A plain ReadAsync (no group) reads the log directly, and a blocking read lets a consumer wait for the next entry rather than polling.

The same log, from any Redis client

Everything above is also reachable through the RESP front-end as the standard Redis Streams X-commands, operating on the same server-side stream:

# Append entries (* = server-assigned ID)
redis-cli -h zaris-0 -p 6379 XADD orders '*' id 1001 total 59.90 status placed
redis-cli -h zaris-0 -p 6379 XLEN orders
redis-cli -h zaris-0 -p 6379 XRANGE orders - +

# Consumer groups
redis-cli -h zaris-0 -p 6379 XGROUP CREATE orders fulfilment 0
redis-cli -h zaris-0 -p 6379 XREADGROUP GROUP fulfilment worker-1 COUNT 10 STREAMS orders '>'
redis-cli -h zaris-0 -p 6379 XACK orders fulfilment 1526919030474-0
redis-cli -h zaris-0 -p 6379 XPENDING orders fulfilment

XREAD (including BLOCK for blocking reads), XREADGROUP, XACK, and XPENDING all map onto the same native stream that client.Streams writes to. A .NET producer calling AddAsync and a Python worker issuing XREADGROUP share one log on one partition.

To make sure this is genuine Redis Streams behavior and not a lookalike, we ran an end-to-end harness that drives StackExchange.Redis against Zaris over a real socket — the append-only log, consumer groups, and blocking reads — and it passes 19/19 interoperability checks. A standard Redis Streams client works unmodified.

What you'd build with it

A replicated, consumer-group-aware log is the substrate for a handful of patterns that otherwise mean bolting on a separate system:

• Event sourcing. Append every state change as an immutable entry; the stream is the source of truth, and range reads replay history.
• Work queues with at-least-once delivery. A consumer group splits the log across workers; entries stay pending until acked, so a crashed worker's work is redeliverable rather than lost.
• Fan-out. Multiple consumer groups each read the whole stream independently — one log, many downstream pipelines, each tracking its own position.
• Durable inter-service messaging. Because the stream is replicated, a message that a producer committed survives a node failover instead of vanishing with the node that happened to hold it.

That last point is where a native, replicated stream earns its keep. Messaging built on a non-replicated log has a quiet failure mode: the node dies and the in-flight backlog dies with it, taking every unacknowledged entry with it. A Zaris stream is placed and replicated like every other collection, so the log — and the pending state that goes with it — is still there on the other side of a failover. The consumer group's read position and its outstanding pending entries survive too, which means a worker that reconnects after the failover picks up exactly where the group left off rather than replaying from the start.

Honest note: at-least-once, not exactly-once

These are Redis Streams semantics, and that means at-least-once delivery, not exactly-once.

Consumer groups plus acknowledgements guarantee that an entry which isn't acked can be redelivered — a worker won't silently drop work. What they don't guarantee is that a given entry is processed exactly once. If a worker finishes its side effect but crashes before AckAsync/XACK lands, the entry is still pending and will be handed out again. So your consumers should be idempotent: key side effects on the entry ID, or design them so reprocessing the same entry is harmless. Build for redelivery and the delivery guarantee is exactly what you want; assume exactly-once and you'll be surprised.

Where it fits

Streams complete the set: five server-side collection kinds — hashes, lists, sets, sorted sets, and now a durable append-only log — all partitioned and replicated the same way your keys are. If you haven't seen how the other four work, the data-structures post covers the shared model they're built on.

If you're already running Redis Streams, the move is close to free: the commands are the ones you know, the client is the one you already use, and the difference shows up when a node goes down and your log is still there.