Skip to main content

31 posts tagged with "Distributed Systems"

Articles about distributed systems architecture and design.

View All Tags

Elastic Scaling: Adding and Removing Zaris Servers Without Downtime

· 5 min read
Clustron Team
Distributed Systems Engineering

Growing and shrinking a Zaris store live under a supervisor

Capacity is never right for long. Traffic grows, a launch lands, a season ends — and the store you sized last quarter is either straining or half-idle. The question that decides whether that's a chore or a crisis is simple: can you change the size of a running store without taking it down?

For Zaris the answer is yes, with an important asymmetry that's worth being honest about up front. Scaling out — adding servers — is the easy, safe, lossless direction, and it happens live while the store keeps serving. Scaling in — removing servers — is also supported, but it's a deliberate, monitored operation because the data on a departing server has to go somewhere first.

This post walks both directions: growing a store with Add-ZrServer, shrinking it with Remove-ZrServer, why scale-in needs a drain, and where the web console fits.

One Connection String, Any Platform: Connecting to Zaris

· 6 min read
Clustron Team
Distributed Systems Engineering

One connection string across .NET config, PowerShell, and the web console

Half of the "works in dev, breaks in prod" connection bugs come down to the same thing: the connection details were spelled three different ways in three different places. The appsettings.json had one host list, the ops runbook pasted another into a PowerShell session, and the string someone copied off a wiki had a stale token baked right into it.

Zaris takes that whole class of problem off the table with a single, platform-agnostic connection-string format. The exact same string works when your .NET app reads it from configuration, when an operator hands it to a PowerShell cmdlet, and when someone copies it straight out of the web console. One canonical spelling, everywhere.

This post breaks the string down part by part, shows all three ways to use it, and makes the case for the one rule that matters most in production: never put a raw token in the string.

Pluggable TLS Trust: Four CA Modes for Locking Down Zaris

· 7 min read
Clustron Team
Distributed Systems Engineering

Four CA modes for pluggable TLS trust in Zaris

The fastest way to make sure TLS gets skipped is to force one trust model on everyone. A dev running a store on a laptop doesn't want to stand up a CA. A bank with an offline root in an HSM doesn't want your tool minting certificates at all. Ship a store that only knows one of those, and half your users turn security off to get their work done.

So Zaris doesn't pick for you. Its TLS trust is pluggable across four certificate-authority modes, from zero-config self-signed all the way to certificates you provision entirely out of band. You choose the mode that matches the PKI you already run, and Zaris slots into it instead of asking you to rebuild around it. TLS isn't one-size-fits-all, and treating it that way is exactly why security gets deferred.

Reaching Zaris From Outside the Cluster: External Clients Without a Native SDK

· 6 min read
Clustron Team
Distributed Systems Engineering

Reaching Zaris from outside the cluster

Run Zaris in Kubernetes and the happy path is easy: in-cluster clients reach the pods by their internal addresses — pod IPs and cluster-DNS names like zaris-0.zaris.svc — and everything routes cleanly. The moment a client lives outside the cluster, that breaks. An external application cannot dial zaris-0.zaris.svc or a pod IP; those names and addresses only resolve and route inside the cluster network.

This is not a Zaris quirk — it is the fundamental split between the cluster's internal network and the outside world. It bites hardest with topology-aware clients. A cluster-mode Redis client, or any client that discovers nodes and routes per-key, fetches a topology map from the server and connects to the addresses in it. If those addresses are internal, an external client gets a map full of endpoints it can never reach.

Zaris solves this with advertised addresses, handled as a per-listener "boundary view."

Three Ways to Fail Over: LoadBalancer, Sentinel, and Cluster on the Zaris RESP Front-End

· 6 min read
Clustron Team
Distributed Systems Engineering

Three ways to fail over on the Zaris RESP front-end

Zaris exposes a Redis-protocol (RESP) front-end so any Redis client, in any language, can point at a Zaris cluster and use it as a drop-in store. But "any Redis client" hides a real problem: Redis client ecosystems do not agree on how to discover topology or how to recover when a node disappears. A plain client expects a single stable endpoint. A Sentinel-based client expects to ask a discovery service "who is the master?" A cluster-mode client expects hash slots and MOVED/ASK redirects.

If a store spoke only one of those dialects, a whole class of clients would be locked out. So the Zaris RESP front-end supports three failover models, each matching how a particular family of Redis clients already expects to work. You pick the model per deployment; the client behaves exactly as it would against real Redis.

One honest note up front, because it shapes everything below: all three models sit in front of the same replicated, partitioned Zaris store. The model you choose changes discovery and redirection semantics — how a client finds a node and what it does when one fails — not where your data actually lives.

Redis Streams, Natively: Append-Only Logs, Consumer Groups, and Blocking Reads

· 6 min read
Clustron Team
Distributed Systems Engineering

A native, replicated append-only log with consumer groups and blocking reads in Zaris

Zaris already speaks in collections — hashes, lists, sets, and sorted sets that live on the server and replicate like any other key. Streams are the fifth kind, and they're the one that turns Zaris from a store into a piece of messaging infrastructure.

A Zaris stream is a real append-only log: entries get monotonic IDs, you can read ranges, and you can attach consumer groups so a pool of workers divides the log between them, tracks per-consumer pending entries, and acknowledges what it has processed. You can also do a blocking read — a consumer parks and waits until new entries arrive instead of hot-looping.

And because it's a native Zaris collection, the log is partitioned and replicated exactly like everything else. A stream survives a node failover, which is the whole reason to reach for durable messaging in the first place.

Data Structures That Replicate: Hashes, Lists, Sets, and Sorted Sets in Zaris

· 6 min read
Clustron Team
Distributed Systems Engineering

Native hashes, lists, sets, and sorted sets, replicated like every other key in Zaris

Zaris 1.x stored values. Zaris 2.0.0 stores collections. Hashes, lists, sets, and sorted sets are now first-class, server-side types with real, typed .NET APIs — and, just as importantly, they behave like everything else in the store.

That last part is what people usually get wrong. In a lot of systems, "data structures" are a client-side illusion: a library serializes a dictionary into a blob, Puts it, and calls that a hash. It works until two clients touch the same collection at once, or until you ask what happens on a failover. In Zaris, a hash is a hash on the server. It sits on the partition that owns its key, it replicates to that partition's replica, and it survives a node loss exactly the way a plain key does.

Zaris 2.0.0: Native Data Structures, Streams, and a Redis-Protocol Front-End

· 7 min read
Clustron Team
Distributed Systems Engineering

Zaris 2.0.0 — Native data structures, Streams, and a Redis-protocol front-end

Today we're shipping Zaris 2.0.0 — the biggest release since Zaris reached general availability, and the first with a major version bump. It turns Zaris from a fast, replicated key/value store into a full data-structure server: native hashes, lists, sets, sorted sets, pub/sub, and Redis Streams, all partitioned and replicated the same way your keys already are.

And it opens the door to everyone else. Every node can now expose a Redis-protocol (RESP) front-end, so an application in Python, Ruby, Java, Go — any language with a Redis client — can point at Zaris and use it as a drop-in, without a native SDK and without rewriting a line.

The version number is a 2, not a 1.x, for a reason: 2.0.0 changes the wire. Read the upgrade note below before you roll it out.

Zaris 0.9.0 Is Here — and It's Generally Available

· 4 min read
Clustron Team
Distributed Systems Engineering

Zaris 0.9.0 — Generally Available

Today we're shipping Zaris 0.9.0, and with it, Zaris hits general availability. It's production-ready, cross-platform, and now backed by commercial support.

If you haven't run into it yet: Zaris is a distributed, in-memory key/value store for .NET. It's fast, replicated, and you can operate it from a browser, PowerShell, or straight from your app. This release is the culmination of that work — the point where "promising project" becomes "thing you can put in production."

Distributed Locks Explained

· 4 min read
Clustron Team
Distributed Systems Engineering

A Practical Guide for .NET Developers

Introduction​

As applications scale across multiple machines, coordination becomes one of the most challenging aspects of system design.

In a single-machine application, mutual exclusion is straightforward. Developers can rely on language primitives such as lock in C#, Mutex, or Semaphore to ensure that only one thread accesses a shared resource at a time.

However, these primitives only work within a single process or machine.

Once applications run across multiple services or nodes, traditional locking mechanisms no longer work. Two services running on different machines cannot rely on a local lock statement to coordinate access to shared state.

This is where distributed locks become essential.

Distributed locks allow services across a cluster to coordinate access to shared resources while ensuring that only one participant holds the lock at a given time.


Local Locks vs Distributed Locks​

A typical local lock in C# looks like this:

private static readonly object _lock = new object();

public void ProcessOrder()
{
lock(_lock)
{
Console.WriteLine("Processing order safely.");
}
}

This works perfectly inside a single process.

However, if your application runs on multiple service instances, each instance has its own _lock object. The lock no longer coordinates work across the system.

Worker A → lock acquired Worker B → lock acquired Worker C → lock acquired

All workers proceed simultaneously, defeating the purpose of locking.

Distributed systems require a shared coordination mechanism.


Distributed Locking


The Idea Behind Distributed Locks​

A distributed lock uses a shared system to represent lock ownership.

Instead of locking memory, a node records lock ownership in a distributed coordination system such as:

  • a distributed key‑value store
  • a coordination service
  • a distributed cache

Example concept:

Lock Key: order:123 Owner: worker‑1

If another worker attempts to acquire the same lock, the operation fails until the lock is released or expires.


Real World Scenario: Distributed Invoice Processing​

Imagine a cluster of workers processing invoices.

Worker A Worker B Worker C

All workers scan for pending invoices.

Without locking:

Worker A → processes invoice #101 Worker B → processes invoice #101 Worker C → processes invoice #101

The same invoice could be processed multiple times.

A distributed lock ensures that only one worker processes the invoice.


Distributed Locks with Clustron DKV​

Clustron DKV provides built‑in primitives for distributed coordination.

Locks are managed through the Locks API, and operations on locked keys must include the correct lock handle.


Acquiring a Lock​

A client first acquires a lock for a specific key.

var locks = ((IDkv)client).Locks;

var key = "order:123";

var lockHandle = await locks.AcquireAsync(key, TimeSpan.FromSeconds(10));

if (lockHandle == null)
{
Console.WriteLine("Failed to acquire lock.");
return;
}

Console.WriteLine("Lock acquired.");

The returned lock handle represents ownership of the lock.


Performing Operations Under the Lock​

Once a lock is acquired, operations must include the lock handle.

var result = await client.PutAsync(
key,
"processed",
Put.WithLock(lockHandle.Handle));

if (result.IsSuccess)
{
Console.WriteLine("Value updated safely.");
}

Only the lock owner can modify the value.


Reading with the Lock​

Reads can also respect the lock ownership.

var value = await client.GetAsync<string>(
key,
Get.WithLock(lockHandle.Handle));

if (value.IsSuccess)
{
Console.WriteLine($"Value: {value.Value}");
}

Competing Operations Without the Lock​

If another worker attempts to modify the key without the lock, the operation fails.

var attempt = await client.PutAsync(key, "new-value");

if (!attempt.IsSuccess)
{
Console.WriteLine($"Operation rejected: {attempt.Status}");
}

Result:

KvStatus.Locked

This prevents concurrent modification of the same resource.


Releasing the Lock​

Once the work is finished, the lock can be released.

await lockHandle.ReleaseAsync();

Console.WriteLine("Lock released.");

After the lock is released, another worker may acquire it.


Example Cluster Workflow​

Worker A → Acquire Lock → Success Worker B → Acquire Lock → Failed Worker C → Acquire Lock → Failed

Worker A → Update Key Worker A → Release Lock

Only one worker performs the operation, ensuring correctness.


Handling Failures​

Locks in Clustron are time‑bound.

When acquiring a lock, a duration must be specified:

await locks.AcquireAsync(key, TimeSpan.FromSeconds(10));

If the client holding the lock crashes or disconnects, the lock eventually expires, allowing another worker to acquire it.

This prevents locks from remaining permanently held.


Best Practices​

When implementing distributed locks:

  • Keep critical sections short
  • Avoid global locks when possible
  • Partition locks by resource
  • Always design for failure scenarios

Final Thoughts​

Distributed locks are one of the most fundamental coordination primitives in distributed systems.

They allow services running across multiple machines to safely coordinate access to shared resources while preserving system correctness.

Platforms like Clustron DKV simplify this by providing built‑in locking primitives, atomic operations, and strong consistency guarantees.

In the next article, we will explore another essential distributed systems primitive:

Leader Election.