Client resilience patterns

Clustron Zaris is a distributed system, so individual operations can encounter routing changes, brief unavailability, and transport failures. The client handles many of these for you automatically. This article explains what it retries on its own, what you are responsible for, and how to write operations that are safe to retry.

What the client retries automatically

The client has a built-in retry loop that transparently retries a bounded set of transient conditions before returning a result to you. It applies to normal operations out of the box; you can adjust its timeouts, or turn it off, through the environment variables in Tuning the client.

The client automatically retries these statuses:

Status	Why it is retried
Moved	Routing is stale; the client refreshes its ownership view and retries.
ReplicaWriteRejected	The write reached a replica instead of the primary; the client re-routes to the primary.
OwnershipChanging	Ownership is mid-transition; the client refreshes and retries.
Unavailable	The partition is briefly unable to serve.
Migrating	The partition is migrating; the client retries with extended backoff.

It also treats these exceptions as transient and retries them: TransportException, SocketException, PartitionUnavailableException, NodeUnavailableException, and TimeoutException.

For the routing statuses (Moved, ReplicaWriteRejected, OwnershipChanging), the client refreshes its routing map between attempts, so the retry lands on the correct node.

Backoff and bounds

Retries use a linear backoff — the delay grows with each attempt and is capped at 2 seconds. Migrating uses a longer multiplier because a migration takes longer to settle. Retries are bounded by both an attempt budget and the operation's deadline, so a persistently failing operation stops after a few attempts rather than retrying indefinitely. When retries are exhausted, the client surfaces the failure — as the final status on the result, or as a TransportException for transport-level failures.

What you handle yourself

Statuses that are not in the automatic set are returned to you to decide on, because the right action depends on your application. See the status reference for the full list; the common ones you handle are:

• Conflict — an optimistic-concurrency check failed. Re-read the current value and reapply your change.
• Locked — the resource is held by a lock. Back off and retry, or coordinate through the lock yourself.
• Timeout — the operation did not complete in time. Retry only if the operation is safe to repeat (see below).
• CapacityExceeded — the node is full and eviction is disabled (this applies only to the no-eviction path, which is not exposed through configuration today). Shed load or retry later.

var result = await client.GetAsync<Order>("order:1001");

switch (result.Status)
{
    case KvStatus.Success:
        Process(result.Value);
        break;
    case KvStatus.NotFound:
        // Expected absence — handle as a miss, not an error.
        break;
    default:
        // Log and decide: surface to the caller, or retry if idempotent.
        Log(result.Status, result.Error);
        break;
}

Make operations safe to retry

A retry only helps if repeating the operation is harmless. Classify your operations before retrying them:

• Reads (GetAsync, scans, queries) are always safe to retry.
• Puts and deletes are idempotent by nature: writing the same value or deleting the same key twice leaves the store in the same state. These are safe to retry.
• Counter increments and decrements are not idempotent. Retrying an increment can double-count if the first attempt actually succeeded but its acknowledgement was lost. For counters that must be exact, guard the update with a transaction and optimistic concurrency rather than a blind retry.

When an operation is not naturally idempotent, make it idempotent — for example by attaching a client-generated operation id you can check, or by expressing the change as a transaction whose conflict detection rejects a duplicate apply.

Failover and connection handling

Resilience also comes from how the client connects:

• Seed servers. Configure more than one seed endpoint so the client can reach the cluster even if one node is down. The client discovers the rest of the cluster from any reachable seed.
• Reconnection. The client re-establishes dropped connections and refreshes routing as the cluster changes, so ownership moves and node restarts are handled without you reconnecting manually.
• Token renewal. If the cluster uses token security, supply a TokenProvider rather than a static Token; it is read on every reconnect, so a rotated token renews access without dropping live work. See Connect to a secured store.

For how long a client should live and how to share it, see Client lifecycle.

Tuning the client

The retry loop, timeouts, and connection behavior have defaults that suit most applications. When you do need to change them, they are read from environment variables on the client process — set them before your application starts. The defaults are chosen deliberately; change them only for a specific reason.

Variable	Default	Effect
ZARIS_REQUEST_TIMEOUT	60 (seconds)	Per-request timeout. Accepts a number of seconds, or infinite / none / 0 for no timeout.
ZARIS_CONNECT_TIMEOUT_MS	3000	Connection timeout in milliseconds (minimum 100).
ZARIS_DISABLE_RETRIES	unset	Set to true to turn off the automatic retry loop (each operation is attempted once). Useful when you want to handle every retry yourself.
ZARIS_MAX_INFLIGHT	4096	Maximum concurrent in-flight requests per connection.
ZARIS_CONNECTIONS	1	Connections opened per node. One connection with request pipelining handles high concurrency; raise this only if benchmarks show a benefit.

For example, to give a batch job a longer request timeout and no client-side retries:

ZARIS_REQUEST_TIMEOUT=300 ZARIS_DISABLE_RETRIES=true dotnet run

The full environment-variable surface — including node, telemetry, and TLS-enrollment variables — is in the configuration reference.

Next steps

• Handle errors
• Status and exception reference
• Transactions
• Client lifecycle