Performance and benchmarking
Before you tune a Clustron Zaris store, measure it. Throughput and latency depend on your workload, your object sizes, your topology, and where the client runs — so a number you generate against your store with a representative workload is worth far more than a rule of thumb. This article shows how to run the built-in load and benchmark cmdlets, how to read what they report, and which store-configuration settings move performance and what each one trades away.
The recommended order is: measure a baseline, change one lever, measure again. Changing several settings at once makes it impossible to attribute a gain or a regression to any single one.
Activate a workspace and connect to the store first (see Connect to a cluster). Every cmdlet below runs against the connected store, or against a one-shot independent connection you pass with -Endpoints.
The tools
Zaris ships purpose-built cmdlets in the client shell, each answering a different question. Use the one that matches what you want to characterize rather than reading a single mixed number.
| Cmdlet | Measures | Workload | Reports |
|---|---|---|---|
| Benchmark-ZrStore | Aggregate throughput under a chosen read/write mix | Concurrent reads and writes, ratio set by -Profile | Ops/sec, total operations, failures |
| Test-ZrThroughput | Sustained write throughput | Continuous write-only | Ops/sec, total operations, failures |
| Test-ZrLatency | Per-operation read latency | Timed individual reads after a warm-up | Min, Avg, P95, P99, Max (ms) |
| Stress-ZrStore | Stability under sustained load | Randomized ~50/50 read/write over long runs | Total operations, failures, error rate, last error |
| Test-ZrItem | Whether a single key exists (a lightweight non-mutating get) | One key lookup | Boolean, or a detailed result object |
Test-ZrItem is a spot check rather than a load tool, but it is the cheapest way to confirm a specific key is present before or after a run.
Throughput and mixed workloads
To measure aggregate throughput under a realistic read/write mix, use Benchmark-ZrStore and pick the -Profile closest to your traffic — ReadHeavy (~10% writes), WriteHeavy (~90% writes), or Mixed (~50% writes). For example, a 30-second mixed run with 32 concurrent workers:
Benchmark-ZrStore -StoreName "orders" -Profile Mixed -DurationSec 30 -Concurrency 32
To isolate raw write throughput — useful when comparing replication or partition settings — use the write-only Test-ZrThroughput:
Test-ZrThroughput -StoreName "orders" -DurationSec 30 -Concurrency 64
Both cmdlets default -ObjectSizeBytes to 100. Set it to a size representative of your real values (for example -ObjectSizeBytes 1024 for ~1 KB records), because throughput in ops/sec falls as objects grow.
Latency
Throughput tells you how many operations complete per second in aggregate; it does not tell you how long any single operation took. For that, use Test-ZrLatency, which pre-populates the store, runs a warm-up pass, then times individual reads and reports the percentile distribution:
Test-ZrLatency -StoreName "orders" -Iterations 10000
Watch P95 and P99, not just the average — tail latency is what your slowest callers actually experience.
Sustained stress
To surface stability and error-handling problems that only appear over time, run Stress-ZrStore for a long duration at high concurrency. It drives a randomized ~50/50 mix and reports the error rate and last error type, so a run that stays at 0% error over 30 minutes is evidence of stability, not just speed:
Stress-ZrStore -StoreName "orders" -DurationSec 1800 -Concurrency 256
Stress-ZrStore also accepts -ExpirationSec to apply a TTL to written items, which keeps a long run from steadily growing the working set.
Watch a run live from a second shell with Watch-ZrStoreMetrics (see Monitoring and metrics). Seeing throughput per node while a benchmark drives load makes an imbalanced partition or a lagging node obvious.
Reading the results
A benchmark result is only meaningful in context. Keep the following in mind when you interpret the numbers.
- Throughput (ops/sec) is the aggregate rate the store sustained across all workers for the run. It scales with
-Concurrencyup to a point, then flattens or regresses once the client, the network, or the cluster saturates. Raising concurrency past that point adds latency without adding throughput — find the knee, don't just push the number up. - Latency percentiles describe the spread of single-operation times.
Test-ZrLatencysorts its samples and reports Min, Avg, P95, P99, and Max in milliseconds. The gap between Avg and P99 is your tail; a low average with a high P99 means most operations are fast but some callers see stalls. - Failures and error rate should be zero (or near it) on a healthy store. A non-zero
ErrorRatePctfromStress-ZrStore, or a rising failure count, points at capacity, timeouts, or a node problem rather than a tuning opportunity — investigate before you trust the throughput number from the same run.
Two things distort results if you ignore them:
- Warm-up. The first operations after a connection or against a cold store pay one-time costs (connection setup, first-touch allocation, an empty cache).
Test-ZrLatencypre-populates and runs a warm-up pass for this reason. For the throughput cmdlets, discard the very first short run and use a-DurationSeclong enough (tens of seconds) that steady state dominates. - Where the client runs. Latency includes the network round trip. Run the cmdlets from a machine close to the cluster — ideally the same subnet or datacenter — so you measure the store rather than the distance to it. A benchmark run across a WAN link mostly measures the link.
Numbers from different environments are not comparable. Object size, concurrency, client location, replication factor, and topology all move the result, so only compare runs that hold every one of those constant except the single lever you are testing.
Tuning levers
Once you have a baseline, the settings below are what actually move throughput and latency. Every one is a trade-off between faster and safer or more consistent — there is no free setting. All live in the store configuration file; see the configuration reference for exact fields, types, and defaults.
| Lever | Config setting | Faster when… | Trade-off |
|---|---|---|---|
| Partition count | topology.partitionCount | More partitions spread ownership and parallelism across nodes | Too many partitions add coordination overhead; null (default) uses one partition per node |
| Replication factor | replication.factor | Fewer copies (1) means less work per write | 1 has no replica and no redundancy; 2+ is recommended for production durability |
| Replication mode | replication.mode | Async — the primary returns immediately without waiting for replicas | Replicas may lag, so a failover can lose the most recent writes. Sync is safer but slower |
| Sync quorum | replication.quorum | BestEffort or Majority acknowledge sooner than All | Weaker quorum means a write can be acknowledged by fewer replicas; only applies when mode is Sync |
| Replica reads | reads.allowReplicaReads | true lets replicas serve reads, spreading read load off the primary | Reads may be stale; false (default) sends reads to the primary for freshness |
| Memory ceiling | memory.maxSizeBytes | A ceiling large enough to hold the working set keeps the hot set resident | Too small a ceiling forces constant eviction; there is no unbounded setting (default 1 GiB) |
A few notes on how these interact:
- Replication mode and quorum are the biggest write-latency lever.
Async(the default) lets the primary acknowledge a write before replicas catch up, which is fastest but means an ill-timed primary failure can lose the newest writes.Syncholds the write until replicas acknowledge according toreplication.quorum(All,Majority, orBestEffort) and waits up tosyncAckTimeoutSeconds. Choose based on how much a lost recent write costs you, not on the benchmark number alone. See Consistency. - Replica reads trade freshness for read throughput. With
allowReplicaReadsset totrue, read-heavy workloads can serve reads from replicas instead of concentrating them on the primary, which raises read throughput — at the cost of possibly returning slightly stale data. - The memory ceiling and eviction shape sustained performance. If the ceiling is too small for your working set, a
Stress-ZrStorerun will show eviction churn and degraded throughput as the node continuously evicts to stay under the ceiling. SizemaxSizeBytesto your working set plus headroom for the eviction band. See Memory and eviction for the ceiling model, the eviction policies, and the trigger/target thresholds.
Change one lever, re-run the same benchmark with identical parameters, and compare. If a change does not move the number you care about, revert it — a setting that adds risk without adding measured performance is not worth keeping.