Fail Closed, Never Destructive: Inside the Zaris License Lifecycle

Licensing is the part of a product most engineers never want to think about, and it shows. The usual pattern is a time bomb: a date is checked, and when it passes, the software punishes you — it throws on startup, it corrupts or wipes local state to stop you "stealing" it, or it simply crashes and takes your data with it. That design treats your own operators as adversaries, and it fails in exactly the moment you can least afford it: in production, after the clock quietly rolled over.

Zaris takes the opposite stance. The license check is a distributed data store's most safety-critical switch, so it's built like one: a single pure, deterministic policy function, a graceful and fully reversible enforcement path that never destroys a byte, and one clock per machine that resumes the instant you activate a key. This post walks through how that lifecycle actually works in the shipping code — because the interesting engineering here is in what it refuses to do to you.

The lifecycle, as a state machine

A never-licensed machine moves through a small, explicit set of stages. An automatic 30-day trial starts on first run with no key and no sign-up. When it elapses, the product does not stop — it enters a grace window where the web console is locked but the data plane keeps serving. Only after the grace window also elapses does the product gracefully stop serving, with your data preserved. Activating a valid license at any point short-circuits straight to Licensed.

The two defaults are both 30 days — a 30-day base trial, then a 30-day product grace after it ends — and both are plain constants, overridable per deployment:

public const int DefaultBaseTrialDays   = 30;
public const int DefaultProductGraceDays = 30;

There are two more terminal-ish states a key can put you in directly: Licensed (a valid Enterprise license, everything works, the local clock is ignored) and Non-Production (a valid dev-tier license, everything works but is flagged as not for production).

The policy is a pure function

The temptation in licensing code is to scatter if (DateTime.Now > expiry) checks across the console, the management service, and the node host — and then watch them disagree, because one read the clock a second later, one cached a stale file, one forgot the grace window. Zaris has exactly one place that decides, and it is a pure function with no I/O, no clock reads, and no SDK calls:

public static LicenseVerdict Evaluate(
    DateTime nowUtc,
    PaidLicenseStatus paid,   // resolved Keyright license (or None)
    TrialStatus trial,        // the local machine-bound trial marker
    int baseTrialDays   = DefaultBaseTrialDays,
    int productGraceDays = DefaultProductGraceDays)

Everything the verdict depends on is an argument. The web console, the management service, and every node host resolve their own inputs — the current time, the paid-license status, the local trial marker — and feed them through this same function, which guarantees one identical verdict everywhere. And because the transition logic touches nothing external, every edge of the state machine is exhaustively unit-testable without mocking a filesystem or a clock: you pass in a nowUtc and assert the stage.

The output is a single record that carries both the human-facing message and the two booleans that drive enforcement:

public sealed record LicenseVerdict(
    LicenseStage Stage,
    bool IsConsoleGated,    // -> the console returns HTTP 402
    bool IsProductStopped,  // -> the data plane refuses to serve
    bool IsPaid, bool IsTrial,
    int DaysRemaining, DateTime? ExpiryUtc,
    string StatusBadge, string Message);

Fail closed — but closed means the trial, not a crash

When the paid-license resolver can't reach the issuing service, or a lease file is unreadable, or anything at all goes wrong resolving the Keyright license, the service does not throw and it does not optimistically assume you're licensed. It falls back to the local trial clock:

try { paid = _paid.Resolve(); }
catch { paid = PaidLicenseStatus.None; } // fail-closed to the trial clock

"Fail closed" here is deliberately not "fail stopped." A transient network blip during license resolution drops you onto the machine's own offline trial marker, which is authoritative and needs no network. If you're still inside your trial or grace window, nothing changes for your running cluster. Only the genuine passage of time — not an outage — advances you toward a stop.

Two enforcement points, neither of them destructive

The verdict exposes two independent switches because the console and the data plane are gated at different times, on purpose.

The console gate (IsConsoleGated). The moment the trial elapses, the web console's admin and /v1 API routes return 402 Payment Required with a message pointing at how to re-enable it. This is middleware sitting in front of the console — it does not touch the data plane at all:

if (sw.IsGated) {
    context.Response.StatusCode = StatusCodes.Status402PaymentRequired;
    // ... reason + support URL ...
}

Crucially, during this stage the product keeps running. Your clients keep reading and writing; replication, coordination, and the data plane are untouched. What you lose is the management surface, with loud warnings telling you the window has closed — a prod-safe nudge, not an outage.

The data-plane gate (IsProductStopped). Only after the grace window elapses does the server start refusing data operations. It does so by returning an error status — never by deleting data, never by crashing:

if (_license?.IsProductStopped == true && !IsLicenseExempt(request.Command))
    return new ZarisResponse {
        Success = false,
        Status  = KvStatus.Unavailable,
        Error   = _license.ProductStoppedMessage // "...activate a license to resume"
    };

Two details make this humane rather than hostile. First, the core engine takes no dependency on the licensing implementation at all — it consults a tiny ILicenseGate interface with two members (IsProductStopped, ProductStoppedMessage) that lives in the abstractions assembly. The node host wires the real Keyright-backed service in behind it. The data store doesn't know what Keyright is.

Second, a stopped server is not a dead server. A specific set of commands stays exempt so an operator can always connect, inspect, and recover:

private static bool IsLicenseExempt(ZarisCommandType cmd) =>
    cmd == ZarisCommandType.Hello
    || cmd == ZarisCommandType.ClusterInfoRequest
    || /* the read-only diagnostic snapshots */ ;

The handshake, the cluster-info probe, and the read-only diagnostic snapshots all keep working. You can connect, see cluster state, confirm the data is intact, and activate a license — all while "stopped." The state is preserved on disk/in memory exactly as it was; the only thing withheld is serving reads and writes, and that is withheld reversibly.

One clock per machine

A trial that resets every time you reinstall, or that grants a fresh 30 days per node in a cluster, isn't a trial. Zaris enforces one machine = one seat: a single trial clock shared by every Zaris process on a box — manager, co-located nodes, console — regardless of the service account they run as. That's achieved by putting the trial marker in a machine-wide location (CommonApplicationData / ProgramData) and mirroring it to a second machine-wide copy.

The marker itself is built to resist casual tampering while being honest that it isn't DRM:

• It's bound to a machine fingerprint — a hash of the OS machine GUID (MachineGuid on Windows, /etc/machine-id on Linux) plus the machine name. A marker file copied to a different machine is simply ignored there; it grants that machine nothing.
• Each copy is HMAC-signed. An edited or forged copy is detected and treated as tampered.
• The earliest valid start wins across the mirrored copies, and extensions only ever move forward, so a trial can't be restarted or quietly rewound by deleting one of two files.
• A monotonic lastSeen is tracked; if the system clock is rolled back beyond a 24-hour tolerance, that's flagged as tampering too — you can't buy time by setting the clock back.

And a tampered marker is never rewarded. Rather than silently granting or silently resetting, the policy treats a detected tamper as the worst case and moves straight to stopped:

if (trial.Tampered)
    return Stopped(tampered: true); // never reward an altered/rolled-back marker

The honest limitation, documented right in the source: deleting every copy of the marker does reset the trial. That's an accepted boundary of the threat model — the goal is to raise the bar against casual circumvention on an open-source-visible codebase, not to pretend this is unbreakable copy protection.

Extensions extend; they never restart

Need more evaluation time? A vendor-issued Trial-tier key extends the window — for example to 90 days total — rather than starting a second trial. When such a key is observed, its expiry is persisted into the marker, moving the trial end forward:

// only ever forward — never shortens, never resets the base start
var extended = current is { } c && c >= untilUtc ? c : untilUtc;

Persisting the extension matters: it means the longer window (and its grace) survives the key later lapsing or the machine going offline. You don't need to stay connected to the licensing service to keep the extension you were granted.

Activation resumes without a restart

Because a stopped product is reversible, activation has to take effect on a live process. It does. The license service caches its verdict and re-evaluates on a periodic refresh (every five minutes by default), and an explicit activation forces an immediate re-evaluation:

public async Task<LicenseVerdict> ActivateAsync(string licenseKey, ...)
{
    await _paid.ActivateAsync(licenseKey.Trim(), ct);
    return Refresh();   // new verdict is live at once; the data plane resumes on its own
}

A stopped cluster that you license mid-outage simply starts serving again on the next evaluation — no rolling restart, no data reload, no window where state is at risk.

There are three ways to get a key in, matching how people actually run things:

• Set-ZrLicense -Key "LIC-…" from the admin shell applies the key across every manager machine in the workspace (each binds its own per-machine seat). -Deactivate releases a seat. Get-ZrLicenseStatus prints each manager's stage and badge.
• ZARIS_SUPPORT_KEY as an environment variable is activated once at startup — set it, restart, done. It's idempotent: re-activating an already-bound machine consumes no extra seat.
• An offline token for air-gapped deployments: a pre-issued signed blob that Keyright verifies entirely offline, no call home.

A handful of environment variables tune the rest for a given deployment: ZARIS_SUPPORT_WINDOW_DAYS and ZARIS_PRODUCT_GRACE_DAYS adjust the two windows, ZARIS_DATA_ROOT relocates the marker, and ZARIS_LICENSE_SERVICE_URL points at a self-hosted issuing service.

Why build it this way

It would have been less code to throw on an expired date. The extra machinery — a pure policy function, a two-stage gate, exempt recovery commands, a mirrored tamper-evident marker, resume-without-restart — all exists to honor one principle: a licensing decision must never cost you your data, and must always be reversible by doing the obvious thing.

The trial lapses and you keep running, with warnings. The grace lapses and the store stops serving while preserving everything and staying reachable for recovery. You activate a key and it resumes, live. And the whole decision comes from one deterministic function that every component agrees with, so there's no corner of the system quietly enforcing something different. Licensing is where a lot of good software turns against its own users; the aim here was to make it the one place a distributed store is allowed to say "no" without ever being allowed to say "gone."