Messaging

Once nodes are in a cluster, they need to talk directly — assign work to a specific peer, broadcast an invalidation to everyone, or ask a question and wait for an answer. That is what the messaging plane provides, reached through node.Messaging (IMessaging).

Messaging is directed communication: you name a recipient (or "everyone") and send. This is distinct from cluster events, which is a decoupled publish/subscribe bus. Reach for messaging when you know who you are talking to.

The transport underneath

All messaging rides a pipelined-TCP transport with MessagePack serialization on the wire. You rarely touch any of this directly — the typed helpers below handle framing and serialization for you — but it is why messaging is cheap enough to use on a hot path.

• Pipelined TCP, built on System.IO.Pipelines, keeps a persistent connection to each peer and streams framed messages over it, so there is no per-message connection setup.
• A prioritized outbound path separates urgent control traffic (heartbeats, immediate sends) from bulk queued traffic, so a flood of application messages cannot starve the liveness signal that keeps the cluster healthy.
• MessagePack gives compact binary serialization of your payloads.

Two levels of API

node.Messaging offers two levels, and you can mix them freely. Use the typed convenience methods to move a payload object between nodes, and drop to the protocol level when you need the wire message itself.

Level	Methods	Use when
Typed convenience	SendAsync<T>, BroadcastAsync<T>, On<T>	You just want to move a payload object between nodes.
Protocol-level (raw wire)	CreateMessage<T>, SendAsync(Message), SendImmediateAsync, RequestAsync<TReply>, AddHandler	You need the correlation id, the message type, the priority plane, or request/reply.

Point-to-point (typed)

Send a typed payload to one named node; the recipient handles it with a matching On<T>:

// sender
await node.Messaging.SendAsync("node-2", new WorkItem { Id = 42, Range = (1000, 2000) });

// recipient — handler receives (payload, fromNodeId)
node.Messaging.On<WorkItem>((item, from) =>
{
    Console.WriteLine($"got work {item.Id} from {from}");
    return Task.CompletedTask;
});

The payload is any serializable POCO. There is no base class or attribute to add.

One typed handler per node

On<T> registers into a single typed-message slot on the node — every On<T> call maps to the same underlying client-message channel, so a later On<T> replaces the earlier one. A node effectively has one typed handler at a time.

The idiomatic pattern is a single envelope type with a discriminator field, and you branch inside the handler:

public sealed class CoordinationMessage
{
    public string Kind { get; set; } = "";   // "assign" | "result"
    public int ChunkId { get; set; }
    public long From { get; set; }
    public long To { get; set; }
    public long Primes { get; set; }
}

node.Messaging.On<CoordinationMessage>(async (msg, from) =>
{
    switch (msg.Kind)
    {
        case "assign": /* compute and reply */ break;
        case "result": /* aggregate */         break;
    }
});

This is what the ClusterCoordination sample does. If you need genuinely separate handlers per message type, register low-level IMessageHandlers via AddHandler (below), which dispatch by message-type string.

Broadcast (typed)

Send a payload to every member at once:

await node.Messaging.BroadcastAsync(new CacheInvalidation { Key = "user:42" });

Every member's On<CacheInvalidation> (or its envelope handler) receives it. Broadcast is a fan-out over the same direct connections — a convenience over sending to each member individually.

Only members can send

SendAsync and BroadcastAsync throw NotSupportedException from a node that does not advertise the member role. See Membership → roles.

Raw wire access

When you need control over the wire Message — its type, correlation id, or priority — build it explicitly. CreateMessage<T> stamps the message with this node's id and the configured serializer:

Message msg = node.Messaging.CreateMessage("orders.reserve", new ReserveRequest { Sku = "ABC", Qty = 3 });

await node.Messaging.SendAsync("node-2", msg);              // queued send
await node.Messaging.SendImmediateAsync("node-2", msg);     // priority/unbuffered plane

A Message carries the following fields:

Field	Purpose
MessageType	The routing key handlers register against.
SenderId	Stamped for you by CreateMessage.
CorrelationId	Ties a reply back to its request (see below).
Payload	The MessagePack-serialized bytes.
TypeInfo, Timestamp, TraceParent	Type metadata, send time, and a W3C trace id for cross-node tracing.

SendImmediateAsync uses the priority plane, bypassing the outbound queue — use it for small, latency-sensitive control messages, not bulk data.

Request/reply

To send a message and await a typed answer, use RequestAsync<TReply>:

var request = node.Messaging.CreateMessage("orders.reserve", new ReserveRequest { Sku = "ABC", Qty = 3 });

ReserveResult? result = await node.Messaging.RequestAsync<ReserveResult>(
    "node-2", request, timeout: TimeSpan.FromSeconds(5));

Under the hood the request is sent on the priority plane and the caller waits for a reply matched by CorrelationId. The responder must send back a message carrying the same correlation id. Because that requires reading the raw message, the responder side uses a low-level IMessageHandler:

public sealed class ReserveHandler : IMessageHandler
{
    private readonly INodusNode _node;
    public ReserveHandler(INodusNode node) => _node = node;

    public string Type => "orders.reserve";
    public MessageDispatchMode DispatchMode => MessageDispatchMode.Inline;

    public async Task HandleAsync(Message request)
    {
        var req    = /* deserialize request.Payload */ ;
        var result = Reserve(req);

        var reply = _node.Messaging.CreateMessage("orders.reserve.reply", result);
        reply.CorrelationId = request.CorrelationId;          // echo it so the caller's await completes
        await _node.Messaging.SendImmediateAsync(request.SenderId, reply);
    }
}

await node.Messaging.AddHandler(new ReserveHandler(node));

RequestAsync throws TimeoutException if no matching reply arrives within the timeout (the default is 50 seconds when you pass none). The request/reply how-to covers the full pattern, timeouts, and error handling.

Choosing the right tool

Use the decision below to pick between direct messaging and cluster events, and within messaging, between the point-to-point, broadcast, and request/reply shapes.

Need	Use
Send work to a specific node	SendAsync<T>
Tell every member something	BroadcastAsync<T>
Ask one node and wait for an answer	RequestAsync<TReply>
Full control over the wire message	CreateMessage + SendAsync(Message)
Announce to decoupled subscribers	node.Events

Next steps

• Cluster events — the decoupled pub/sub alternative to direct messaging.
• Sending messages — the task-focused how-to.
• Request/reply — timeouts and error handling in depth.