UA-.NETStandard/docs/HighAvailability.md

# High Availability and OPC UA Redundancy

This guide explains how the OPC UA .NET Standard high-availability APIs
implement OPC 10000-4 §6.6 Redundancy. It covers server, client,
subscription, and session redundancy, plus [Kubernetes](Kubernetes.md)
and active/active extensions. Worked examples are in
`samples/Redundancy/RedundantServer` and
`samples/Redundancy/RedundantClient`.

Redundancy and high availability are opt-in. Add the relevant
`OPCFoundation.NetStandard.Opc.Ua.Redundancy.*` NuGet package to your
application, such as `OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server`
or `.Client`. The standard `OPCFoundation.NetStandard.Opc.Ua.Client` and
`OPCFoundation.NetStandard.Opc.Ua.Server` libraries alone do not provide
OPC UA redundancy.

For distributed PubSub active/standby publishers and subscribers, see the PubSub counterpart: [PubSub High Availability](PubSubHighAvailability.md).

## Contents

- [Redundancy overview (as per Part 4 §6.6.1)](#redundancy-overview-as-per-part-4-661)
- [Server redundancy (as per Part 4 §6.6.2)](#server-redundancy-as-per-part-4-662)
  - [Server.ServerRedundancy model](#serverserverredundancy-model)
  - [Add* and Use* API convention](#add-and-use-api-convention)
- [ServiceLevel and load balancing (as per Part 4 §6.6.2.4.2 and §6.6.2.4.3)](#servicelevel-and-load-balancing-as-per-part-4-66242-and-66243)
- [Non-transparent failover modes and client actions (as per Part 4 §6.6.2.4.5)](#non-transparent-failover-modes-and-client-actions-as-per-part-4-66245)
- [Manual failover and Maintenance (as per Part 4 §6.6.5)](#manual-failover-and-maintenance-as-per-part-4-665)
- [HotAndMirrored and Transparent state mirroring](#hotandmirrored-and-transparent-state-mirroring)
  - [Active/passive address-space consistency](#activepassive-address-space-consistency)
  - [Strong active/passive historian](#strong-activepassive-historian)
- [Client redundancy (as per Part 4 §6.6.3)](#client-redundancy-as-per-part-4-663)
- [Network redundancy (as per Part 4 §6.6.4)](#network-redundancy-as-per-part-4-664)
- [Beyond §6.6: distributed extensions](#beyond-66-distributed-extensions)
  - [Dynamic peer discovery (beyond §6.6, opt-in)](#dynamic-peer-discovery-beyond-66-opt-in)
  - [Sharing values across replicas: distributed value cache (beyond §6.6, opt-in)](#sharing-values-across-replicas-distributed-value-cache-beyond-66-opt-in)
  - [Shared certificate stores (distributed trust lists) (beyond §6.6, opt-in)](#shared-certificate-stores-distributed-trust-lists-beyond-66-opt-in)
  - [Distributed PushManagement transactions (beyond §6.6, opt-in)](#distributed-pushmanagement-transactions-beyond-66-opt-in)
  - [GetEndpoints load direction (beyond §6.6, opt-in)](#getendpoints-load-direction-beyond-66-opt-in)
  - [Client-side high availability (replica sets)](#client-side-high-availability-replica-sets)
- [Kubernetes deployment](#kubernetes-deployment)
- [Samples](#samples)
- [Security considerations](#security-considerations)
  - [Record context and plaintext ownership](#record-context-and-plaintext-ownership)
  - [Shared application identity](#shared-application-identity)

## Redundancy overview (as per Part 4 §6.6.1)

OPC UA defines three independent but composable redundancy dimensions: **server redundancy** gives clients multiple servers that expose the same data, **client redundancy** lets backup clients take over work from an active client, and **network redundancy** gives a client and server more than one communication path. The stack implements standardized discovery and failover metadata in the core server/client packages, and adds opt-in distributed state through `OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server`.

Server redundancy is either **transparent** or **non-transparent**. In transparent redundancy, the redundant set looks like one server and failover is hidden from the client. In non-transparent redundancy, each server has its own identity and endpoint, and the client reads `Server.ServerRedundancy` plus `Server.ServiceLevel` to decide what to do.

## Server redundancy (as per Part 4 §6.6.2)

`Opc.Ua.RedundancySupport` is the stack enum published by `Server.ServerRedundancy.RedundancySupport` and consumed by the client. The modes are:

| Mode | OPC UA meaning | Stack behavior |
| --- | --- | --- |
| `None` | No server redundancy is advertised. | Single-server default. |
| `Cold` | Only one server is active at a time; backups may be unavailable or not running. | Client connects to one server and creates a fresh session/subscriptions on failover. |
| `Warm` | Backup servers are running but have less functionality or no process data. | Client can connect to peers, keep subscriptions created but sampling/publishing inactive on backups, and promote the highest-`ServiceLevel` server. |
| `Hot` | Multiple servers are running and can independently provide data. | Client connects to peers and either hands reporting over to one active server or merges reporting streams. |
| `HotAndMirrored` | Hot servers mirror communication state. | `ManagedSession` keeps one active session, may keep lightweight ServiceLevel status-check sessions to backups, and fails over by opening a channel and re-activating the mirrored session with the existing `AuthenticationToken`; subscriptions are not recreated client-side. |
| `Transparent` | One virtual server identity hides physical failover. | Server publishes `CurrentServerId`; deployment infrastructure supplies the virtual address and shared certificate/endpoint identity. |

### Server.ServerRedundancy model

Register `AddServerRedundancy(...)` on the server builder to populate the live `Server.ServerRedundancy` nodes after the server starts. `ServerRedundancyOptions.Mode` writes `RedundancySupport`. `PeerServerUris` and `RedundantPeers` populate `RedundantServerArray`; for non-transparent modes the startup task also populates `ServerUriArray`, and for transparent mode it publishes `CurrentServerId`.

`Server.ServerRedundancy` is exposed as the generated subtype for the configured mode — `TransparentRedundancyType` for `Transparent`, `NonTransparentRedundancyType` for `Cold`/`Warm`/`Hot`/`HotAndMirrored`, or the base `ServerRedundancyType` for `None` — so a client that browses the node sees the correct `HasTypeDefinition` and only the members the standard defines for that subtype (`CurrentServerId` for transparent, `ServerUriArray` for non-transparent).

The mode can be changed at runtime through the injectable `IServerRedundancyController.ChangeModeAsync(mode)`: the node is re-materialized as the new subtype in place and a `ModelChange` is raised, so connected clients observe the new type definition and members without a restart.

For non-transparent redundancy, set `RedundantPeers` when clients should resolve peers through `FindServers`; `ConfiguredRedundantServerSetProvider` exposes those `ApplicationDescription` entries server-side. `AdvertiseNtrsCapability` defaults to `true`, so non-transparent modes add the `NTRS` discovery capability for GDS/NTRS registration.

All servers in a `RedundantServerSet` must have identical application AddressSpaces: identical NodeIds, browse paths, AddressSpace structure, and `ServiceLevel` algorithm. Only local server diagnostics may differ. `UseDistributedAddressSpace(...)` helps satisfy this by mirroring node topology and values through `INodeStateStore`/`ISharedKeyValueStore`, but application-specific method handlers and callbacks still need to be attached by each node manager.

Configure [`UseReplicaNodeIdentity`](ReplicaNodeIdentity.md) for replicated
address spaces in either mode. It reserves identical shared namespace indexes
before node-manager construction and binds the standard factory's assignment
policy across replicas. The guarantee concerns raw wire NodeIds, not client-side
URI remapping. Incompatible layouts, modes and peer/store contracts fail
explicitly; built-in diagnostics and configuration remain local.

### Add* and Use* API convention

High-availability builder methods follow the stack's standard `Add*`/`Use*` convention: `Add*` methods wire OPC 10000-4 §6.6 nodes and methods that are part of the standardized server model, while `Use*` methods register beyond-spec extension building blocks that make a redundant deployment work. For the stack-wide DI conventions and per-package entry points, see [DependencyInjection.md](DependencyInjection.md).

Standardized redundancy nodes and methods (`Add*`):

| API | Description |
| --- | --- |
| `AddServerRedundancy(options)` | Publishes the `Server.ServerRedundancy` metadata (RedundancySupport, `ServerUriArray` / `RedundantServerArray`, `NTRS` capability). Metadata only — it does not drive `ServiceLevel`. |
| `AddServerServiceLevel(provider)` | Drives the `Server.ServiceLevel` node from an `IServiceLevelProvider` (for example `LeaderServiceLevelProvider`; use a `ConstantServiceLevelProvider` for a fixed level). |
| `AddRequestServerStateChange(options)` | Enables the standard `Server.RequestServerStateChange` method for administrator-driven Maintenance / NoData failover. |

Beyond-spec distributed building blocks (`Use*`):

| API | Description |
| --- | --- |
| `UseReplicaNodeIdentity(replicaSetId, namespaceUris, mode, writerAssignedIds)` | Required shared identity contract for replica sets: fixed namespace slots, guarded standard factory and unchanged hydrated IDs. |
| `UseDistributedAddressSpace(options)` | Active/passive shared-store address-space replication with leader election (one writer; standbys hydrate from the shared store). Also registers the injectable `IDistributedValueCache` and the `SharedKeyValue` certificate-store support. |
| `UseDistributedSessions(options)` | Mirrors session metadata to the shared store for fast reconnect after failover (opt-in `EnableFastReconnect`). |
| `UseDistributedSubscriptionMirroring()` | Mirrors subscription / monitored-item definitions, retransmission state and continuation-point envelopes to the shared store. |
| `UseDistributedHistorian(options)` | Adds the protected, leader-write active/passive historian and durable portable HistoryRead continuation store. Requires a cross-process linearizable store, authenticated `IRecordProtector`, and leader election; active/active and eventual configurations fail closed. |
| `UseReplicatedAddressSpace(options)` | Active/active multi-writer address-space replication over CRDT gossip (no leader). |
| `UseReplicatedSessions(options)` | Active/active CRDT-gossiped session metadata for cross-replica fast reconnect. |
| `UseActiveActiveRedundancy(options)` | Single-call active/active: wires the replicated address space **and** sessions from one set of gossip options (session gossip on `GossipPort` + 1). This is the entry point that "selects active/active"; the two `UseReplicated*` methods remain for advanced setups. |
| `UseRedundancyConsistency(options)` | Selects the shared-store consistency model for the whole set — eventual (CRDT + Raft strong keyspaces) or strong (Raft). Call it before the other `Use*` methods so they compose over the chosen store. |
| `UseServerLoadDirection(options)` | Opt-in GetEndpoints load direction: answers discovery on a balancing URL with the best replica's endpoints. |
| `UsePeerDiscovery(factory)`, `UseStaticPeerDiscovery(peers)`, `UseDnsPeerDiscovery(options)`, `UseLdsPeerDiscovery(findServers)` | Dynamic peer discovery for the client-facing `RedundantServerSet` (`FindServers`) and, for active/active, the CRDT gossip fabric. Static configuration (`AddServerRedundancy`) is the fallback used until discovery finds peers. |
| `UseKubernetesRaftConsensus(...)` | (`Opc.Ua.Redundancy.Kubernetes`) Builds the Raft cluster from the StatefulSet ordinal and headless-Service DNS. |
| `UseKubernetesPeerDiscovery(...)` | (`Opc.Ua.Redundancy.Kubernetes`) Discovers peer ServerUris from EndpointSlices, bridged into the generic peer-discovery seam. |

`AddServerRedundancy(...)` only publishes the redundancy metadata. It does not calculate or drive `Server.ServiceLevel`; register a ServiceLevel provider with `AddServerServiceLevel(...)`, or register an `IServiceLevelProvider` plus `ServiceLevelStartupTask`, when clients and Kubernetes readiness need live health or leader-state values.

On the standard server, `ServiceLevelStartupTask` claims explicit ownership via
`IServerServiceLevelControl`. Only one provider can claim the node for that
server's lifetime; a second claim fails instead of creating competing writers.
Provider publications and session-headroom updates share the server's internal
coordination, and session churn cannot overwrite any provider-owned value,
including Healthy-band standby/load levels such as 210 or a fixed 255.
Without an explicit owner, the session-headroom heuristic is unchanged.
Custom `IServerContext` implementations can implement this optional capability;
contexts without it retain their existing direct provider-publication behavior.

```csharp
services.AddOpcUa()
    .AddServer(server =>
    {
        // normal endpoint, security, and application configuration
    })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseDistributedAddressSpace(options =>
    {
        options.UseLeaderElection = true;
        options.NodeId = Environment.MachineName;
    })
    .AddServerRedundancy(options =>
    {
        options.Mode = RedundancySupport.Warm;
        options.CurrentServerId = Environment.MachineName;
        options.AddRedundantPeer(
            "urn:example:ha-server-2",
            new ArrayOf<string>(["opc.tcp://ha-server-2:4840"]));
        options.PeerServiceLevel = ServiceLevels.DegradedMaximum;
    })
    .AddServerServiceLevel(new LeaderServiceLevelProvider(leaderElection, RedundancySupport.Warm))
    .AddRequestServerStateChange(options =>
    {
        options.ServiceLevelSelector = state => state == ServerState.Running
            ? ServiceLevels.Maximum
            : ServiceLevels.Maintenance;
    });
```

## ServiceLevel and load balancing (as per Part 4 §6.6.2.4.2 and §6.6.2.4.3)

OPC 10000-4 Table 105 defines `ServiceLevel` as a byte split into mandatory sub-ranges. The stack exposes these constants and helpers in `ServiceLevels` and `ServiceLevelSubrange`.

| Range | Name | Client interpretation |
| --- | --- | --- |
| `0` | Maintenance | New clients should not connect; connected clients should disconnect and use `EstimatedReturnTime` before retrying. |
| `1` | NoData | Server is not operational for process data; clients may connect only for status/diagnostics. |
| `2..199` | Degraded | Server is partially operational; clients should prefer any Healthy peer and may choose a higher degraded peer when no Healthy peer exists. |
| `200..255` | Healthy | Server is fully operational; clients connect to the highest value for selection/load balancing and should not fail over while the current server remains Healthy. |

`ConstantServiceLevelProvider` reports a fixed value, defaulting to `255`, preserving single-instance behavior. `LeaderServiceLevelProvider` follows an `ILeaderElection`: the leader reports `255`, Cold standbys report `1`, Warm standbys report `199`, and Hot/HotAndMirrored standbys report `255` unless explicit levels are supplied. Optional health and connected-client delegates cap or decrement Healthy values so Hot servers can load-balance within the 200-255 range. `IServiceLevelController` lets `RequestServerStateChange` override the published value for manual maintenance.

`SharedStoreLeaseElection` bounds local authority by the last successfully
confirmed lease. Its injected `TimeProvider` drives both renewal and an
independent expiry timer, so failed or blocked store operations cannot keep a
replica authoritative past that deadline. `IsLeader` also checks the deadline
when read, without invoking application callbacks. The expiry timer raises
`LeadershipChanged(false)` to update dependent service levels. Subscriber
failures are logged without interrupting other subscribers or lease operations.
Lease validity starts at the write attempt, not at receipt of
its reply; an expired or superseded operation cannot restore leadership. A
fresh, confirmed acquisition is required after expiry. Overlapping successful
calls can confirm the same owned lease without waiting for one another.
A failed store observation started before a newer successful confirmation cannot
revoke or reschedule that confirmed lease; a fresh ownership-loss observation
still revokes authority. Expiry and disposal invalidate all outstanding attempts.
The UTC lease record is also bounded by local elapsed time, so moving the local
clock backwards cannot extend authority. Replicas still require unique identities,
suitably synchronized clocks and a linearizable compare-and-swap store; this local
safety mechanism does not replace backend fencing or provide consensus.

Client-side, `DefaultServerRedundancyHandler.FetchRedundancyInfoAsync` reads `RedundancySupport`, `ServiceLevel`, `EstimatedReturnTime`, `RedundantServerArray`, `ServerUriArray`, and `CurrentServerId` as applicable. `ServerRedundancyInfo.ServiceLevelSubrange` is calculated with `ServiceLevels.GetSubrange`.

`ManagedSession` stores the default handler's basic snapshot as soon as that
read succeeds, within the existing two-second metadata-read bound. Peer discovery
runs separately in the background, in parallel with a two-second bound per peer;
connection establishment does not wait for every backup. A peer timeout leaves
its `Endpoint` unresolved without discarding the mode, peer URIs or other resolved
endpoints. Caller cancellation is propagated, and session disposal cancels and
drains the background refresh.

```csharp
ManagedSession session = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(endpoint)
    .WithServerRedundancy(new DefaultServerRedundancyHandler())
    .ConnectAsync(ct);

var handler = new DefaultServerRedundancyHandler();
ServerRedundancyInfo info = await handler.FetchRedundancyInfoAsync(session, ct);
Console.WriteLine($"Mode={info.Mode}, ServiceLevel={info.ServiceLevel} ({info.ServiceLevelSubrange})");

ServerFailoverDecision decision = handler.ShouldFailover(info, session.ConfiguredEndpoint);
if (decision.IsFailoverWarranted)
{
    ConfiguredEndpoint? target = handler.SelectFailoverTarget(info, session.ConfiguredEndpoint);
    Console.WriteLine($"Fail over to {target?.EndpointUrl}: {decision.Reason}");
}
```

## Non-transparent failover modes and client actions (as per Part 4 §6.6.2.4.5)

`ManagedSession` implements the Table 107 client patterns over `ManagedSession` instances discovered from `RedundantServerArray`/`ServerUriArray` and resolved with `IRedundantServerEndpointResolver`. The default resolver uses `FindServers` at the current endpoint to locate peers, then requires `GetEndpoints` results to match the peer application URI, security policy/mode and URL scheme.

The resolver retains a peer's discovery addresses before contacting that peer.
Thus a Cold backup that was offline at connect time can be resolved at failover
without another `FindServers` call to the failed primary. Failed resolutions are
not cached as permanent misses, and a failed failover invalidates the resolved
endpoint so the next attempt can discover it again. Cached addresses never bypass
endpoint or session security validation. If no peer address was learned before
the primary became unreachable, a reachable discovery service or a custom
`IRedundantServerEndpointResolver` is still required to translate its application URI.

| Mode | Table 107 actions | `ManagedSession` realization |
| --- | --- | --- |
| Cold | Initial connection to one active server. At failover open a SecureChannel, create/activate a session, create subscriptions/monitored items, activate sampling, then activate publishing. | Starts with the initial endpoint only. `FailoverAsync` selects a peer, connects it, applies subscriptions only to the current session, and enables reporting/publishing. |
| Warm | Connect to more than one server and create subscriptions/monitored items on backups; sampling and publishing become active at failover. | Connects all peers, stores templates with `AddSubscriptionAsync`, creates subscriptions on each connected server, and keeps only the selected server in Reporting/publishing while backups are Sampling/publishing disabled. |
| Hot (a) | Connect to more than one server, create subscriptions everywhere, activate sampling everywhere, publish from one server; at failover activate publishing on the next server. | Default `HotRedundancyNotificationMode.ReportingHandoff`: all connected peers host subscriptions; current server Reports and publishes, backups Sample only. |
| Hot (b) | Connect to more than one server, create subscriptions everywhere, activate sampling and publishing everywhere; client handles duplicate streams. | `HotRedundancyNotificationMode.ReportingMerge`: all connected peers Report and publish; duplicate suppression/merge is a client responsibility above the event callback. |
| HotAndMirrored | Client normally connects to one server; optional status sessions only; on failover open a SecureChannel and call `ActivateSession` against the mirrored session. | Keeps a single active session and, when `EnableHotAndMirroredStatusChecks` is set, periodically reads ServiceLevel from backup status sessions. `FailoverAsync` reuses the mirrored `AuthenticationToken` and single-use nonce by calling `ActivateSession` on the backup; it does not recreate subscriptions because the server mirrors session and subscription state. |

`ManagedSession.RefreshServiceLevelsAsync` reads service levels from connected sessions and selects the highest value. The default handler will not fail over away from a server that is still in the Healthy sub-range, defers retries for Maintenance using `EstimatedReturnTime` or a longer default backoff, and selects a Running peer with an operational `ServiceLevel`.

```csharp
ManagedSession coldClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(ct: ct);

await coldClient.AddSubscriptionAsync("process", subscriptionTemplate, ct);
await coldClient.FailoverAsync(ct);
```

```csharp
ManagedSession warmClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(ct: ct);

await warmClient.AddSubscriptionAsync("process", subscriptionTemplate, ct);
await warmClient.RefreshServiceLevelsAsync(ct);
```

Hot handoff is the default Hot mode: every connected peer hosts the subscriptions and samples, but only the active server publishes, so the client sees a single stream and the next server starts publishing on failover. Use it for fast failover without client-side de-duplication.

```csharp
ManagedSession hotHandoffClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(
        new ManagedSessionOptions
        {
            HotNotificationMode = HotRedundancyNotificationMode.ReportingHandoff
        },
        ct);
```

Hot merge has every connected peer publish at once, so the client receives duplicate streams with near-zero failover latency and merges or de-duplicates them in the `NotificationReceived` handler (by application key, source timestamp, EventId, or sequence context). Use it when the lowest possible failover gap is worth handling duplicates.

```csharp
ManagedSession hotMergeClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(
        new ManagedSessionOptions
        {
            HotNotificationMode = HotRedundancyNotificationMode.ReportingMerge
        },
        ct);

hotMergeClient.NotificationReceived += (sender, args) =>
{
    // Merge or de-duplicate streams by application key, source timestamp, EventId, or sequence context.
};
```

For a `HotAndMirrored` server, the client keeps one active session and, with `EnableHotAndMirroredStatusChecks`, periodically polls the backups' `ServiceLevel` over lightweight status-check sessions so it can pick the healthiest survivor. Because the server mirrors session and subscription state, failover re-activates the mirrored session on the survivor and does not recreate subscriptions. Use it when the server set runs `HotAndMirrored` (or `Transparent`) mirroring.

```csharp
ManagedSession mirroredClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(
        new ManagedSessionOptions
        {
            EnableHotAndMirroredStatusChecks = true,
            HotAndMirroredStatusCheckInterval = TimeSpan.FromSeconds(5)
        },
        ct);
```

## Manual failover and Maintenance (as per Part 4 §6.6.5)

OPC 10000-4 §6.6.5 allows a server to be taken out of the set by shutdown or by moving `ServiceLevel` to Maintenance, either through a vendor tool or `Server.RequestServerStateChange`. `AddRequestServerStateChange(...)` wires the standard method and validates administrative access through `IConfigurationNodeManager.HasApplicationSecureAdminAccess` unless `RequestServerStateChangeOptions.AdminAccessValidator` is supplied.

The startup task updates `Server.ServiceLevel`, `Server.ServerStatus.State`, `Server.ServerStatus.SecondsTillShutdown`, `Server.ServerStatus.ShutdownReason`, and `Server.EstimatedReturnTime`. The current implementation publishes the requested maintenance/no-data state so clients back off; it does **not** install a transport-level hook that rejects newly created sessions.

Non-transparent servers registered with `AddServerRedundancy(...)` advertise the `NTRS` capability by default. In Kubernetes deployments, register the non-transparent set or transparent virtual address with GDS/NTRS according to your discovery model.

## HotAndMirrored and Transparent state mirroring

OPC 10000-4 requires HotAndMirrored and Transparent sets to synchronize enough state that a client can continue after failover. The spec names sessions, subscriptions, registered nodes, continuation points, sequence numbers, sent notifications, and synchronized EventIds.

Every replica reads and writes that state through a shared store. One replica serves the client's active session; the others stand by and continuously mirror the session, subscription, retransmission, continuation-point, EventId, and node-state records so any standby can resume the client after a failover. Records are serialized through `IRecordProtector`, so the store is treated as an untrusted conduit.

```mermaid
flowchart LR
    Client -->|active session| R1["Replica 1 (active)"]
    R1 -. mirror .-> Store[("Shared store<br/>(IRecordProtector)")]
    R2["Replica 2 (standby)"] -. mirror .-> Store
    R3["Replica 3 (standby)"] -. mirror .-> Store
    R1 -. failover .-> R2
```

You enable mirroring by registering the seams below on the server builder; each one mirrors one category of state and is independently opt-in:

- `UseDistributedSessions(...)` installs `DistributedSessionManager` and `ISharedSessionStore`. Session records include the session id, authentication token, nonces, client certificate chain, security policy/mode, endpoint URL, session timeout, client description, and user identity material. `EnableFastReconnect` defaults to `false`; when enabled, a failover reconnect still performs full `ActivateSession` signature validation against the mirrored server nonce. (`UseDistributedSessionMirroring(...)` is the mirroring-only variant: it mirrors the session record — keyed by a digest of the authentication token and protected by the configured `IRecordProtector` — without replacing the session manager.)
- `ISingleUseNonceRegistry` and `SharedSingleUseNonceRegistry` enforce the security boundary: the mirrored `serverNonce` is consumed exactly once across the replica set, and the authentication token is only a lookup key. The nonce is consumed only after the restored activation has passed the client signature, user identity and continuity checks, so a failed or abandoned attempt (a wrong identity, a slow identity provider, a client-side timeout) leaves the session restorable on this or another replica; a restored session whose first activation fails is discarded again without touching the shared entry.
- The replica that serves a session owns its record. While the client uses the session, the owner mirrors a liveness heartbeat (`DistributedSessionOptions.LivenessInterval`, default one second; written at most once per quarter of the session timeout), so a session in normal use stays restorable long after its last activation — a session times out after its last Service request, not its last activation (OPC 10000-4 5.7.2.1). Only the owner mirrors activations and deletes the record. A restore takes ownership with a conditional (compare-and-swap) write; the previous replica then drops its stale copy without deleting the record, also when the client fails back to it. Over a store without compare-and-swap (the CRDT gossip store) the conditional writes fall back to compare-then-write.
- A restored session keeps the `SessionId` the client received from `CreateSession`, so its diagnostics nodes, audit events and mirrored continuation points stay addressable across any number of failovers. To make that safe, `DistributedSessionManager` assigns new sessions a GUID `SessionId` that is unique across the replica set instead of a per-process counter value.
- Replicas of a non-transparent (HotAndMirrored) set have their own ApplicationUri and certificate. The record carries the certificate of the server that created the session, and a restored session accepts a client signature over either that certificate or the restoring replica's own one (the stack's client signs the failover server's certificate). Encrypted user tokens must still be decryptable by the restoring replica: with RSA user token policies the client encrypts for the certificate it signs, and a session whose user token is encrypted with an ECC or RSA-DH `EphemeralKey` is not restored at all, because the private part of that key never leaves the replica that issued it — the reconnect is rejected before anything is consumed and the client re-creates the session.
- A periodic sweep (`SweepInterval`, default five minutes) removes the records of sessions that can no longer be restored — for example those of a replica that crashed or shut down, which nobody closes — and the consumed-nonce markers that no live record references and that are older than `NonceMarkerRetention` (default one hour). A restore that finds an expired record deletes it as well.
- `SharedKeyValueSessionStore` and `DistributedSessionManagerFactory` refuse an external (non in-memory) store without an `IRecordProtector`, like the dependency injection registration; pass `NullRecordProtector.Instance` explicitly to knowingly store session secrets unprotected.
- Mirrored session records carry a `SecurityStateVersion`. A replica only restores a Session from a record written at the version it understands; a record written by a peer running an older version is treated as missing security state and fails closed, so a rolling upgrade degrades to a normal Session recreate rather than comparing identity state it cannot interpret. Version 4 also mirrors whether the client application certificate passed validation when the Session was created; a Session whose certificate error an `OnApplicationCertificateError` override accepted is restored without the `TrustedApplication` role or application-based role mappings, like the original. A version 3 record is still restored, with its certificate treated as not validated. Version 5 adds the liveness heartbeat, the owning replica, the original server certificate and whether the user token needs an `EphemeralKey`; version 3 and 4 records are still restored, without an owner. The stored user identity is an encoded continuity key (token type plus a delimited issuer and subject), not the human-readable `ClientUserIdOfSession` diagnostic value. Activation records are stamped with a per-Session activation sequence so that when two activations overlap the mirror keeps the newest `serverNonce` instead of whichever write happens to land last.
- `UseDistributedSubscriptionMirroring(...)` registers `SharedKeyValueSubscriptionStore` as `ISubscriptionStore`. It mirrors subscription definitions and monitored-item definitions: publishing interval, lifetime, keepalive, priority, node id, attribute id, monitoring mode, sampling interval, queue size, filters, discard policy, and related metadata. It also installs `SharedKeyValueMonitoredItemQueueFactory` as the `IMonitoredItemQueueFactory`, which continuously mirrors each monitored item's data/event queue contents to the shared store through a non-blocking, coalesced background drain (protected at rest by the configured `IRecordProtector`); on promotion the store's asynchronous restore members re-hydrate those queues so queued-but-unpublished notifications survive a failover.
- The same store also implements `ISubscriptionRetransmissionStore`. Retransmission state is mirrored asynchronously through a non-blocking background drain: `NextSequenceNumber`, sent `NotificationMessage` entries, acknowledgements, and deletes are coalesced and persisted so `Republish` can continue after failover without blocking the publishing path.
- `IContinuationPointStore` mirrors the generic Browse continuation envelope.
- `IHistoryContinuationPointStore` separately persists versioned, protected
  HistoryRead state for providers that expose a stable
  `IHistorianProviderIdentity` and portable resume tokens. Persistence is
  awaited before the continuation point is returned, and restore atomically
  claims the record so only one replica can resume it.
- `IEventIdProvider` is an optional event-publishing seam. `DeterministicEventIdProvider` derives EventIds from a shared replica-set seed and stable event fields so Transparent/HotAndMirrored replicas can publish the same logical EventId and clients do not double-process events; the default remains the existing random GUID EventId behavior.
- `RegisterNodes` returns the input NodeIds in this stack, so registered-node handles are already replica-consistent when the AddressSpace NodeIds are identical.
- `UseDistributedAddressSpace(...)` mirrors node topology and values through `INodeStateStore`, helping each server present the same NodeIds and browse paths. Each node manager is restricted to its declared non-standard `NamespaceUris`; namespace-zero infrastructure remains replica-local, and a replicated root may not contain foreign descendants. Custom and shared-namespace partitions implement `ILocalAddressSpaceOwnership` with a stable `PartitionId` and an `OwnsNode(NodeId)` predicate. Hydration updates existing node graphs in place, preserving concrete node types, callbacks, and handles. Tracked subtree membership is independent of mutable child links, so deleting or replacing a graph also detaches its old replication callbacks.

  On stores with authoritative reads, a protected partition marker distinguishes
  a completed empty partition from an interrupted initial seed. Validated
  snapshots and deltas provide a bulk-hydration path; missing or invalid snapshot
  chunks trigger complete streamed fallback. A corrupt record under a valid
  node/value key fails that scan rather than becoming an authoritative omission.
  Failed snapshot publication leaves the previous manifest and delta log intact.

  Startup buffers the live feed during hydration. The most recent accepted
  topology sequence determines both graph changes and root membership; rejected
  stale deletions cannot prune newer roots during cleanup. Value ordering remains
  independent of topology ordering. Values arriving before their target node are
  retained and applied when its topology becomes available, rather than being
  marked as delivered while the node is absent. An accepted parent tombstone
  prevents older pending values from crossing into recreated descendants.
  See the consistency requirements below for
  eventually consistent payload stores, where absence is not authoritative.

Notes:

- Hydration fully materializes the mirrored node graph (topology plus values): the snapshot + delta log reduces the store round trips and per-node work, but the whole graph is resident. True on-demand fault-in (materializing a node only when it is first browsed/read) would further cut time-to-ready and memory for very large, sparsely-accessed graphs; it requires an asynchronous node-resolution seam through the core node-manager read/browse path and is tracked as future work in OPCFoundation/UA-.NETStandard#3938. The eventual-consistency conflict-free replicated data type (CRDT) path already exchanges a compact state snapshot plus deltas.
- `SharedKeyValueSubscriptionStore` restores subscription definitions, retransmission state, and — when the shared-store `SharedKeyValueMonitoredItemQueueFactory` is installed (the default for `UseDistributedSubscriptionMirroring`) — the per-monitored-item data/event queues. The monitored-item creation path pre-fetches each queue asynchronously through `ISubscriptionStore.RestoreDataChangeMonitoredItemQueueAsync`/`RestoreEventMonitoredItemQueueAsync` and hands the pre-hydrated queue to the (still synchronous) `MonitoredItem` constructor, so a networked store never blocks the creation path; the original synchronous `RestoreDataChangeMonitoredItemQueue`/`RestoreEventMonitoredItemQueue` remain as the fallback for local/durable stores. Queue contents are mirrored continuously as they are enqueued, so after a failover the promoted replica resumes with the values that were queued-but-not-yet-published on the failed replica instead of losing them; `Republish` of already-sent notifications is likewise preserved. Snapshots are coalesced per monitored item (the latest state wins), so the mirror cost scales with the unpublished queue tail rather than every sampled value.
- Generic Browse continuation-point mirroring remains best-effort because a
  custom node manager's opaque `ContinuationPoint.Data` cannot be reconstructed
  automatically. Built-in historian continuations are recoverable when the
  provider identity matches on the promoted replica and its resume token is
  explicitly portable; otherwise HistoryRead fails closed with
  `BadContinuationPointInvalid`.
- Deterministic EventIds (`DeterministicEventIdProvider`) are opt-in rather than on by default: they change the EventId values a server emits — a single-server deployment keeps the standard random GUID EventIds — and they require a shared replica-set seed. Enable them for a HotAndMirrored/Transparent set so every replica emits the same logical EventId and clients do not double-process events. They are only as stable as the event fields used, so Alarms & Conditions clients should still call `ConditionRefresh` after failover as required by OPC UA.

### Active/passive address-space consistency

`InMemoryNodeStateStore` requires a linearizable coordinator for
`election/addressspace-sequence`. A bare `ReplicatedSharedKeyValueStore` cannot
provide this primitive. Use `UseRedundancyConsistency(...)` before
`UseDistributedAddressSpace(...)`: the hybrid mode keeps CRDT payloads on the
bulk store and routes the sequence key to shared Raft. The address-space
registration contributes its coordination and configured lease keys even when
the application's strong-prefix list is customized. Startup validates the
coordinator before election or shared-state mutation; direct writes enforce the
same requirement. A process-local coordinator is allowed only with
process-local payload storage, not as a substitute for shared coordination.

For direct construction, explicitly compose existing stores:

```csharp
await using var hybrid = new HybridSharedKeyValueStore(
    crdtPayloadStore, sharedRaftStore);
using var stateStore = new InMemoryNodeStateStore(
    hybrid, messageContext, recordProtector);
```

The identity contract is protected at `election/addressspace-identity/v1` and
also requires strong routing. A new authoritative store initializes it atomically.
An existing unbound store is rejected. For a verified new hybrid deployment,
provision the matching contract on the fresh shared Raft backend before attaching
the also-new CRDT payload backend; an empty eventual scan cannot authorize
bootstrap. See [stored identity contracts](ReplicaNodeIdentity.md#stored-contracts-and-peer-admission).

Both backend instances above are supplied by the application and retain their
existing disposal ownership. Every writer replica must use the same replicated
Raft coordination state, not a separate single-node development coordinator.

Strong sequence allocation does not make CRDT payload reads linearizable.
With CRDT bulk data, hydration applies observed updates and sequenced tombstones
but does not infer deletion, initial seeding, or partition initialization from
missing rows. Local predefined graphs must be provisioned independently.
Automatic snapshots/compaction are unavailable, explicit snapshot publication
is rejected, and deltas are retained so a delayed older primary-row write
cannot hide a newer completed update. Select `RedundancyConsistencyMode.Strong`
when authoritative bootstrap, missing-root cleanup, and compacted snapshots are
required. Coordinator-free active/active replication remains the separate
`UseReplicatedAddressSpace(...)` module.

Sequence reservations are not completed publications. Protected pending
reservations survive uncertain writes and prevent snapshot publication or log
trimming past unfinished work, including after restart. Concurrent snapshot
attempts fail instead of publishing an unvalidated cut. Failed or cancelled
publications require reconciliation before compaction can resume; the module
does not automatically discard their reservations or assume the payload was
never written.

### Strong active/passive historian

`UseDistributedHistorian(...)` places raw, modified, at-time, processed, annotation, structured, and event history behind the same linearizable `ISharedKeyValueStore` used by the strong active/passive topology. Immutable segments are published by one manifest compare-and-swap. Each provider instance has a unique writer id, and a newly active writer publishes a higher writer epoch before it can append; a stale former writer is then rejected with `BadNotWritable`.

HistoryRead cursors pin an immutable manifest generation. The separate `IHistoryContinuationPointStore` persists the versioned, protected server cursor before returning it, then atomically claims it during restore. A token-reuse session takeover therefore preserves the continuation owner and lets the promoted replica resume raw, event, and processed reads without shifting pages, duplicates, or gaps. Superseded generations are retained for the configured continuation lifetime.

This module intentionally has no eventual or active/active adapter. Ordered historical values, deletes, modification chains, events, and atomic update batches do not have a safe CRDT merge contract, so requesting the distributed historian outside a strong active/passive topology fails startup. `NullRecordProtector` is also rejected: shared archive and continuation records must be authenticated.

The redundancy samples exercise both guarantees: the client writes and reads a distinctive raw-history marker through the active replica, resumes paged raw, event, and processed reads after terminating that process, then reads the same marker and observes new raw/event writes through the promoted replica.

## Client redundancy (as per Part 4 §6.6.3)

OPC UA client redundancy is implemented with `TransferSubscriptions` plus server diagnostics. `ClientFailoverCoordinator` helps a backup client find the active client's session by `ActiveSessionId` or `ActiveSessionName`, discover subscription ids from diagnostics, verify the backup uses the same user display name when configured, and call `TransferSubscriptionsAsync` with `SendInitialValues` defaulting to `true`.

Name-based discovery excludes the backup's own session and rejects multiple
matching active sessions rather than selecting an arbitrary client. Supply
`ActiveSessionId` when names are not unique; that explicit identity bypasses
name-based discovery.

```csharp
var coordinator = new ClientFailoverCoordinator();
ArrayOf<TransferResult> results = await coordinator.TransferActiveSubscriptionsAsync(
    backupSession,
    new ClientRedundancyTransferOptions
    {
        ActiveSessionName = "LineA-PrimaryClient",
        ActiveUserDisplayName = backupSession.Identity.DisplayName,
        SendInitialValues = true
    },
    ct);
```

A transferred subscription needs a client-side owner on the backup session, otherwise the backup's publish engine sees an unknown `SubscriptionId` and deletes it (or never publishes and it expires). Prepare the backup before the takeover: restore the active client's subscriptions with `ISession.Load` (or add `Subscription` objects whose `TransferId` is the active client's subscription id and whose monitored items carry the same client handles). The coordinator then transfers those through the session, which binds them and resumes publishing; ids without a prepared subscription are still moved with the raw service and are left to the caller.

OPC UA does not standardize how active and backup clients exchange `SessionId` or subscription ids. In this stack the client replica set coordinates through the registered client-side shared store (`AddRedundantClientSharedStore` / `AddRaftClientSharedStore` — a CRDT- or [Raft](https://raft.github.io/)-backed `ISharedKeyValueStore` that the `ClientReplicaCoordinator` consumes).

## Network redundancy (as per Part 4 §6.6.4)

Transparent network redundancy is handled below OPC UA by routers, virtual adapters, load balancers, or Kubernetes Services. Non-transparent network redundancy uses multiple endpoints for the same logical server; the client selects another endpoint and recreates only the SecureChannel while reusing the session and subscriptions.

`NetworkRedundancyOptions.AlternateEndpoints` and `NetworkRedundancyEndpointSelector` model non-transparent endpoint alternates. `ManagedSession` accepts these options and cycles to the next endpoint for the same logical server when reconnecting.

```csharp
ManagedSession session = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(primaryEndpoint)
    .WithNetworkRedundancy(new ArrayOf<ConfiguredEndpoint>([secondaryEndpoint, tertiaryEndpoint]))
    .ConnectAsync(ct);
```

## Beyond §6.6: distributed extensions

The base package uses `ISharedKeyValueStore` as the common seam for address-space, session, subscription, retransmission, nonce, and lease records. The in-memory implementation is for tests and single-process samples; multi-pod production deployments need a networked, authenticated, encrypted, and capacity-bounded backend. `IRecordProtector` protects serialized records before they reach the store.

The same seam also backs a shared PKI: `SharedKeyValueCertificateStore` keeps the trusted, issuer and rejected certificate lists and their CRLs in the shared store so every replica makes the same trust decision, with fail-closed record integrity and read-through live propagation. See *Shared certificate stores (distributed trust lists)* below.

```mermaid
flowchart LR
    Client1[Client 1] --> A
    Client2[Client 2] --> B
    A["Replica A"] <-->|CRDT gossip| B["Replica B"]
    B <-->|CRDT gossip| C["Replica C"]
    A <-->|CRDT gossip| C
    A -.->|strong keys| Raft[("Raft<br/>(linearizable)")]
    B -. strong keys .-> Raft
    C -. strong keys .-> Raft
```

Three active/active replicas converge bulk state (address space, sessions) leaderlessly over CRDT gossip, while the exactly-once keyspaces ride the linearizable Raft store. Clients connect to any replica (each is a writer) and fail over between them.

`OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server` goes further than OPC 10000-4 §6.6. It provides active/active multi-writer address-space replication with CRDTs and gossip (`UseReplicatedAddressSpace`) and CRDT-backed session metadata (`UseReplicatedSessions`). CRDT state is eventually consistent and cannot provide compare-and-swap; the single-use nonce registry and other areas that require exactly-once behavior must and are by default mapped to a strongly consistent store (see *Consistency modes*).

Selecting active/active is a single call — `UseActiveActiveRedundancy` wires both the replicated address space and the replicated session store from one set of gossip options (session gossip runs on `GossipPort` + 1 by convention):

```csharp
services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseActiveActiveRedundancy(aa =>
    {
        aa.ReplicaId = Crdt.ReplicaId.New();
        aa.GossipPort = 4840;
        aa.Tls = mutualTlsOptions;      // required unless AllowUnauthenticatedGossip is set
        aa.EnableFastReconnect = true;  // mirrored sessions fast-reconnect after failover
        aa.AddPeer(peerEndpoint);       // static seed; or configure peer discovery (below)
    });
```

The individual methods are available for advanced setups that need to diverge from the shared defaults (for example a different session gossip port or transport):

```csharp
services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseReplicatedAddressSpace(options =>
    {
        options.ReplicaId = Crdt.ReplicaId.New();
        options.UseTcpGossip(IPAddress.Any, port: 4840, tls: mutualTlsOptions);
        options.AddPeer(peerEndpoint);
    })
    .UseReplicatedSessions();
```

Networked CRDT gossip fails closed unless peers are authenticated. Configure TCP gossip with mutual TLS (`GossipTlsOptions` with a server certificate, required client certificates, a client certificate, and remote certificate validation). UDP gossip has no built-in peer authentication and should only be enabled with `AllowUnauthenticatedGossip` inside an isolated development/test network. Address-space values are not secrets, but LWW CRDT frames require integrity/authenticity because a forged high-clock update wins convergence.

> For Kubernetes, add a NetworkPolicy for the gossip port in addition to any Kubernetes API or shared-store policies. Allow ingress and egress only between the replica pods that participate in the CRDT fabric, and keep the gossip port closed to clients, other namespaces, and infrastructure that is not part of the replica set.

### Dynamic peer discovery (beyond §6.6, opt-in)

Static peer lists work for a fixed replica set, but an elastically-scaled deployment (a Kubernetes `Deployment`/`StatefulSet`, an autoscaling group) adds and removes replicas at runtime. The `IPeerDiscovery` seam keeps the peer set current without a restart: a discovery mechanism periodically resolves the peers and updates (a) the client-facing `RedundantServerSet` published through `FindServers` (OPC 10000-4 §6.6.2.4.5.1) and (b), for active/active, the CRDT gossip fabric. **Static configuration (`AddServerRedundancy`) is the fallback** used until discovery finds peers.

| Mechanism | API | Source |
| --- | --- | --- |
| Static (fallback) | `UseStaticPeerDiscovery(peers)` | A fixed configured peer set. |
| DNS (dependency-independent) | `UseDnsPeerDiscovery(options)` | One A/AAAA record per replica — for example a Kubernetes headless service. No cloud SDK required. |
| LDS / LDS-ME | `UseLdsPeerDiscovery(findServers)` | `FindServers` / `FindServersOnNetwork` against a Local Discovery Server. The `FindServers` call is supplied as a delegate so the server package stays free of a client dependency. |
| Kubernetes | `UseKubernetesPeerDiscovery(...)` (`Opc.Ua.Redundancy.Kubernetes`) | EndpointSlices for the server Service, bridged into the generic seam. |
| Custom | `UsePeerDiscovery(factory)` | Any `IPeerDiscovery` implementation. |

```csharp
services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseActiveActiveRedundancy(aa => aa.GossipPort = 4840)
    .AddServerRedundancy(r => r.Mode = RedundancySupport.HotAndMirrored)  // static fallback
    // Dynamic: a Kubernetes headless service resolves to one address per replica.
    .UseDnsPeerDiscovery(dns =>
    {
        dns.HostName = "servers.default.svc.cluster.local";
        dns.GossipPort = 4840;       // CRDT fabric endpoint
        dns.ApplicationPort = 4840;  // OPC UA endpoint published to clients
    });
```

Each discovered peer carries a client-facing identity (`ServerUri` + `DiscoveryUrls`) and, optionally, a CRDT gossip endpoint. The refresh loop publishes the identities to the `RedundantServerSet` and pushes any gossip endpoints into the live active/active transports through an `IGossipPeerSink`. The gossip transport is **add-only**, so a scaled-up replica is added to the fabric immediately, while a decommissioned peer is not actively removed — it ages out as gossip to a dead endpoint simply fails and is retried. LDS and Kubernetes discovery yield OPC UA endpoints (not gossip endpoints), so pair them with DNS discovery of the gossip headless service when you also want the gossip fabric to scale dynamically. Discovery inherits the same secure-gossip defaults: newly-added peers join an already-authenticated transport, so mutual TLS still governs the fabric.



CRDT gossip is eventually consistent — it stays available under network partitions but is not linearizable: it converges without a leader but cannot offer a linearizable compare-and-swap or a change-feed, so exactly-once primitives — the single-use session nonce, lease/leader election — need a strongly consistent (linearizable, quorum-based) backend. `OPCFoundation.NetStandard.Opc.Ua.Redundancy` provides that backend as a Raft layer — Raft is the same consensus algorithm used by systems like etcd and Kafka to replicate state — and `UseRedundancyConsistency` lets you pick the model for the whole `RedundantServerSet`:

- **Eventual (default).** Bulk replicated state (address space, sessions, subscriptions) stays on the leaderless CRDT store; a `HybridSharedKeyValueStore` routes only the strong keyspaces (`nonce/`, `lease/`, `election/` by default, configurable) to a linearizable `RaftSharedKeyValueStore`. Behaviour and performance for the common case are unchanged, and the single-use nonce registry and lease election become linearizable with no extra backend.
- **Strong.** All shared state is served by the linearizable `RaftSharedKeyValueStore`; no CRDT is used. Choose this when every replicated value must be linearizable, accepting that writes require a quorum.

```csharp
services.AddOpcUa()
    .AddServer(server => { })
    // Pick the model BEFORE the distributed features so they compose over it.
    .UseRedundancyConsistency(options =>
    {
        options.Mode = RedundancyConsistencyMode.Eventual; // or Strong
        // options.BulkStoreFactory = sp => /* CRDT bulk store */;    // eventual mode
        // options.RaftConsensusFactory = sp => /* RaftCs replica */; // multi-pod
    })
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseDistributedAddressSpace()
    .UseDistributedSessions(o => o.EnableFastReconnect = true);
```

`UseRedundancyConsistency` registers a shared `IRaftConsensus` plus a native `RaftLeaderElection` (`ILeaderElection`). Raft leadership is decided by the consensus protocol itself — a single leader per term, no split-brain — which is stronger than the lease-CAS `SharedStoreLeaseElection`. Because the distributed features register their store and election with `TryAddSingleton`, call `UseRedundancyConsistency` first and they compose over the chosen store and election.

The consensus engine is pluggable through the `IRaftConsensus` seam. By default the DI registration uses a single-node in-package Raft replica (`DefaultRaftConsensus.CreateSingleNode()`, a real RaftNode with in-memory storage/transport that elects itself leader); `InProcessRaftConsensus` is a lighter deterministic in-process alternative for tests and in-process replica sets. For a multi-pod cluster, build a multi-node replica with `DefaultRaftConsensus.CreateCluster(nodeId, memberIds, transport, …)` — a `RaftCs.Transport.NanoMsg` transport (peers addressed by DNS) plus an in-memory or durable `RaftCs.Storage.File` write-ahead log (WAL) — and register it through `RaftConsensusFactory`. On Kubernetes, `UseKubernetesRaftConsensus` (in `Opc.Ua.Redundancy.Kubernetes`) does this for you from the StatefulSet ordinal and headless-Service DNS. RaftCs ships its own `IRaftTransport` (NanoMsg), so Raft shares the NanoMsg substrate with the CRDT gossip layer. The `samples/Redundancy/RedundantServer` sample (`docker-compose.yml` with the `active-passive.env` env file) is a runnable multi-node example.

On the client side, `AddRaftClientSharedStore` registers a Raft-backed `ISharedKeyValueStore` and a native `RaftLeaderElection` for a client replica set in one call (the `ClientReplicaCoordinator` consumes both), `AddCrdtClientSharedStore` registers the CRDT-gossip (eventual) equivalent, and `AddRedundantClientSharedStore(mode, …)` is the mode-aware entry point that selects between them.

#### Snapshots of the Raft store

`RaftSharedKeyValueStore` is a replicated state machine: without a snapshot, a restarted replica rebuilds its state by applying the whole consensus log. `IRaftSnapshotManager` captures and restores that state so an application can bound the log. Every registration above (`UseRedundancyConsistency`, `AddRaftClientSharedStore`, `AddRedundantClientSharedStore`) registers it as a singleton next to the store; in eventual mode the `HybridSharedKeyValueStore` delegates to its Raft store, so a snapshot holds the strong keyspaces only.

- `CreateSnapshotAsync` commits a barrier and captures the state while the store applies it, so the snapshot holds exactly the commands committed up to and including the barrier. `RaftKeyValueSnapshot.LastIncludedCommand` is that position in the committed command stream (the commands delivered on `IRaftConsensus.Committed`, the same sequence on every replica); `RaftKeyValueSnapshot.Data` is the versioned encoding to persist, and it carries the position.
- `RestoreSnapshotAsync` replaces the state before the store applies the log; it shares the startup gate, so a store that is starting or has started rejects it with `InvalidOperationException`. Pass `RaftSnapshotLogReplay.AfterSnapshot` when the consensus delivers only the commands after the snapshot (its log was compacted up to the snapshot, or it starts empty), or `RaftSnapshotLogReplay.FromFirstCommand` when it replays its persisted log from the beginning; the store then skips the commands the snapshot already holds, so no command — in particular no compare-and-swap — is applied twice.
- A corrupt snapshot is rejected with `FormatException`: the decoder accepts only what the encoder produces (strict UTF-8 keys, canonical value markers, no payload for a null value, no duplicate key).

The replay mode must match the log the consensus actually delivers after the restore, and recovery decides it from what was persisted, never from what the application intended to do. `FromFirstCommand` skips the first `LastIncludedCommand` delivered commands, so it is valid only while the log is complete (starts at the first command); on a log compacted up to the snapshot it would silently skip committed updates made after the snapshot. `AfterSnapshot` applies every delivered command, so it is valid only when the log starts exactly after `LastIncludedCommand`. The example therefore names each snapshot by its position and relies on the log persisting its own compaction boundary (in command positions, updated atomically with the removal of the prefix); `ICompactedLog` stands for that application-owned adapter over the consensus engine's log.

```csharp
// Application-owned adapter over the consensus engine's persisted log, in committed-command positions.
interface ICompactedLog
{
    // The position of the last command removed by compaction, or 0 while the log is complete.
    long CompactedThrough { get; }

    // Removes the commands up to and including lastIncludedCommand and persists the new CompactedThrough.
    Task CompactThroughAsync(long lastIncludedCommand, CancellationToken ct);
}

// Application-owned. Contract: when PublishDurably returns, finalPath durably names the complete contents of
// tempPath (a power loss cannot undo the rename); on failure it throws. Both paths are on the same volume.
static class DurableFile
{
    const int MOVEFILE_REPLACE_EXISTING = 0x1;
    const int MOVEFILE_WRITE_THROUGH = 0x8;
    const int O_RDONLY = 0; // 0 on Linux, macOS and the BSDs
    const int F_FULLFSYNC = 51; // macOS

    public static void PublishDurably(string tempPath, string finalPath)
    {
        if (OperatingSystem.IsWindows())
        {
            // Returns only after the move has been flushed to disk.
            if (!MoveFileEx(tempPath, finalPath, MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH))
            {
                throw new Win32Exception(Marshal.GetLastPInvokeError());
            }
            return;
        }

        // rename(2) is atomic; it is durable only once the containing directory is flushed. The directory is opened
        // first so that its descriptor also reports a writeback error of the rename (Linux reports such errors only
        // to descriptors that were open when they occurred).
        int fd = open(Path.GetDirectoryName(Path.GetFullPath(finalPath))!, O_RDONLY);
        if (fd < 0)
        {
            throw new Win32Exception(Marshal.GetLastPInvokeError());
        }
        try
        {
            File.Move(tempPath, finalPath, overwrite: true);

            // On macOS fsync does not flush the drive's volatile cache; F_FULLFSYNC does (see fsync(2)).
            int flushed = OperatingSystem.IsMacOS() ? fcntl(fd, F_FULLFSYNC) : fsync(fd);
            if (flushed != 0)
            {
                throw new Win32Exception(Marshal.GetLastPInvokeError());
            }
        }
        finally
        {
            close(fd);
        }
    }

    [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
    static extern bool MoveFileEx(string lpExistingFileName, string lpNewFileName, int dwFlags);

    [DllImport("libc", SetLastError = true)]
    static extern int open(string path, int flags);

    [DllImport("libc", SetLastError = true)]
    static extern int fsync(int fd);

    // fcntl is variadic; F_FULLFSYNC takes no argument, so only the fixed parameters are passed.
    [DllImport("libc", SetLastError = true)]
    static extern int fcntl(int fd, int cmd);

    [DllImport("libc")]
    static extern int close(int fd);
}

static string SnapshotFile(string directory, long position) =>
    Path.Combine(directory, $"{position:D20}.snapshot"); // zero-padded: ordinal order is position order

IRaftSnapshotManager snapshots = provider.GetRequiredService<IRaftSnapshotManager>();

// Periodically, or when the log grows large.
RaftKeyValueSnapshot snapshot = await snapshots.CreateSnapshotAsync(ct);
string snapshotPath = SnapshotFile(snapshotDirectory, snapshot.LastIncludedCommand);

// Write the new snapshot to a temporary file; every published snapshot stays intact.
string tempPath = snapshotPath + ".tmp";
await using (var stream = new FileStream(
    tempPath, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize: 4096,
    FileOptions.Asynchronous | FileOptions.WriteThrough))
{
    await stream.WriteAsync(snapshot.Data.Memory, ct);
    await stream.FlushAsync(ct);
    stream.Flush(flushToDisk: true); // fsync (F_FULLFSYNC on macOS) / FlushFileBuffers: the bytes are on disk before the rename
}

// Publish atomically and durably: the file is either absent or complete, also after a power loss.
DurableFile.PublishDurably(tempPath, snapshotPath);

// Only after the publication is durable compact the log, and only then delete the snapshots older than the new boundary.
await log.CompactThroughAsync(snapshot.LastIncludedCommand, ct);
foreach (string older in Directory.EnumerateFiles(snapshotDirectory, "*.snapshot")
    .Where(f => StringComparer.Ordinal.Compare(f, snapshotPath) < 0))
{
    File.Delete(older);
}

// On restart, before the first operation on the store. A leftover *.tmp file is an unpublished,
// possibly partial write: ignore or delete it.
long compactedThrough = log.CompactedThrough;
if (compactedThrough > 0)
{
    // Compacted log: it delivers only the commands after compactedThrough, so only the snapshot at exactly
    // that position fits. If it is missing, the read throws: do not fall back to another snapshot.
    var data = ByteString.From(await File.ReadAllBytesAsync(SnapshotFile(snapshotDirectory, compactedThrough), ct));
    await snapshots.RestoreSnapshotAsync(data, RaftSnapshotLogReplay.AfterSnapshot, ct);
}
else if (Directory.EnumerateFiles(snapshotDirectory, "*.snapshot")
    .OrderBy(f => f, StringComparer.Ordinal).LastOrDefault() is string newest)
{
    // Complete log: never compacted, or the process stopped between publishing the snapshot and compacting.
    // The consensus replays from the first command and the store skips what the snapshot holds.
    var data = ByteString.From(await File.ReadAllBytesAsync(newest, ct));
    await snapshots.RestoreSnapshotAsync(data, RaftSnapshotLogReplay.FromFirstCommand, ct);
}
// Otherwise there is no snapshot and the store applies the whole log.
```

The crash windows resolve from the persisted state alone:

- Stopped before the rename: only a `*.tmp` file is new; the log and the published snapshots are unchanged.
- Stopped after the rename, before compaction: `CompactedThrough` is still the previous boundary. With a complete log (`0`) the newest snapshot is restored with `FromFirstCommand`; with a log compacted up to an earlier snapshot, that earlier snapshot, still present because older snapshots are deleted only after compaction, is restored with `AfterSnapshot`. Using the new snapshot here with either mode would apply commands twice or skip them.
- Stopped after compaction, before the deletion: `CompactedThrough` names the new snapshot; the older files are merely left over.

Never overwrite the published snapshot in place (for example with `File.WriteAllBytesAsync(snapshotPath, ...)`): that truncates it first, so an interrupted write can destroy the only snapshot after the log prefix it replaces was compacted. Keep the snapshot that matches the log's compaction boundary until the log is compacted up to a newer, durably published one:

- Write the temporary file in the same directory as the snapshots. The rename replaces the target atomically (`rename(2)` on POSIX, `MoveFileEx` on Windows) only on the same filesystem/volume; across volumes it degrades to a non-atomic copy and delete.
- `Flush(flushToDisk: true)` makes the file contents durable before the rename, so the rename never publishes a file whose data is still in the page cache.
- An atomic rename is not yet a durable one, and `CompactThroughAsync` must run only after `PublishDurably` returned: if the compaction boundary survived a power loss but the rename did not, `CompactedThrough` would name a missing snapshot and recovery could not proceed. `File.Move` alone guarantees neither: on Windows it does not request write-through, so `PublishDurably` calls `MoveFileEx` with `MOVEFILE_WRITE_THROUGH`; on POSIX the rename is a change to the containing directory, so `PublishDurably` flushes that directory after it — with `fsync` on Linux and the BSDs, and with `fcntl(F_FULLFSYNC)` on macOS, where `fsync` does not flush the drive's volatile cache. .NET offers no managed API for either.
- Network filesystems and some container volume drivers may not honour these guarantees; keep snapshots on a local or block-backed volume whose atomic-rename and fsync semantics are documented.

`LastIncludedCommand` counts committed application commands, not entries of the consensus engine's own log (which also holds leader no-ops and configuration changes), because `IRaftConsensus` does not expose log indexes. Compacting an engine's log therefore needs the engine to map the position to its own index.

For Kubernetes, run the Raft members as a `StatefulSet` with stable network identities (so each Raft peer keeps its node id and address across restarts) and an odd member count (3 or 5) for a fault-tolerant quorum; a durable `RaftCs.Storage.File` WAL on a `PersistentVolume` lets a restarted pod rejoin without a full snapshot. See the [Kubernetes High Availability Deployment](Kubernetes.md) guide.

### Sharing values across replicas: distributed value cache (beyond §6.6, opt-in)

OPC 10000-4 §6.6.2.2 requires identical NodeIds and AddressSpaces across a `RedundantServerSet` but does not standardize how live process *values* are cached or shared between replicas. `Opc.Ua.Redundancy.Server` adds an opt-in provider for this: `IDistributedValueCache` (default implementation `DistributedValueCache`) stores a node's last value in the distributed `INodeStateStore` and serves it back within a freshness bound. `UseDistributedAddressSpace` registers `IDistributedValueCache` in dependency injection (routed per node through the server's `INodeStateStoreRegistry`, populated when the server starts); a consumer that constructs the node-state store directly can build a `DistributedValueCache` over it instead.

`DistributedValueParticipation.EnableDistributedValueParticipation(variable, cache, maxAge, liveRead)` wires a variable's asynchronous read/write callbacks to the cache: writes cache write-through, and reads serve the last cached value while it is fresh (falling back to `liveRead` and caching the result). This lets the active replica publish a value once and every replica serve it, and it lets a promoted replica seed from the last value the former leader shared so a simulated/derived value continues across a failover rather than restarting. Monitored items are unaffected — they read through the normal pipeline — so a monitored item observes the shared value only where the read path participates, consistent with OPC UA monitored-item semantics and the requirement that monitored items "read as today".

```csharp
// Injected from DI (registered by UseDistributedAddressSpace).
IDistributedValueCache cache = /* ... */;

variable.EnableDistributedValueParticipation(
    cache,
    maxAge: TimeSpan.FromSeconds(10),
    liveRead: ct => new ValueTask<DataValue>(ReadLiveValue()));

// The active replica writes each new value through to the shared store;
// a promoted replica reads the last shared value on becoming leader.
await cache.CacheAsync(variable.NodeId, new DataValue(value, StatusCodes.Good, DateTimeUtc.Now));
(bool fresh, DataValue last) = await cache.TryGetAsync(variable.NodeId, maxAge);
```

Because the value cache writes through the shared `INodeStateStore`, its confidentiality/integrity is governed by the same `IRecordProtector` as the rest of the mirrored state (fail-closed for external stores without a protector). Cross-process value continuity therefore requires a store that propagates writes between replicas: it is verified on the strong-consistency (Raft) topology; on active/active eventual it depends on the CRDT address-space gossip transport. The `samples/Redundancy/RedundantServer` sample wires its `HighAvailability.Counter` through this provider — see the sample README's *Sharing values across replicas* section.

### Shared certificate stores (distributed trust lists) (beyond §6.6, opt-in)

OPC UA replicas in a `RedundantServerSet` must present the same PKI: a certificate trusted (or rejected) on one replica should be trusted (or rejected) on every replica, and CRLs must be shared. By default each replica keeps its trusted, issuer and rejected lists in a local `DirectoryCertificateStore` (or Windows `X509Store`), so trust decisions can diverge across replicas.

`SharedKeyValueCertificateStore` stores the **trusted**, **issuer** and **rejected** certificate lists and their **CRLs** in the same `ISharedKeyValueStore` that backs the rest of the high-availability state, and plugs in through the existing certificate store provider model so any `CertificateStoreIdentifier` can point at it. It holds **public certificates only** (`NoPrivateKeys` is `true`, `SupportsLoadPrivateKey` is `false`); distributing the application instance certificate with its private key is a future capability.

**Integrity, fail-closed.** Trust-list integrity is critical: an attacker with write access to the shared store must not be able to inject a trusted CA. Every record (certificate or CRL) is written through an `IRecordProtector` and verified on read; a forged or tampered record fails the authenticity check and is skipped (fail-closed) — it is never returned to a validator. An external, network-reachable store **must** use an authenticating protector (for example `AesCbcHmacRecordProtector`, keyed from a Kubernetes Secret / KMS, the same record-protection model used for mirrored sessions and subscriptions); an in-memory, single-process store may use the no-op `NullRecordProtector` (the default).

**Live propagation.** The certificate validator enumerates the trusted/issuer store on each validation (it caches the store instance, not the certificate list), and `SharedKeyValueCertificateStore.EnumerateAsync` reads the current shared-store state on every call, so a certificate trusted or rejected on one replica is observed by the others' next validation automatically — no restart and no explicit refresh. On a network-backed store (Raft/Redis), reading on every validation costs a round-trip; a future enhancement can cache the enumerated list locally and use `ISharedKeyValueStore.WatchAsync` to invalidate it (and emit `CertificateManager` `TrustListUpdated` / `CrlUpdated` change events) when another replica changes the shared state, keeping the read path fast while remaining live.

```csharp
using Opc.Ua;
using Opc.Ua.Security.Certificates;

// The shared store and protector come from the HA infrastructure
// (or an InMemorySharedKeyValueStore + NullRecordProtector for a single process).
var provider = new SharedKeyValueCertificateStoreProvider(sharedStore, protector);

CertificateManager manager = CertificateManagerFactory.Create(
    securityConfiguration,
    telemetry,
    options => options.AddStoreProvider(provider));
```

The provider is injectable — construct it from the DI-registered `ISharedKeyValueStore` and `IRecordProtector` and pass it through `CertificateManagerOptions.AddStoreProvider`; the direct construction above is the fallback. Point the security configuration's trusted/issuer/rejected stores at the distributed backend by setting each `CertificateStoreIdentifier` to store type `CertificateStoreType.SharedKeyValue` with a `kv:` store path (for example `kv:pki/trusted`, `kv:pki/issuer`, `kv:pki/rejected`); the built-in `Directory` and `X509Store` providers remain available. Every replica in the set uses the same shared store and the same paths, so they share one trusted list, one issuer list and one rejected store. The rejected-list trim (`AddRejectedAsync`) keeps the newest N entries best-effort (advisory, not security-critical). See [Certificate Manager](CertificateManager.md) for the certificate lifecycle model.

### Distributed PushManagement transactions (beyond §6.6, opt-in)

OPC UA Part 12 §§7.10.2-7.10.11 define a **single, server-wide** PushManagement configuration transaction: one Session owns it at a time, its `UpdateCertificate` / `CreateSelfSignedCertificate` / `DeleteCertificate` / TrustList changes are staged, and nothing takes effect until `ApplyChanges`. On a stand-alone server the default `PushConfigurationTransactionCoordinator` enforces this per process. In a `RedundantServerSet` that is not enough on its own: each replica has its own coordinator, so two administrators connected to two replicas could each open a transaction and stage conflicting certificate rotations at the same time.

`UseDistributedPushConfiguration()` (from `Opc.Ua.Redundancy.Server`) opts a server into a transaction that is single **across the whole replica set**, reusing the same high-availability building blocks as the rest of this document (the shared `ISharedKeyValueStore`, `IRecordProtector` and, optionally, the elected `ILeaderElection`). It is opt-in: a server that does not call it behaves exactly as before.

**Cluster-wide single ownership.** `DistributedPushConfigurationTransactionCoordinator` wraps the per-server coordinator and guards it with a compare-and-swap **lease** in the shared store — the same lease pattern used by `SharedStoreLeaseElection` for leader election. A replica must hold the lease to start a transaction, so at most one replica has an active transaction at any time. The lease is acquired (or renewed) at an `await` boundary immediately **before** the synchronous `Stage`, so the coordinator's synchronous `Stage`/`CancelChanges` contract is preserved with no sync-over-async; a background loop renews the lease while the transaction is active and releases it as soon as the transaction is applied, cancelled, reset, or its owning Session closes. A replica that crashes stops renewing, and once the lease expires a standby can start a new transaction. Another replica that tries to stage while the lease is held is refused with `Bad_TransactionPending`, exactly as a second Session on one server would be.

**Pending regenerated keys across replicas.** `CreateSigningRequest(regeneratePrivateKey: true)` (§7.10.10) produces a private key that must survive until a matching `UpdateCertificate` consumes it — possibly on a **different** replica after a failover. `SharedKeyValuePendingCertificateKeyStore` persists that key in the shared store (scoped per CertificateGroup/CertificateType) instead of a local directory. It consumes the key atomically (only one replica wins a concurrent `TryTake`), wipes every private-key working buffer, and — like every other mirrored record — protects each entry with an `IRecordProtector` (authenticated encryption) so the shared store can be treated as untrusted. An external, network-reachable store **must** provide a protector; without one the pending-key store **fails closed** rather than persist a private key unprotected, matching the fail-closed default of the distributed session, subscription and trust-list stores.

**Composition with leader election.** Distributed PushManagement composes transparently with whichever leader election is configured. By default the shared-store lease alone enforces single ownership, so any replica a client reaches may own the transaction. Setting `RequireLeadership = true` funnels every transaction through the elected leader; when the Kubernetes extension is configured the leader is decided by the **Kubernetes-native `Lease`** already registered by `UseKubernetesLeaderElection` — no second Kubernetes client is created. Raft and shared-store leader election compose the same way.

```csharp
using Opc.Ua.Redundancy.Server;

builder
    // ... existing HA setup (shared store, record protector, leader election) ...
    .UseKubernetesLeaderElection()          // ILeaderElection = Kubernetes-native Lease
    .UseDistributedPushConfiguration(options =>
    {
        options.ReplicaId = Environment.GetEnvironmentVariable("POD_NAME")!;
        options.LeaseDuration = TimeSpan.FromSeconds(30);   // lease validity without renewal
        options.RenewInterval = TimeSpan.FromSeconds(10);   // background renew cadence
        options.KeyPrefix = "pushconfig/";                  // shared-store keyspace
        options.RequireLeadership = true;                   // only the K8s-elected leader owns the transaction
        // For an external store, configure authenticated encryption (or register an IRecordProtector):
        options.RecordProtectorFactory = sp => new AesCbcHmacRecordProtector(GetPushConfigKey());
    });
```

Give every replica a distinct `ReplicaId`, keep `RenewInterval` well below `LeaseDuration`, and share the same `KeyPrefix` and record-protection key across the set. See [Certificate Manager — PushManagement Transactions](CertificateManager.md#pushmanagement-transactions-opc-ua-part-12-7102-71011) for the underlying transaction model.

### GetEndpoints load direction (beyond §6.6, opt-in)

`UseServerLoadDirection(...)` lets a `GetEndpoints` request on a **dedicated balancing discovery URL** be answered with the endpoints of the best member of the `RedundantServerSet`, so a Client that connects there is **directed** to the active server (active/passive) or spread across equally-healthy servers by load (active/active). This is a non-standard, server-side load-direction *hint*: it **complements**, and never replaces, the standard client-driven `RedundantServerArray` + `ServiceLevel` selection of OPC 10000-4 §6.6.2.4, which remains the authoritative Failover mechanism.

```mermaid
sequenceDiagram
    participant Client
    participant LB as Balancing discovery URL
    participant Set as RedundantServerSet
    participant Best as Best server
    Client->>LB: 1. GetEndpoints (balancing URL)
    LB->>Set: 2. Rank members by ServiceLevel tier, then load
    Set-->>LB: 3. Endpoints of the best member
    LB-->>Client: 4. Return the best server's endpoints
    Client->>Best: 5. CreateSession/ActivateSession to the target's own endpoint + certificate
```

How it works. Each member publishes three integrity-protected records to the shared store: its health `ServiceLevel` (`svc/<uri>`), a separate **load weight** (`load/<uri>`, 0 idle … 255 fully loaded, from an injected `ILoadWeightProvider`), and its own `EndpointDescription`s (`endpoint/<uri>`, observed from normal discovery). A request that arrives on the balancing URL is resolved by the direction policy: rank members by health-`ServiceLevel` tier (Healthy › Degraded › NoData › Maintenance); if a peer is in a strictly higher tier it wins (active/passive → the active server), otherwise the least-loaded member of the top tier is chosen using the load weight, with random tie-breaking among equally-loaded members (active/active → load spreading). The local server serves its own endpoints when it is the best choice, when the peer view is stale, or when the target's endpoints are unavailable (fail-safe). `ServiceLevel` keeps its OPC UA meaning (health and Failover); load is a *separate* signal, so load never triggers a spurious Failover and a stale load weight only affects tie-breaking.

```csharp
services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseRedundancyConsistency(RedundancyConsistencyMode.Eventual)   // or Strong
    .AddServerRedundancy(r => r.Mode = RedundancySupport.Hot)
    .AddServerServiceLevel(new LeaderServiceLevelProvider(/* … */)) // real health signal
    .UseServerLoadDirection(options =>
    {
        options.BalancingEndpointUrl = "opc.tcp://ha.example.com:4840"; // the virtual/LB discovery URL
        // options.StalenessWindow = TimeSpan.FromSeconds(15);
        // options.LoadBandSize = 16;             // quantize load to damp oscillation
        // options.HealthSubBandSize = 0;         // 0 = all Healthy is one eligibility tier
    });
// Register an ILoadWeightProvider for load-aware balancing (default is a constant 0 = random spread).
```

Consistency. The direction inputs follow `UseRedundancyConsistency`: in **Eventual** (default) the `svc/`, `load/`, and `endpoint/` records ride the leaderless CRDT store; in **Strong** they are linearizable via Raft. The high-churn `load/` weight is always coalesced (`LoadPublishInterval`) so it never issues a write per load tick — a per-write quorum on a rapidly changing value is the overhead to avoid, and a slightly stale load tie-break still picks a healthy peer.

Conformance and client behavior. `GetEndpoints` (§5.4.2) is defined to describe *the server that received the request*, so directing to a peer is a deliberate extension; it is kept **gated** — plain discovery on any other URL returns the local server unchanged, so generic clients and tooling that inspect a specific server are unaffected. A returned `EndpointDescription` carries the target's own `ApplicationUri` and certificate, so the CreateSession server-endpoint tamper check (§5.6.2) still holds once the Client connects to the target. The stack's own Client needs no change to follow the redirect (it uses the returned endpoints to connect); point it at the balancing URL and pair it with `WithServerRedundancy()` so the standard client-driven `ServiceLevel` selection remains the authoritative complement — it already fails a Client that lands on a non-Healthy server over to the active one and, on a connect failure, retries the normal discovery URL / `FindServers`/`RedundantServerArray` path. This is **connect-time placement, not continuous balancing** — ongoing Failover still relies on the Client monitoring `ServiceLevel` per §6.6.

Security. The store is treated as an untrusted conduit: every direction and endpoint record is written through the configured `IRecordProtector` and verified before use, so a forged or tampered record is dropped fail-closed. Configure an authenticated-encryption protector (`AesCbcHmacRecordProtector`) for any networked backend; the default `NullRecordProtector` is for single-process/in-memory use only. Because the redirect returns a peer's certificate, Clients must trust every member's certificate (non-transparent redundancy uses per-server certificates). The balancing endpoint is typically unauthenticated discovery, so a rogue replica could bias or deny direction, but it cannot force a Client onto an untrusted server (the Client validates the returned certificate); this is the same rogue-replica trust boundary as the rest of the shared store. Keep the balancing decision damped (load banding + random tie-break + the load coalescing interval) to avoid herding every Client onto a momentary best.

### Client-side high availability (replica sets)

OPC 10000-4 §6.6 covers redundant *servers* and the client behavior for connecting to them. A complementary, beyond-spec capability is a redundant *client* replica set: two or more client processes that cooperate so that exactly one — the leader — holds the active session and subscriptions while the others stand by (hot, warm, or cold) and take over on leader loss. This is provided by `OPCFoundation.NetStandard.Opc.Ua.Client.Redundancy`.

```mermaid
flowchart LR
    L["Client 1 (leader)"] <-->|leader election, session secret| KV[("ISharedKeyValueStore<br/>+ IRecordProtector")]
    S["Client 2 (standby)"] <-->|watch, promote| KV
    L -->|active session| Server["OPC UA Server"]
    S -.->|takes over on leader loss| Server
```

The two client replicas elect a leader and share the leader's session secret through the protected store; on leader loss the standby is promoted and reuses the token to re-activate (or recreates the session and transfers subscriptions).

The simplest way to consume this is `AddRedundantClientSession(...)` in `OPCFoundation.NetStandard.Opc.Ua.Redundancy.Client`: register a client shared store and election, then a redundant session, and resolve a single, stable `ISession`. Failover, leader election, and the underlying session churn are hidden — the application reasons only about `ISession`, and service calls block until this replica is the leader with a live session (synchronous members throw `BadInvalidState` until then).

```csharp
services
    .AddRaftClientSharedStore()                     // ISharedKeyValueStore + ILeaderElection
    .AddRedundantClientSession(options =>
    {
        options.NodeId = replicaId;
        options.Mode = ClientStandbyMode.Hot;
        options.CreateSessionAsync = ct => new ManagedSessionBuilder(configuration, telemetry)
            .UseEndpoint(endpoint)
            .WithServerRedundancy()
            .WithTokenReuseFailover()
            .ConnectAsync(ct);
    });

// A hosted service starts the coordinator; resolve the transparent handle anywhere and use it:
ISession session = provider.GetRequiredService<ISession>();
BrowseResponse response = await session.BrowseAsync(/* … */, ct);   // blocks until this replica leads
```

Under the hood it reuses the same seams as the server, now shared from `Opc.Ua.Core` under the `Opc.Ua.Redundancy` namespace:

- **Leader election** among the client replicas (`ILeaderElection`, e.g. `SharedStoreLeaseElection`), so exactly one replica is the active client at a time.
- **Shared session secret.** The leader persists its session secrets (`AuthenticationToken`, nonces) through `ISharedKeyValueStore`, encrypted and integrity-protected at rest by `IRecordProtector`. The coordinator is fail-closed: a non-in-memory store with no real protector is rejected. A follower promoted to leader reuses the token to `ActivateSession` against a `HotAndMirrored` server, or recreates the session and transfers subscriptions otherwise.
- **Standby modes** (`ClientStandbyMode`): Cold (elect only; connect+activate on promotion), Warm (keep a connected session, create subscriptions on promotion), Hot (keep a connected session with subscriptions sampling-only, enable publishing on promotion).

For non-DI callers, `RedundantClientSessionBuilder` builds the same `RedundantClientSession` (an `ISession`) directly. The lower-level `ClientReplicaSetBuilder` / `ClientReplicaCoordinator` seam is still available when you want to hold and drive the leader `ManagedSession` yourself:

```csharp
var coordinator = new ClientReplicaSetBuilder(telemetry)
    .WithNodeId(replicaId)
    .WithStandbyMode(ClientStandbyMode.Hot)
    .UseRedundancy(election, sharedStore, recordProtector)
    .UseSession(ct => new ManagedSessionBuilder(configuration, telemetry)
        .UseEndpoint(endpoint)
        .WithServerRedundancy()
        .WithTokenReuseFailover()
        .ConnectAsync(ct))
    .ConfigureLeader((session, fastActivated, ct) => ApplySubscriptionsAsync(session, ct))
    .Build();
await coordinator.StartAsync(ct);
```

The full token-reuse fast-activate from shared secrets and subscription recreate/transfer are exercised by the client replica-set sample and integration tests.

## Kubernetes deployment

Use the consolidated [Kubernetes High Availability Deployment](Kubernetes.md) guide for the `Opc.Ua.Redundancy.Kubernetes` package. It covers Kubernetes Lease election, EndpointSlice peer discovery, ServiceLevel-driven readiness, StatefulSet/Deployment and Service manifests, RBAC, probes, time synchronization, secrets, and GDS/NTRS registration.

## Samples

`samples/Redundancy/RedundantServer` demonstrates the server-side distributed and redundancy registrations, with `docker-compose` files for active/active and active/passive replica sets. Its strong active/passive topology historizes a stable Counter and event notifier through `UseDistributedHistorian`. `samples/Redundancy/RedundantClient` shows the recommended managed-client pattern: a single `ManagedSession` with `WithServerRedundancy()` and token-reuse failover that connects to any server, reads its redundancy metadata, and fails over transparently. Its `--history` workflow holds raw, event, and processed cursors across active-process loss and verifies post-promotion writes.

`samples/Redundancy/RedundantClient/docker-compose.yml` runs the client and server in one env-driven compose file that covers the full HA matrix. `COMPOSE_PROFILES` (defaulted by an adjacent `.env`) picks one server profile (`server-eventual`, scalable active/active, or `server-strong`, a fixed 3-node Raft quorum) and one client profile (`client-independent`, scalable, or `client-coordinated`, a fixed 3-node client replica set); the eventual server and independent clients scale with `--scale server=N --scale client=M`. **Both sides log their failover / HA behavior, including data loss**: the client turns transparent reconnects and leader handoffs into explicit `FAILOVER:`, `ACTIVE CLIENT:`, `DATA LOSS:`, and `HA OK: Counter continued …` lines, and the server logs its active/standby role transitions and a per-replica `Counter` heartbeat. `server-eventual` shows data loss on failover because each replica owns an independent `Counter`; `server-strong` (active/passive Raft) keeps the `Counter` on a linearizable store and shows **no** data loss. See the [RedundantClient README](../samples/Redundancy/RedundantClient/README.md#run-with-docker-compose-one-file-the-full-ha-matrix-env-driven).

## Security considerations

Distributed high-availability deployments protect the shared store as part of the OPC UA trust boundary. Use an authenticated and encrypted channel to the store, protect serialized records with `IRecordProtector`, provision record-protection keys through a key ring or secret manager, apply TTLs and quotas to session/nonce/subscription keys, and fail closed if a required strongly consistent store is unavailable.

### Record context and plaintext ownership

The 2.0 `IRecordProtector` contract requires a `ByteString context` on both `Protect`
and `TryUnprotect`. For stored records, use
`RecordProtectionContext.Create(recordType, fullStoreKey)`: strict UTF-8 encoding
of `recordType + "|" + fullStoreKey`. Type labels cannot contain `|`; keys retain
their exact case, escaping and configured prefixes. Readers use the actual key
returned by scans and change feeds, not just a type label or a reconstructed
identifier. This also binds snapshot chunks to their generation and ordinal,
and historical segments to their individual keys, independently of the storage backend.

```csharp
ByteString binding = RecordProtectionContext.Create("application-record", key);
ByteString envelope = protector.Protect(binding, plaintext);
await store.SetAsync(key, envelope, cancellationToken).ConfigureAwait(false);

(bool found, ByteString stored) = await store.TryGetAsync(key, cancellationToken).ConfigureAwait(false);
if (!found || !protector.TryUnprotect(binding, stored, out ByteString recovered))
{
    throw new ServiceResultException(StatusCodes.BadSecurityChecksFailed);
}
```

Protect separately when the same plaintext belongs under two different keys;
do not copy a ciphertext between a current-record pointer and a generation
record. A wrong key, type, context or modified envelope fails closed. Never retry
with an empty context after a failed bound read. When binding is genuinely not
needed, pass `default(ByteString)` or `ByteString.Empty` explicitly; both encode
the same empty context. The string convenience extensions only convert to UTF-8
and invoke the required byte-context member once. They do not select a capability
or provide a fallback.

`AesCbcHmacRecordProtector` retains the envelope
`[version:1][keyId:4 LE][IV:16][ciphertext][HMAC-SHA256:32]`. Its MAC authenticates
`[contextLength:4 LE][context][header][ciphertext]`, including a zero context
length for an explicitly unbound record, and is checked before decryption.
`KeyRingRecordProtector` supplies the same context to active and retired keys.

Private-key consumers require `IOwnedRecordProtector.TryUnprotectOwned(context,
envelope, out byte[] plaintext)`. This transfers the original decrypted array,
independent of the store input, so the caller can wipe it with
`CryptoUtils.ZeroMemory` in a `finally` block. Do not implement this by calling
`TryUnprotect(...).ToArray()`: that leaves an unwiped decrypted copy.
`SharedKeyValuePendingCertificateKeyStore` rejects protectors without this
ownership contract. Owned key-ring reads skip members that cannot supply owned
buffers, without invoking their immutable-plaintext read path. If no capable
member authenticates a record, the ring returns `false` and an empty buffer;
pending-key stores treat that record as absent rather than throwing.
Custom KMS/HSM providers must implement the owned contract to read pending keys
protected by those members, including records retained during key rotation.
`NullRecordProtector` remains a non-authenticating, process-local test/demo
option, not a substitute for protection of a networked shared store.

### Shared application identity

Transparent redundancy uses one logical application identity. Every replica that serves the transparent virtual endpoint must use the same endpoint URL, ApplicationUri, application instance certificate, and private key so clients can validate one server identity and complete `ActivateSession` against any replica. The compromise blast radius of that shared private key is the entire transparent set: an attacker can impersonate the virtual server, terminate or redirect client trust, and participate in failover until the certificate is revoked and trust lists are updated. Prefer non-transparent redundancy with per-replica certificates when that shared-key risk is unacceptable.

The `samples/Redundancy/RedundantServer` sample includes a worked transparent deployment (`Transparent/docker-compose.yml`): two `REDUNDANCY_MODE=transparent` replicas that share one `ApplicationUri` and one certificate and mirror session + address-space state by CRDT gossip, behind an `nginx` TCP load balancer that publishes a single virtual endpoint. Each replica advertises the load-balancer host, so discovery and `CreateSession` echo the one endpoint the client uses, and a mirrored session resumes on the survivor when a replica fails. The sample seeds the shared certificate into a volume for convenience; a secured deployment provisions it from a secret to every replica (see `docs/Kubernetes.md`).

Rotate the transparent shared application certificate/key as a coordinated replica-set operation. Issue the replacement certificate with the same virtual identity and endpoint subject alternative names, distribute the new private key through the certificate store or a secret manager without baking it into images, stage client and GDS trust-list updates so both old and new certificates are trusted during the overlap, restart or hot-reload replicas in a controlled window, verify every replica presents the new certificate at the virtual endpoint, then revoke/remove the old certificate and securely delete the old private key from all nodes and secret versions. Treat an emergency rotation after suspected compromise as a fail-closed event: drain or stop replicas that still hold the old key before restoring client access.

On the client side, a client replica set shares the same trust boundary. The leader persists its session secret (`AuthenticationToken`, nonces) through `ISharedKeyValueStore`, so it must be encrypted and integrity-protected at rest by `IRecordProtector`; the `ClientReplicaCoordinator` is fail-closed and rejects a non-in-memory store that has no real protector. Because non-transparent redundancy and GetEndpoints load direction return per-server certificates, a failing-over or directed client must trust every member's certificate, and a promoted standby re-validates the server endpoint and certificate on reconnect. Treat the client shared store, like the server's, as an untrusted conduit: authenticate and encrypt the channel to it and scope its keys with TTLs and quotas.