UA-.NETStandard/docs/HighAvailability.md

High Availability and OPC UA Redundancy

This guide explains how the OPC UA .NET Standard high-availability APIs implement OPC 10000-4 §6.6 Redundancy. It covers server, client, subscription, and session redundancy, plus Kubernetes and active/active extensions. Worked examples are in samples/Redundancy/RedundantServer and samples/Redundancy/RedundantClient.

Redundancy and high availability are opt-in. Add the relevant OPCFoundation.NetStandard.Opc.Ua.Redundancy.* NuGet package to your application, such as OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server or .Client. The standard OPCFoundation.NetStandard.Opc.Ua.Client and OPCFoundation.NetStandard.Opc.Ua.Server libraries alone do not provide OPC UA redundancy.

For distributed PubSub active/standby publishers and subscribers, see the PubSub counterpart: PubSub High Availability.

Contents

• Redundancy overview (as per Part 4 §6.6.1)
• Server redundancy (as per Part 4 §6.6.2)
  • Server.ServerRedundancy model
  • Add* and Use* API convention
• ServiceLevel and load balancing (as per Part 4 §6.6.2.4.2 and §6.6.2.4.3)
• Non-transparent failover modes and client actions (as per Part 4 §6.6.2.4.5)
• Manual failover and Maintenance (as per Part 4 §6.6.5)
• HotAndMirrored and Transparent state mirroring
  • Active/passive address-space consistency
  • Strong active/passive historian
• Client redundancy (as per Part 4 §6.6.3)
• Network redundancy (as per Part 4 §6.6.4)
• Beyond §6.6: distributed extensions
  • Dynamic peer discovery (beyond §6.6, opt-in)
  • Sharing values across replicas: distributed value cache (beyond §6.6, opt-in)
  • Shared certificate stores (distributed trust lists) (beyond §6.6, opt-in)
  • Distributed PushManagement transactions (beyond §6.6, opt-in)
  • GetEndpoints load direction (beyond §6.6, opt-in)
  • Client-side high availability (replica sets)
• Kubernetes deployment
• Samples
• Security considerations
  • Record context and plaintext ownership
  • Shared application identity

Redundancy overview (as per Part 4 §6.6.1)

OPC UA defines three independent but composable redundancy dimensions: server redundancy gives clients multiple servers that expose the same data, client redundancy lets backup clients take over work from an active client, and network redundancy gives a client and server more than one communication path. The stack implements standardized discovery and failover metadata in the core server/client packages, and adds opt-in distributed state through OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server.

Server redundancy is either transparent or non-transparent. In transparent redundancy, the redundant set looks like one server and failover is hidden from the client. In non-transparent redundancy, each server has its own identity and endpoint, and the client reads Server.ServerRedundancy plus Server.ServiceLevel to decide what to do.

Server redundancy (as per Part 4 §6.6.2)

Opc.Ua.RedundancySupport is the stack enum published by Server.ServerRedundancy.RedundancySupport and consumed by the client. The modes are:

Mode	OPC UA meaning	Stack behavior
None	No server redundancy is advertised.	Single-server default.
Cold	Only one server is active at a time; backups may be unavailable or not running.	Client connects to one server and creates a fresh session/subscriptions on failover.
Warm	Backup servers are running but have less functionality or no process data.	Client can connect to peers, keep subscriptions created but sampling/publishing inactive on backups, and promote the highest-ServiceLevel server.
Hot	Multiple servers are running and can independently provide data.	Client connects to peers and either hands reporting over to one active server or merges reporting streams.
HotAndMirrored	Hot servers mirror communication state.	ManagedSession keeps one active session, may keep lightweight ServiceLevel status-check sessions to backups, and fails over by opening a channel and re-activating the mirrored session with the existing AuthenticationToken; subscriptions are not recreated client-side.
Transparent	One virtual server identity hides physical failover.	Server publishes CurrentServerId; deployment infrastructure supplies the virtual address and shared certificate/endpoint identity.

Server.ServerRedundancy model

Register AddServerRedundancy(...) on the server builder to populate the live Server.ServerRedundancy nodes after the server starts. ServerRedundancyOptions.Mode writes RedundancySupport. PeerServerUris and RedundantPeers populate RedundantServerArray; for non-transparent modes the startup task also populates ServerUriArray, and for transparent mode it publishes CurrentServerId.

Server.ServerRedundancy is exposed as the generated subtype for the configured mode — TransparentRedundancyType for Transparent, NonTransparentRedundancyType for Cold/Warm/Hot/HotAndMirrored, or the base ServerRedundancyType for None — so a client that browses the node sees the correct HasTypeDefinition and only the members the standard defines for that subtype (CurrentServerId for transparent, ServerUriArray for non-transparent).

The mode can be changed at runtime through the injectable IServerRedundancyController.ChangeModeAsync(mode): the node is re-materialized as the new subtype in place and a ModelChange is raised, so connected clients observe the new type definition and members without a restart.

For non-transparent redundancy, set RedundantPeers when clients should resolve peers through FindServers; ConfiguredRedundantServerSetProvider exposes those ApplicationDescription entries server-side. AdvertiseNtrsCapability defaults to true, so non-transparent modes add the NTRS discovery capability for GDS/NTRS registration.

All servers in a RedundantServerSet must have identical application AddressSpaces: identical NodeIds, browse paths, AddressSpace structure, and ServiceLevel algorithm. Only local server diagnostics may differ. UseDistributedAddressSpace(...) helps satisfy this by mirroring node topology and values through INodeStateStore/ISharedKeyValueStore, but application-specific method handlers and callbacks still need to be attached by each node manager.

Configure UseReplicaNodeIdentity for replicated address spaces in either mode. It reserves identical shared namespace indexes before node-manager construction and binds the standard factory's assignment policy across replicas. The guarantee concerns raw wire NodeIds, not client-side URI remapping. Incompatible layouts, modes and peer/store contracts fail explicitly; built-in diagnostics and configuration remain local.

Add* and Use* API convention

High-availability builder methods follow the stack's standard Add*/Use* convention: Add* methods wire OPC 10000-4 §6.6 nodes and methods that are part of the standardized server model, while Use* methods register beyond-spec extension building blocks that make a redundant deployment work. For the stack-wide DI conventions and per-package entry points, see DependencyInjection.md.

Standardized redundancy nodes and methods (Add*):

API	Description
AddServerRedundancy(options)	Publishes the Server.ServerRedundancy metadata (RedundancySupport, ServerUriArray / RedundantServerArray, NTRS capability). Metadata only — it does not drive ServiceLevel.
AddServerServiceLevel(provider)	Drives the Server.ServiceLevel node from an IServiceLevelProvider (for example LeaderServiceLevelProvider; use a ConstantServiceLevelProvider for a fixed level).
AddRequestServerStateChange(options)	Enables the standard Server.RequestServerStateChange method for administrator-driven Maintenance / NoData failover.

Beyond-spec distributed building blocks (Use*):

API	Description
UseReplicaNodeIdentity(replicaSetId, namespaceUris, mode, writerAssignedIds)	Required shared identity contract for replica sets: fixed namespace slots, guarded standard factory and unchanged hydrated IDs.
UseDistributedAddressSpace(options)	Active/passive shared-store address-space replication with leader election (one writer; standbys hydrate from the shared store). Also registers the injectable IDistributedValueCache and the SharedKeyValue certificate-store support.
UseDistributedSessions(options)	Mirrors session metadata to the shared store for fast reconnect after failover (opt-in EnableFastReconnect).
UseDistributedSubscriptionMirroring()	Mirrors subscription / monitored-item definitions, retransmission state and continuation-point envelopes to the shared store.
UseDistributedHistorian(options)	Adds the protected, leader-write active/passive historian and durable portable HistoryRead continuation store. Requires a cross-process linearizable store, authenticated IRecordProtector, and leader election; active/active and eventual configurations fail closed.
UseReplicatedAddressSpace(options)	Active/active multi-writer address-space replication over CRDT gossip (no leader).
UseReplicatedSessions(options)	Active/active CRDT-gossiped session metadata for cross-replica fast reconnect.
UseActiveActiveRedundancy(options)	Single-call active/active: wires the replicated address space and sessions from one set of gossip options (session gossip on GossipPort + 1). This is the entry point that "selects active/active"; the two UseReplicated* methods remain for advanced setups.
UseRedundancyConsistency(options)	Selects the shared-store consistency model for the whole set — eventual (CRDT + Raft strong keyspaces) or strong (Raft). Call it before the other Use* methods so they compose over the chosen store.
UseServerLoadDirection(options)	Opt-in GetEndpoints load direction: answers discovery on a balancing URL with the best replica's endpoints.
UsePeerDiscovery(factory), UseStaticPeerDiscovery(peers), UseDnsPeerDiscovery(options), UseLdsPeerDiscovery(findServers)	Dynamic peer discovery for the client-facing RedundantServerSet (FindServers) and, for active/active, the CRDT gossip fabric. Static configuration (AddServerRedundancy) is the fallback used until discovery finds peers.
UseKubernetesRaftConsensus(...)	(Opc.Ua.Redundancy.Kubernetes) Builds the Raft cluster from the StatefulSet ordinal and headless-Service DNS.
UseKubernetesPeerDiscovery(...)	(Opc.Ua.Redundancy.Kubernetes) Discovers peer ServerUris from EndpointSlices, bridged into the generic peer-discovery seam.

AddServerRedundancy(...) only publishes the redundancy metadata. It does not calculate or drive Server.ServiceLevel; register a ServiceLevel provider with AddServerServiceLevel(...), or register an IServiceLevelProvider plus ServiceLevelStartupTask, when clients and Kubernetes readiness need live health or leader-state values.

On the standard server, ServiceLevelStartupTask claims explicit ownership via IServerServiceLevelControl. Only one provider can claim the node for that server's lifetime; a second claim fails instead of creating competing writers. Provider publications and session-headroom updates share the server's internal coordination, and session churn cannot overwrite any provider-owned value, including Healthy-band standby/load levels such as 210 or a fixed 255. Without an explicit owner, the session-headroom heuristic is unchanged. Custom IServerContext implementations can implement this optional capability; contexts without it retain their existing direct provider-publication behavior.

services.AddOpcUa()
    .AddServer(server =>
    {
        // normal endpoint, security, and application configuration
    })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseDistributedAddressSpace(options =>
    {
        options.UseLeaderElection = true;
        options.NodeId = Environment.MachineName;
    })
    .AddServerRedundancy(options =>
    {
        options.Mode = RedundancySupport.Warm;
        options.CurrentServerId = Environment.MachineName;
        options.AddRedundantPeer(
            "urn:example:ha-server-2",
            new ArrayOf<string>(["opc.tcp://ha-server-2:4840"]));
        options.PeerServiceLevel = ServiceLevels.DegradedMaximum;
    })
    .AddServerServiceLevel(new LeaderServiceLevelProvider(leaderElection, RedundancySupport.Warm))
    .AddRequestServerStateChange(options =>
    {
        options.ServiceLevelSelector = state => state == ServerState.Running
            ? ServiceLevels.Maximum
            : ServiceLevels.Maintenance;
    });

ServiceLevel and load balancing (as per Part 4 §6.6.2.4.2 and §6.6.2.4.3)

OPC 10000-4 Table 105 defines ServiceLevel as a byte split into mandatory sub-ranges. The stack exposes these constants and helpers in ServiceLevels and ServiceLevelSubrange.

Range	Name	Client interpretation
0	Maintenance	New clients should not connect; connected clients should disconnect and use EstimatedReturnTime before retrying.
1	NoData	Server is not operational for process data; clients may connect only for status/diagnostics.
2..199	Degraded	Server is partially operational; clients should prefer any Healthy peer and may choose a higher degraded peer when no Healthy peer exists.
200..255	Healthy	Server is fully operational; clients connect to the highest value for selection/load balancing and should not fail over while the current server remains Healthy.

ConstantServiceLevelProvider reports a fixed value, defaulting to 255, preserving single-instance behavior. LeaderServiceLevelProvider follows an ILeaderElection: the leader reports 255, Cold standbys report 1, Warm standbys report 199, and Hot/HotAndMirrored standbys report 255 unless explicit levels are supplied. Optional health and connected-client delegates cap or decrement Healthy values so Hot servers can load-balance within the 200-255 range. IServiceLevelController lets RequestServerStateChange override the published value for manual maintenance.

SharedStoreLeaseElection bounds local authority by the last successfully confirmed lease. Its injected TimeProvider drives both renewal and an independent expiry timer, so failed or blocked store operations cannot keep a replica authoritative past that deadline. IsLeader also checks the deadline when read, without invoking application callbacks. The expiry timer raises LeadershipChanged(false) to update dependent service levels. Subscriber failures are logged without interrupting other subscribers or lease operations. Lease validity starts at the write attempt, not at receipt of its reply; an expired or superseded operation cannot restore leadership. A fresh, confirmed acquisition is required after expiry. Overlapping successful calls can confirm the same owned lease without waiting for one another. A failed store observation started before a newer successful confirmation cannot revoke or reschedule that confirmed lease; a fresh ownership-loss observation still revokes authority. Expiry and disposal invalidate all outstanding attempts. The UTC lease record is also bounded by local elapsed time, so moving the local clock backwards cannot extend authority. Replicas still require unique identities, suitably synchronized clocks and a linearizable compare-and-swap store; this local safety mechanism does not replace backend fencing or provide consensus.

Client-side, DefaultServerRedundancyHandler.FetchRedundancyInfoAsync reads RedundancySupport, ServiceLevel, EstimatedReturnTime, RedundantServerArray, ServerUriArray, and CurrentServerId as applicable. ServerRedundancyInfo.ServiceLevelSubrange is calculated with ServiceLevels.GetSubrange.

ManagedSession stores the default handler's basic snapshot as soon as that read succeeds, within the existing two-second metadata-read bound. Peer discovery runs separately in the background, in parallel with a two-second bound per peer; connection establishment does not wait for every backup. A peer timeout leaves its Endpoint unresolved without discarding the mode, peer URIs or other resolved endpoints. Caller cancellation is propagated, and session disposal cancels and drains the background refresh.

ManagedSession session = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(endpoint)
    .WithServerRedundancy(new DefaultServerRedundancyHandler())
    .ConnectAsync(ct);

var handler = new DefaultServerRedundancyHandler();
ServerRedundancyInfo info = await handler.FetchRedundancyInfoAsync(session, ct);
Console.WriteLine($"Mode={info.Mode}, ServiceLevel={info.ServiceLevel} ({info.ServiceLevelSubrange})");

ServerFailoverDecision decision = handler.ShouldFailover(info, session.ConfiguredEndpoint);
if (decision.IsFailoverWarranted)
{
    ConfiguredEndpoint? target = handler.SelectFailoverTarget(info, session.ConfiguredEndpoint);
    Console.WriteLine($"Fail over to {target?.EndpointUrl}: {decision.Reason}");
}

Non-transparent failover modes and client actions (as per Part 4 §6.6.2.4.5)

ManagedSession implements the Table 107 client patterns over ManagedSession instances discovered from RedundantServerArray/ServerUriArray and resolved with IRedundantServerEndpointResolver. The default resolver uses FindServers at the current endpoint to locate peers, then requires GetEndpoints results to match the peer application URI, security policy/mode and URL scheme.

The resolver retains a peer's discovery addresses before contacting that peer. Thus a Cold backup that was offline at connect time can be resolved at failover without another FindServers call to the failed primary. Failed resolutions are not cached as permanent misses, and a failed failover invalidates the resolved endpoint so the next attempt can discover it again. Cached addresses never bypass endpoint or session security validation. If no peer address was learned before the primary became unreachable, a reachable discovery service or a custom IRedundantServerEndpointResolver is still required to translate its application URI.

Mode	Table 107 actions	ManagedSession realization
Cold	Initial connection to one active server. At failover open a SecureChannel, create/activate a session, create subscriptions/monitored items, activate sampling, then activate publishing.	Starts with the initial endpoint only. FailoverAsync selects a peer, connects it, applies subscriptions only to the current session, and enables reporting/publishing.
Warm	Connect to more than one server and create subscriptions/monitored items on backups; sampling and publishing become active at failover.	Connects all peers, stores templates with AddSubscriptionAsync, creates subscriptions on each connected server, and keeps only the selected server in Reporting/publishing while backups are Sampling/publishing disabled.
Hot (a)	Connect to more than one server, create subscriptions everywhere, activate sampling everywhere, publish from one server; at failover activate publishing on the next server.	Default HotRedundancyNotificationMode.ReportingHandoff: all connected peers host subscriptions; current server Reports and publishes, backups Sample only.
Hot (b)	Connect to more than one server, create subscriptions everywhere, activate sampling and publishing everywhere; client handles duplicate streams.	HotRedundancyNotificationMode.ReportingMerge: all connected peers Report and publish; duplicate suppression/merge is a client responsibility above the event callback.
HotAndMirrored	Client normally connects to one server; optional status sessions only; on failover open a SecureChannel and call ActivateSession against the mirrored session.	Keeps a single active session and, when EnableHotAndMirroredStatusChecks is set, periodically reads ServiceLevel from backup status sessions. FailoverAsync reuses the mirrored AuthenticationToken and single-use nonce by calling ActivateSession on the backup; it does not recreate subscriptions because the server mirrors session and subscription state.

ManagedSession.RefreshServiceLevelsAsync reads service levels from connected sessions and selects the highest value. The default handler will not fail over away from a server that is still in the Healthy sub-range, defers retries for Maintenance using EstimatedReturnTime or a longer default backoff, and selects a Running peer with an operational ServiceLevel.

ManagedSession coldClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(ct: ct);

await coldClient.AddSubscriptionAsync("process", subscriptionTemplate, ct);
await coldClient.FailoverAsync(ct);

ManagedSession warmClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(ct: ct);

await warmClient.AddSubscriptionAsync("process", subscriptionTemplate, ct);
await warmClient.RefreshServiceLevelsAsync(ct);

Hot handoff is the default Hot mode: every connected peer hosts the subscriptions and samples, but only the active server publishes, so the client sees a single stream and the next server starts publishing on failover. Use it for fast failover without client-side de-duplication.

ManagedSession hotHandoffClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(
        new ManagedSessionOptions
        {
            HotNotificationMode = HotRedundancyNotificationMode.ReportingHandoff
        },
        ct);

Hot merge has every connected peer publish at once, so the client receives duplicate streams with near-zero failover latency and merges or de-duplicates them in the NotificationReceived handler (by application key, source timestamp, EventId, or sequence context). Use it when the lowest possible failover gap is worth handling duplicates.

ManagedSession hotMergeClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(
        new ManagedSessionOptions
        {
            HotNotificationMode = HotRedundancyNotificationMode.ReportingMerge
        },
        ct);

hotMergeClient.NotificationReceived += (sender, args) =>
{
    // Merge or de-duplicate streams by application key, source timestamp, EventId, or sequence context.
};

For a HotAndMirrored server, the client keeps one active session and, with EnableHotAndMirroredStatusChecks, periodically polls the backups' ServiceLevel over lightweight status-check sessions so it can pick the healthiest survivor. Because the server mirrors session and subscription state, failover re-activates the mirrored session on the survivor and does not recreate subscriptions. Use it when the server set runs HotAndMirrored (or Transparent) mirroring.

ManagedSession mirroredClient = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(initialEndpoint)
    .ConnectRedundantAsync(
        new ManagedSessionOptions
        {
            EnableHotAndMirroredStatusChecks = true,
            HotAndMirroredStatusCheckInterval = TimeSpan.FromSeconds(5)
        },
        ct);

Manual failover and Maintenance (as per Part 4 §6.6.5)

OPC 10000-4 §6.6.5 allows a server to be taken out of the set by shutdown or by moving ServiceLevel to Maintenance, either through a vendor tool or Server.RequestServerStateChange. AddRequestServerStateChange(...) wires the standard method and validates administrative access through IConfigurationNodeManager.HasApplicationSecureAdminAccess unless RequestServerStateChangeOptions.AdminAccessValidator is supplied.

The startup task updates Server.ServiceLevel, Server.ServerStatus.State, Server.ServerStatus.SecondsTillShutdown, Server.ServerStatus.ShutdownReason, and Server.EstimatedReturnTime. The current implementation publishes the requested maintenance/no-data state so clients back off; it does not install a transport-level hook that rejects newly created sessions.

Non-transparent servers registered with AddServerRedundancy(...) advertise the NTRS capability by default. In Kubernetes deployments, register the non-transparent set or transparent virtual address with GDS/NTRS according to your discovery model.

HotAndMirrored and Transparent state mirroring

OPC 10000-4 requires HotAndMirrored and Transparent sets to synchronize enough state that a client can continue after failover. The spec names sessions, subscriptions, registered nodes, continuation points, sequence numbers, sent notifications, and synchronized EventIds.

Every replica reads and writes that state through a shared store. One replica serves the client's active session; the others stand by and continuously mirror the session, subscription, retransmission, continuation-point, EventId, and node-state records so any standby can resume the client after a failover. Records are serialized through IRecordProtector, so the store is treated as an untrusted conduit.

flowchart LR
    Client -->|active session| R1["Replica 1 (active)"]
    R1 -. mirror .-> Store[("Shared store<br/>(IRecordProtector)")]
    R2["Replica 2 (standby)"] -. mirror .-> Store
    R3["Replica 3 (standby)"] -. mirror .-> Store
    R1 -. failover .-> R2

You enable mirroring by registering the seams below on the server builder; each one mirrors one category of state and is independently opt-in:

• UseDistributedSessions(...) installs DistributedSessionManager and ISharedSessionStore. Session records include the session id, authentication token, nonces, client certificate chain, security policy/mode, endpoint URL, session timeout, client description, and user identity material. EnableFastReconnect defaults to false; when enabled, a failover reconnect still performs full ActivateSession signature validation against the mirrored server nonce. (UseDistributedSessionMirroring(...) is the mirroring-only variant: it mirrors the session record — keyed by a digest of the authentication token and protected by the configured IRecordProtector — without replacing the session manager.)

• ISingleUseNonceRegistry and SharedSingleUseNonceRegistry enforce the security boundary: the mirrored serverNonce is consumed exactly once across the replica set, and the authentication token is only a lookup key. The nonce is consumed only after the restored activation has passed the client signature, user identity and continuity checks, so a failed or abandoned attempt (a wrong identity, a slow identity provider, a client-side timeout) leaves the session restorable on this or another replica; a restored session whose first activation fails is discarded again without touching the shared entry.

• The replica that serves a session owns its record. While the client uses the session, the owner mirrors a liveness heartbeat (DistributedSessionOptions.LivenessInterval, default one second; written at most once per quarter of the session timeout), so a session in normal use stays restorable long after its last activation — a session times out after its last Service request, not its last activation (OPC 10000-4 5.7.2.1). Only the owner mirrors activations and deletes the record. A restore takes ownership with a conditional (compare-and-swap) write; the previous replica then drops its stale copy without deleting the record, also when the client fails back to it. Over a store without compare-and-swap (the CRDT gossip store) the conditional writes fall back to compare-then-write.

• A restored session keeps the SessionId the client received from CreateSession, so its diagnostics nodes, audit events and mirrored continuation points stay addressable across any number of failovers. To make that safe, DistributedSessionManager assigns new sessions a GUID SessionId that is unique across the replica set instead of a per-process counter value.

• Replicas of a non-transparent (HotAndMirrored) set have their own ApplicationUri and certificate. The record carries the certificate of the server that created the session, and a restored session accepts a client signature over either that certificate or the restoring replica's own one (the stack's client signs the failover server's certificate). Encrypted user tokens must still be decryptable by the restoring replica: with RSA user token policies the client encrypts for the certificate it signs, and a session whose user token is encrypted with an ECC or RSA-DH EphemeralKey is not restored at all, because the private part of that key never leaves the replica that issued it — the reconnect is rejected before anything is consumed and the client re-creates the session.

• A periodic sweep (SweepInterval, default five minutes) removes the records of sessions that can no longer be restored — for example those of a replica that crashed or shut down, which nobody closes — and the consumed-nonce markers that no live record references and that are older than NonceMarkerRetention (default one hour). A restore that finds an expired record deletes it as well.

• SharedKeyValueSessionStore and DistributedSessionManagerFactory refuse an external (non in-memory) store without an IRecordProtector, like the dependency injection registration; pass NullRecordProtector.Instance explicitly to knowingly store session secrets unprotected.

• Mirrored session records carry a SecurityStateVersion. A replica only restores a Session from a record written at the version it understands; a record written by a peer running an older version is treated as missing security state and fails closed, so a rolling upgrade degrades to a normal Session recreate rather than comparing identity state it cannot interpret. Version 4 also mirrors whether the client application certificate passed validation when the Session was created; a Session whose certificate error an OnApplicationCertificateError override accepted is restored without the TrustedApplication role or application-based role mappings, like the original. A version 3 record is still restored, with its certificate treated as not validated. Version 5 adds the liveness heartbeat, the owning replica, the original server certificate and whether the user token needs an EphemeralKey; version 3 and 4 records are still restored, without an owner. The stored user identity is an encoded continuity key (token type plus a delimited issuer and subject), not the human-readable ClientUserIdOfSession diagnostic value. Activation records are stamped with a per-Session activation sequence so that when two activations overlap the mirror keeps the newest serverNonce instead of whichever write happens to land last.

• UseDistributedSubscriptionMirroring(...) registers SharedKeyValueSubscriptionStore as ISubscriptionStore. It mirrors subscription definitions and monitored-item definitions: publishing interval, lifetime, keepalive, priority, node id, attribute id, monitoring mode, sampling interval, queue size, filters, discard policy, and related metadata. It also installs SharedKeyValueMonitoredItemQueueFactory as the IMonitoredItemQueueFactory, which continuously mirrors each monitored item's data/event queue contents to the shared store through a non-blocking, coalesced background drain (protected at rest by the configured IRecordProtector); on promotion the store's asynchronous restore members re-hydrate those queues so queued-but-unpublished notifications survive a failover.

• The same store also implements ISubscriptionRetransmissionStore. Retransmission state is mirrored asynchronously through a non-blocking background drain: NextSequenceNumber, sent NotificationMessage entries, acknowledgements, and deletes are coalesced and persisted so Republish can continue after failover without blocking the publishing path.

• IContinuationPointStore mirrors the generic Browse continuation envelope.

• IHistoryContinuationPointStore separately persists versioned, protected HistoryRead state for providers that expose a stable IHistorianProviderIdentity and portable resume tokens. Persistence is awaited before the continuation point is returned, and restore atomically claims the record so only one replica can resume it.

• IEventIdProvider is an optional event-publishing seam. DeterministicEventIdProvider derives EventIds from a shared replica-set seed and stable event fields so Transparent/HotAndMirrored replicas can publish the same logical EventId and clients do not double-process events; the default remains the existing random GUID EventId behavior.

• RegisterNodes returns the input NodeIds in this stack, so registered-node handles are already replica-consistent when the AddressSpace NodeIds are identical.

• UseDistributedAddressSpace(...) mirrors node topology and values through INodeStateStore, helping each server present the same NodeIds and browse paths. Each node manager is restricted to its declared non-standard NamespaceUris; namespace-zero infrastructure remains replica-local, and a replicated root may not contain foreign descendants. Custom and shared-namespace partitions implement ILocalAddressSpaceOwnership with a stable PartitionId and an OwnsNode(NodeId) predicate. Hydration updates existing node graphs in place, preserving concrete node types, callbacks, and handles. Tracked subtree membership is independent of mutable child links, so deleting or replacing a graph also detaches its old replication callbacks.

  On stores with authoritative reads, a protected partition marker distinguishes a completed empty partition from an interrupted initial seed. Validated snapshots and deltas provide a bulk-hydration path; missing or invalid snapshot chunks trigger complete streamed fallback. A corrupt record under a valid node/value key fails that scan rather than becoming an authoritative omission. Failed snapshot publication leaves the previous manifest and delta log intact.

  Startup buffers the live feed during hydration. The most recent accepted topology sequence determines both graph changes and root membership; rejected stale deletions cannot prune newer roots during cleanup. Value ordering remains independent of topology ordering. Values arriving before their target node are retained and applied when its topology becomes available, rather than being marked as delivered while the node is absent. An accepted parent tombstone prevents older pending values from crossing into recreated descendants. See the consistency requirements below for eventually consistent payload stores, where absence is not authoritative.

Notes:

• Hydration fully materializes the mirrored node graph (topology plus values): the snapshot + delta log reduces the store round trips and per-node work, but the whole graph is resident. True on-demand fault-in (materializing a node only when it is first browsed/read) would further cut time-to-ready and memory for very large, sparsely-accessed graphs; it requires an asynchronous node-resolution seam through the core node-manager read/browse path and is tracked as future work in OPCFoundation/UA-.NETStandard#3938. The eventual-consistency conflict-free replicated data type (CRDT) path already exchanges a compact state snapshot plus deltas.
• SharedKeyValueSubscriptionStore restores subscription definitions, retransmission state, and — when the shared-store SharedKeyValueMonitoredItemQueueFactory is installed (the default for UseDistributedSubscriptionMirroring) — the per-monitored-item data/event queues. The monitored-item creation path pre-fetches each queue asynchronously through ISubscriptionStore.RestoreDataChangeMonitoredItemQueueAsync/RestoreEventMonitoredItemQueueAsync and hands the pre-hydrated queue to the (still synchronous) MonitoredItem constructor, so a networked store never blocks the creation path; the original synchronous RestoreDataChangeMonitoredItemQueue/RestoreEventMonitoredItemQueue remain as the fallback for local/durable stores. Queue contents are mirrored continuously as they are enqueued, so after a failover the promoted replica resumes with the values that were queued-but-not-yet-published on the failed replica instead of losing them; Republish of already-sent notifications is likewise preserved. Snapshots are coalesced per monitored item (the latest state wins), so the mirror cost scales with the unpublished queue tail rather than every sampled value.
• Generic Browse continuation-point mirroring remains best-effort because a custom node manager's opaque ContinuationPoint.Data cannot be reconstructed automatically. Built-in historian continuations are recoverable when the provider identity matches on the promoted replica and its resume token is explicitly portable; otherwise HistoryRead fails closed with BadContinuationPointInvalid.
• Deterministic EventIds (DeterministicEventIdProvider) are opt-in rather than on by default: they change the EventId values a server emits — a single-server deployment keeps the standard random GUID EventIds — and they require a shared replica-set seed. Enable them for a HotAndMirrored/Transparent set so every replica emits the same logical EventId and clients do not double-process events. They are only as stable as the event fields used, so Alarms & Conditions clients should still call ConditionRefresh after failover as required by OPC UA.

Active/passive address-space consistency

InMemoryNodeStateStore requires a linearizable coordinator for election/addressspace-sequence. A bare ReplicatedSharedKeyValueStore cannot provide this primitive. Use UseRedundancyConsistency(...) before UseDistributedAddressSpace(...): the hybrid mode keeps CRDT payloads on the bulk store and routes the sequence key to shared Raft. The address-space registration contributes its coordination and configured lease keys even when the application's strong-prefix list is customized. Startup validates the coordinator before election or shared-state mutation; direct writes enforce the same requirement. A process-local coordinator is allowed only with process-local payload storage, not as a substitute for shared coordination.

For direct construction, explicitly compose existing stores:

await using var hybrid = new HybridSharedKeyValueStore(
    crdtPayloadStore, sharedRaftStore);
using var stateStore = new InMemoryNodeStateStore(
    hybrid, messageContext, recordProtector);

The identity contract is protected at election/addressspace-identity/v1 and also requires strong routing. A new authoritative store initializes it atomically. An existing unbound store is rejected. For a verified new hybrid deployment, provision the matching contract on the fresh shared Raft backend before attaching the also-new CRDT payload backend; an empty eventual scan cannot authorize bootstrap. See stored identity contracts.

Both backend instances above are supplied by the application and retain their existing disposal ownership. Every writer replica must use the same replicated Raft coordination state, not a separate single-node development coordinator.

Strong sequence allocation does not make CRDT payload reads linearizable. With CRDT bulk data, hydration applies observed updates and sequenced tombstones but does not infer deletion, initial seeding, or partition initialization from missing rows. Local predefined graphs must be provisioned independently. Automatic snapshots/compaction are unavailable, explicit snapshot publication is rejected, and deltas are retained so a delayed older primary-row write cannot hide a newer completed update. Select RedundancyConsistencyMode.Strong when authoritative bootstrap, missing-root cleanup, and compacted snapshots are required. Coordinator-free active/active replication remains the separate UseReplicatedAddressSpace(...) module.

Sequence reservations are not completed publications. Protected pending reservations survive uncertain writes and prevent snapshot publication or log trimming past unfinished work, including after restart. Concurrent snapshot attempts fail instead of publishing an unvalidated cut. Failed or cancelled publications require reconciliation before compaction can resume; the module does not automatically discard their reservations or assume the payload was never written.

Strong active/passive historian

UseDistributedHistorian(...) places raw, modified, at-time, processed, annotation, structured, and event history behind the same linearizable ISharedKeyValueStore used by the strong active/passive topology. Immutable segments are published by one manifest compare-and-swap. Each provider instance has a unique writer id, and a newly active writer publishes a higher writer epoch before it can append; a stale former writer is then rejected with BadNotWritable.

HistoryRead cursors pin an immutable manifest generation. The separate IHistoryContinuationPointStore persists the versioned, protected server cursor before returning it, then atomically claims it during restore. A token-reuse session takeover therefore preserves the continuation owner and lets the promoted replica resume raw, event, and processed reads without shifting pages, duplicates, or gaps. Superseded generations are retained for the configured continuation lifetime.

This module intentionally has no eventual or active/active adapter. Ordered historical values, deletes, modification chains, events, and atomic update batches do not have a safe CRDT merge contract, so requesting the distributed historian outside a strong active/passive topology fails startup. NullRecordProtector is also rejected: shared archive and continuation records must be authenticated.

The redundancy samples exercise both guarantees: the client writes and reads a distinctive raw-history marker through the active replica, resumes paged raw, event, and processed reads after terminating that process, then reads the same marker and observes new raw/event writes through the promoted replica.

Client redundancy (as per Part 4 §6.6.3)

OPC UA client redundancy is implemented with TransferSubscriptions plus server diagnostics. ClientFailoverCoordinator helps a backup client find the active client's session by ActiveSessionId or ActiveSessionName, discover subscription ids from diagnostics, verify the backup uses the same user display name when configured, and call TransferSubscriptionsAsync with SendInitialValues defaulting to true.

Name-based discovery excludes the backup's own session and rejects multiple matching active sessions rather than selecting an arbitrary client. Supply ActiveSessionId when names are not unique; that explicit identity bypasses name-based discovery.

var coordinator = new ClientFailoverCoordinator();
ArrayOf<TransferResult> results = await coordinator.TransferActiveSubscriptionsAsync(
    backupSession,
    new ClientRedundancyTransferOptions
    {
        ActiveSessionName = "LineA-PrimaryClient",
        ActiveUserDisplayName = backupSession.Identity.DisplayName,
        SendInitialValues = true
    },
    ct);

A transferred subscription needs a client-side owner on the backup session, otherwise the backup's publish engine sees an unknown SubscriptionId and deletes it (or never publishes and it expires). Prepare the backup before the takeover: restore the active client's subscriptions with ISession.Load (or add Subscription objects whose TransferId is the active client's subscription id and whose monitored items carry the same client handles). The coordinator then transfers those through the session, which binds them and resumes publishing; ids without a prepared subscription are still moved with the raw service and are left to the caller.

OPC UA does not standardize how active and backup clients exchange SessionId or subscription ids. In this stack the client replica set coordinates through the registered client-side shared store (AddRedundantClientSharedStore / AddRaftClientSharedStore — a CRDT- or Raft-backed ISharedKeyValueStore that the ClientReplicaCoordinator consumes).

Network redundancy (as per Part 4 §6.6.4)

Transparent network redundancy is handled below OPC UA by routers, virtual adapters, load balancers, or Kubernetes Services. Non-transparent network redundancy uses multiple endpoints for the same logical server; the client selects another endpoint and recreates only the SecureChannel while reusing the session and subscriptions.

NetworkRedundancyOptions.AlternateEndpoints and NetworkRedundancyEndpointSelector model non-transparent endpoint alternates. ManagedSession accepts these options and cycles to the next endpoint for the same logical server when reconnecting.

ManagedSession session = await new ManagedSessionBuilder(configuration, telemetry)
    .UseEndpoint(primaryEndpoint)
    .WithNetworkRedundancy(new ArrayOf<ConfiguredEndpoint>([secondaryEndpoint, tertiaryEndpoint]))
    .ConnectAsync(ct);

Beyond §6.6: distributed extensions

The base package uses ISharedKeyValueStore as the common seam for address-space, session, subscription, retransmission, nonce, and lease records. The in-memory implementation is for tests and single-process samples; multi-pod production deployments need a networked, authenticated, encrypted, and capacity-bounded backend. IRecordProtector protects serialized records before they reach the store.

The same seam also backs a shared PKI: SharedKeyValueCertificateStore keeps the trusted, issuer and rejected certificate lists and their CRLs in the shared store so every replica makes the same trust decision, with fail-closed record integrity and read-through live propagation. See Shared certificate stores (distributed trust lists) below.

flowchart LR
    Client1[Client 1] --> A
    Client2[Client 2] --> B
    A["Replica A"] <-->|CRDT gossip| B["Replica B"]
    B <-->|CRDT gossip| C["Replica C"]
    A <-->|CRDT gossip| C
    A -.->|strong keys| Raft[("Raft<br/>(linearizable)")]
    B -. strong keys .-> Raft
    C -. strong keys .-> Raft

Three active/active replicas converge bulk state (address space, sessions) leaderlessly over CRDT gossip, while the exactly-once keyspaces ride the linearizable Raft store. Clients connect to any replica (each is a writer) and fail over between them.

OPCFoundation.NetStandard.Opc.Ua.Redundancy.Server goes further than OPC 10000-4 §6.6. It provides active/active multi-writer address-space replication with CRDTs and gossip (UseReplicatedAddressSpace) and CRDT-backed session metadata (UseReplicatedSessions). CRDT state is eventually consistent and cannot provide compare-and-swap; the single-use nonce registry and other areas that require exactly-once behavior must and are by default mapped to a strongly consistent store (see Consistency modes).

Selecting active/active is a single call — UseActiveActiveRedundancy wires both the replicated address space and the replicated session store from one set of gossip options (session gossip runs on GossipPort + 1 by convention):

services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseActiveActiveRedundancy(aa =>
    {
        aa.ReplicaId = Crdt.ReplicaId.New();
        aa.GossipPort = 4840;
        aa.Tls = mutualTlsOptions;      // required unless AllowUnauthenticatedGossip is set
        aa.EnableFastReconnect = true;  // mirrored sessions fast-reconnect after failover
        aa.AddPeer(peerEndpoint);       // static seed; or configure peer discovery (below)
    });

The individual methods are available for advanced setups that need to diverge from the shared defaults (for example a different session gossip port or transport):

services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseReplicatedAddressSpace(options =>
    {
        options.ReplicaId = Crdt.ReplicaId.New();
        options.UseTcpGossip(IPAddress.Any, port: 4840, tls: mutualTlsOptions);
        options.AddPeer(peerEndpoint);
    })
    .UseReplicatedSessions();

Networked CRDT gossip fails closed unless peers are authenticated. Configure TCP gossip with mutual TLS (GossipTlsOptions with a server certificate, required client certificates, a client certificate, and remote certificate validation). UDP gossip has no built-in peer authentication and should only be enabled with AllowUnauthenticatedGossip inside an isolated development/test network. Address-space values are not secrets, but LWW CRDT frames require integrity/authenticity because a forged high-clock update wins convergence.

For Kubernetes, add a NetworkPolicy for the gossip port in addition to any Kubernetes API or shared-store policies. Allow ingress and egress only between the replica pods that participate in the CRDT fabric, and keep the gossip port closed to clients, other namespaces, and infrastructure that is not part of the replica set.

Dynamic peer discovery (beyond §6.6, opt-in)

Static peer lists work for a fixed replica set, but an elastically-scaled deployment (a Kubernetes Deployment/StatefulSet, an autoscaling group) adds and removes replicas at runtime. The IPeerDiscovery seam keeps the peer set current without a restart: a discovery mechanism periodically resolves the peers and updates (a) the client-facing RedundantServerSet published through FindServers (OPC 10000-4 §6.6.2.4.5.1) and (b), for active/active, the CRDT gossip fabric. Static configuration (AddServerRedundancy) is the fallback used until discovery finds peers.

Mechanism	API	Source
Static (fallback)	UseStaticPeerDiscovery(peers)	A fixed configured peer set.
DNS (dependency-independent)	UseDnsPeerDiscovery(options)	One A/AAAA record per replica — for example a Kubernetes headless service. No cloud SDK required.
LDS / LDS-ME	UseLdsPeerDiscovery(findServers)	FindServers / FindServersOnNetwork against a Local Discovery Server. The FindServers call is supplied as a delegate so the server package stays free of a client dependency.
Kubernetes	UseKubernetesPeerDiscovery(...) (Opc.Ua.Redundancy.Kubernetes)	EndpointSlices for the server Service, bridged into the generic seam.
Custom	UsePeerDiscovery(factory)	Any IPeerDiscovery implementation.

services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseActiveActiveRedundancy(aa => aa.GossipPort = 4840)
    .AddServerRedundancy(r => r.Mode = RedundancySupport.HotAndMirrored)  // static fallback
    // Dynamic: a Kubernetes headless service resolves to one address per replica.
    .UseDnsPeerDiscovery(dns =>
    {
        dns.HostName = "servers.default.svc.cluster.local";
        dns.GossipPort = 4840;       // CRDT fabric endpoint
        dns.ApplicationPort = 4840;  // OPC UA endpoint published to clients
    });

Each discovered peer carries a client-facing identity (ServerUri + DiscoveryUrls) and, optionally, a CRDT gossip endpoint. The refresh loop publishes the identities to the RedundantServerSet and pushes any gossip endpoints into the live active/active transports through an IGossipPeerSink. The gossip transport is add-only, so a scaled-up replica is added to the fabric immediately, while a decommissioned peer is not actively removed — it ages out as gossip to a dead endpoint simply fails and is retried. LDS and Kubernetes discovery yield OPC UA endpoints (not gossip endpoints), so pair them with DNS discovery of the gossip headless service when you also want the gossip fabric to scale dynamically. Discovery inherits the same secure-gossip defaults: newly-added peers join an already-authenticated transport, so mutual TLS still governs the fabric.

CRDT gossip is eventually consistent — it stays available under network partitions but is not linearizable: it converges without a leader but cannot offer a linearizable compare-and-swap or a change-feed, so exactly-once primitives — the single-use session nonce, lease/leader election — need a strongly consistent (linearizable, quorum-based) backend. OPCFoundation.NetStandard.Opc.Ua.Redundancy provides that backend as a Raft layer — Raft is the same consensus algorithm used by systems like etcd and Kafka to replicate state — and UseRedundancyConsistency lets you pick the model for the whole RedundantServerSet:

• Eventual (default). Bulk replicated state (address space, sessions, subscriptions) stays on the leaderless CRDT store; a HybridSharedKeyValueStore routes only the strong keyspaces (nonce/, lease/, election/ by default, configurable) to a linearizable RaftSharedKeyValueStore. Behaviour and performance for the common case are unchanged, and the single-use nonce registry and lease election become linearizable with no extra backend.
• Strong. All shared state is served by the linearizable RaftSharedKeyValueStore; no CRDT is used. Choose this when every replicated value must be linearizable, accepting that writes require a quorum.

services.AddOpcUa()
    .AddServer(server => { })
    // Pick the model BEFORE the distributed features so they compose over it.
    .UseRedundancyConsistency(options =>
    {
        options.Mode = RedundancyConsistencyMode.Eventual; // or Strong
        // options.BulkStoreFactory = sp => /* CRDT bulk store */;    // eventual mode
        // options.RaftConsensusFactory = sp => /* RaftCs replica */; // multi-pod
    })
    .UseReplicaNodeIdentity("example-set", ["urn:example:model", "urn:example:instances"])
    .UseDistributedAddressSpace()
    .UseDistributedSessions(o => o.EnableFastReconnect = true);

UseRedundancyConsistency registers a shared IRaftConsensus plus a native RaftLeaderElection (ILeaderElection). Raft leadership is decided by the consensus protocol itself — a single leader per term, no split-brain — which is stronger than the lease-CAS SharedStoreLeaseElection. Because the distributed features register their store and election with TryAddSingleton, call UseRedundancyConsistency first and they compose over the chosen store and election.

The consensus engine is pluggable through the IRaftConsensus seam. By default the DI registration uses a single-node in-package Raft replica (DefaultRaftConsensus.CreateSingleNode(), a real RaftNode with in-memory storage/transport that elects itself leader); InProcessRaftConsensus is a lighter deterministic in-process alternative for tests and in-process replica sets. For a multi-pod cluster, build a multi-node replica with DefaultRaftConsensus.CreateCluster(nodeId, memberIds, transport, …) — a RaftCs.Transport.NanoMsg transport (peers addressed by DNS) plus an in-memory or durable RaftCs.Storage.File write-ahead log (WAL) — and register it through RaftConsensusFactory. On Kubernetes, UseKubernetesRaftConsensus (in Opc.Ua.Redundancy.Kubernetes) does this for you from the StatefulSet ordinal and headless-Service DNS. RaftCs ships its own IRaftTransport (NanoMsg), so Raft shares the NanoMsg substrate with the CRDT gossip layer. The samples/Redundancy/RedundantServer sample (docker-compose.yml with the active-passive.env env file) is a runnable multi-node example.

On the client side, AddRaftClientSharedStore registers a Raft-backed ISharedKeyValueStore and a native RaftLeaderElection for a client replica set in one call (the ClientReplicaCoordinator consumes both), AddCrdtClientSharedStore registers the CRDT-gossip (eventual) equivalent, and AddRedundantClientSharedStore(mode, …) is the mode-aware entry point that selects between them.

For Kubernetes, run the Raft members as a StatefulSet with stable network identities (so each Raft peer keeps its node id and address across restarts) and an odd member count (3 or 5) for a fault-tolerant quorum; a durable RaftCs.Storage.File WAL on a PersistentVolume lets a restarted pod rejoin without a full snapshot. See the Kubernetes High Availability Deployment guide.

Sharing values across replicas: distributed value cache (beyond §6.6, opt-in)

OPC 10000-4 §6.6.2.2 requires identical NodeIds and AddressSpaces across a RedundantServerSet but does not standardize how live process values are cached or shared between replicas. Opc.Ua.Redundancy.Server adds an opt-in provider for this: IDistributedValueCache (default implementation DistributedValueCache) stores a node's last value in the distributed INodeStateStore and serves it back within a freshness bound. UseDistributedAddressSpace registers IDistributedValueCache in dependency injection (routed per node through the server's INodeStateStoreRegistry, populated when the server starts); a consumer that constructs the node-state store directly can build a DistributedValueCache over it instead.

DistributedValueParticipation.EnableDistributedValueParticipation(variable, cache, maxAge, liveRead) wires a variable's asynchronous read/write callbacks to the cache: writes cache write-through, and reads serve the last cached value while it is fresh (falling back to liveRead and caching the result). This lets the active replica publish a value once and every replica serve it, and it lets a promoted replica seed from the last value the former leader shared so a simulated/derived value continues across a failover rather than restarting. Monitored items are unaffected — they read through the normal pipeline — so a monitored item observes the shared value only where the read path participates, consistent with OPC UA monitored-item semantics and the requirement that monitored items "read as today".

// Injected from DI (registered by UseDistributedAddressSpace).
IDistributedValueCache cache = /* ... */;

variable.EnableDistributedValueParticipation(
    cache,
    maxAge: TimeSpan.FromSeconds(10),
    liveRead: ct => new ValueTask<DataValue>(ReadLiveValue()));

// The active replica writes each new value through to the shared store;
// a promoted replica reads the last shared value on becoming leader.
await cache.CacheAsync(variable.NodeId, new DataValue(value, StatusCodes.Good, DateTimeUtc.Now));
(bool fresh, DataValue last) = await cache.TryGetAsync(variable.NodeId, maxAge);

Because the value cache writes through the shared INodeStateStore, its confidentiality/integrity is governed by the same IRecordProtector as the rest of the mirrored state (fail-closed for external stores without a protector). Cross-process value continuity therefore requires a store that propagates writes between replicas: it is verified on the strong-consistency (Raft) topology; on active/active eventual it depends on the CRDT address-space gossip transport. The samples/Redundancy/RedundantServer sample wires its HighAvailability.Counter through this provider — see the sample README's Sharing values across replicas section.

Shared certificate stores (distributed trust lists) (beyond §6.6, opt-in)

OPC UA replicas in a RedundantServerSet must present the same PKI: a certificate trusted (or rejected) on one replica should be trusted (or rejected) on every replica, and CRLs must be shared. By default each replica keeps its trusted, issuer and rejected lists in a local DirectoryCertificateStore (or Windows X509Store), so trust decisions can diverge across replicas.

SharedKeyValueCertificateStore stores the trusted, issuer and rejected certificate lists and their CRLs in the same ISharedKeyValueStore that backs the rest of the high-availability state, and plugs in through the existing certificate store provider model so any CertificateStoreIdentifier can point at it. It holds public certificates only (NoPrivateKeys is true, SupportsLoadPrivateKey is false); distributing the application instance certificate with its private key is a future capability.

Integrity, fail-closed. Trust-list integrity is critical: an attacker with write access to the shared store must not be able to inject a trusted CA. Every record (certificate or CRL) is written through an IRecordProtector and verified on read; a forged or tampered record fails the authenticity check and is skipped (fail-closed) — it is never returned to a validator. An external, network-reachable store must use an authenticating protector (for example AesCbcHmacRecordProtector, keyed from a Kubernetes Secret / KMS, the same record-protection model used for mirrored sessions and subscriptions); an in-memory, single-process store may use the no-op NullRecordProtector (the default).

Live propagation. The certificate validator enumerates the trusted/issuer store on each validation (it caches the store instance, not the certificate list), and SharedKeyValueCertificateStore.EnumerateAsync reads the current shared-store state on every call, so a certificate trusted or rejected on one replica is observed by the others' next validation automatically — no restart and no explicit refresh. On a network-backed store (Raft/Redis), reading on every validation costs a round-trip; a future enhancement can cache the enumerated list locally and use ISharedKeyValueStore.WatchAsync to invalidate it (and emit CertificateManager TrustListUpdated / CrlUpdated change events) when another replica changes the shared state, keeping the read path fast while remaining live.

using Opc.Ua;
using Opc.Ua.Security.Certificates;

// The shared store and protector come from the HA infrastructure
// (or an InMemorySharedKeyValueStore + NullRecordProtector for a single process).
var provider = new SharedKeyValueCertificateStoreProvider(sharedStore, protector);

CertificateManager manager = CertificateManagerFactory.Create(
    securityConfiguration,
    telemetry,
    options => options.AddStoreProvider(provider));

The provider is injectable — construct it from the DI-registered ISharedKeyValueStore and IRecordProtector and pass it through CertificateManagerOptions.AddStoreProvider; the direct construction above is the fallback. Point the security configuration's trusted/issuer/rejected stores at the distributed backend by setting each CertificateStoreIdentifier to store type CertificateStoreType.SharedKeyValue with a kv: store path (for example kv:pki/trusted, kv:pki/issuer, kv:pki/rejected); the built-in Directory and X509Store providers remain available. Every replica in the set uses the same shared store and the same paths, so they share one trusted list, one issuer list and one rejected store. The rejected-list trim (AddRejectedAsync) keeps the newest N entries best-effort (advisory, not security-critical). See Certificate Manager for the certificate lifecycle model.

Distributed PushManagement transactions (beyond §6.6, opt-in)

OPC UA Part 12 §§7.10.2-7.10.11 define a single, server-wide PushManagement configuration transaction: one Session owns it at a time, its UpdateCertificate / CreateSelfSignedCertificate / DeleteCertificate / TrustList changes are staged, and nothing takes effect until ApplyChanges. On a stand-alone server the default PushConfigurationTransactionCoordinator enforces this per process. In a RedundantServerSet that is not enough on its own: each replica has its own coordinator, so two administrators connected to two replicas could each open a transaction and stage conflicting certificate rotations at the same time.

UseDistributedPushConfiguration() (from Opc.Ua.Redundancy.Server) opts a server into a transaction that is single across the whole replica set, reusing the same high-availability building blocks as the rest of this document (the shared ISharedKeyValueStore, IRecordProtector and, optionally, the elected ILeaderElection). It is opt-in: a server that does not call it behaves exactly as before.

Cluster-wide single ownership. DistributedPushConfigurationTransactionCoordinator wraps the per-server coordinator and guards it with a compare-and-swap lease in the shared store — the same lease pattern used by SharedStoreLeaseElection for leader election. A replica must hold the lease to start a transaction, so at most one replica has an active transaction at any time. The lease is acquired (or renewed) at an await boundary immediately before the synchronous Stage, so the coordinator's synchronous Stage/CancelChanges contract is preserved with no sync-over-async; a background loop renews the lease while the transaction is active and releases it as soon as the transaction is applied, cancelled, reset, or its owning Session closes. A replica that crashes stops renewing, and once the lease expires a standby can start a new transaction. Another replica that tries to stage while the lease is held is refused with Bad_TransactionPending, exactly as a second Session on one server would be.

Pending regenerated keys across replicas. CreateSigningRequest(regeneratePrivateKey: true) (§7.10.10) produces a private key that must survive until a matching UpdateCertificate consumes it — possibly on a different replica after a failover. SharedKeyValuePendingCertificateKeyStore persists that key in the shared store (scoped per CertificateGroup/CertificateType) instead of a local directory. It consumes the key atomically (only one replica wins a concurrent TryTake), wipes every private-key working buffer, and — like every other mirrored record — protects each entry with an IRecordProtector (authenticated encryption) so the shared store can be treated as untrusted. An external, network-reachable store must provide a protector; without one the pending-key store fails closed rather than persist a private key unprotected, matching the fail-closed default of the distributed session, subscription and trust-list stores.

Composition with leader election. Distributed PushManagement composes transparently with whichever leader election is configured. By default the shared-store lease alone enforces single ownership, so any replica a client reaches may own the transaction. Setting RequireLeadership = true funnels every transaction through the elected leader; when the Kubernetes extension is configured the leader is decided by the Kubernetes-native Lease already registered by UseKubernetesLeaderElection — no second Kubernetes client is created. Raft and shared-store leader election compose the same way.

using Opc.Ua.Redundancy.Server;

builder
    // ... existing HA setup (shared store, record protector, leader election) ...
    .UseKubernetesLeaderElection()          // ILeaderElection = Kubernetes-native Lease
    .UseDistributedPushConfiguration(options =>
    {
        options.ReplicaId = Environment.GetEnvironmentVariable("POD_NAME")!;
        options.LeaseDuration = TimeSpan.FromSeconds(30);   // lease validity without renewal
        options.RenewInterval = TimeSpan.FromSeconds(10);   // background renew cadence
        options.KeyPrefix = "pushconfig/";                  // shared-store keyspace
        options.RequireLeadership = true;                   // only the K8s-elected leader owns the transaction
        // For an external store, configure authenticated encryption (or register an IRecordProtector):
        options.RecordProtectorFactory = sp => new AesCbcHmacRecordProtector(GetPushConfigKey());
    });

Give every replica a distinct ReplicaId, keep RenewInterval well below LeaseDuration, and share the same KeyPrefix and record-protection key across the set. See Certificate Manager — PushManagement Transactions for the underlying transaction model.

GetEndpoints load direction (beyond §6.6, opt-in)

UseServerLoadDirection(...) lets a GetEndpoints request on a dedicated balancing discovery URL be answered with the endpoints of the best member of the RedundantServerSet, so a Client that connects there is directed to the active server (active/passive) or spread across equally-healthy servers by load (active/active). This is a non-standard, server-side load-direction hint: it complements, and never replaces, the standard client-driven RedundantServerArray + ServiceLevel selection of OPC 10000-4 §6.6.2.4, which remains the authoritative Failover mechanism.

sequenceDiagram
    participant Client
    participant LB as Balancing discovery URL
    participant Set as RedundantServerSet
    participant Best as Best server
    Client->>LB: 1. GetEndpoints (balancing URL)
    LB->>Set: 2. Rank members by ServiceLevel tier, then load
    Set-->>LB: 3. Endpoints of the best member
    LB-->>Client: 4. Return the best server's endpoints
    Client->>Best: 5. CreateSession/ActivateSession to the target's own endpoint + certificate

How it works. Each member publishes three integrity-protected records to the shared store: its health ServiceLevel (svc/<uri>), a separate load weight (load/<uri>, 0 idle … 255 fully loaded, from an injected ILoadWeightProvider), and its own EndpointDescriptions (endpoint/<uri>, observed from normal discovery). A request that arrives on the balancing URL is resolved by the direction policy: rank members by health-ServiceLevel tier (Healthy › Degraded › NoData › Maintenance); if a peer is in a strictly higher tier it wins (active/passive → the active server), otherwise the least-loaded member of the top tier is chosen using the load weight, with random tie-breaking among equally-loaded members (active/active → load spreading). The local server serves its own endpoints when it is the best choice, when the peer view is stale, or when the target's endpoints are unavailable (fail-safe). ServiceLevel keeps its OPC UA meaning (health and Failover); load is a separate signal, so load never triggers a spurious Failover and a stale load weight only affects tie-breaking.

services.AddOpcUa()
    .AddServer(server => { })
    .AddNodeManager<MyNodeManagerFactory>()
    .UseRedundancyConsistency(RedundancyConsistencyMode.Eventual)   // or Strong
    .AddServerRedundancy(r => r.Mode = RedundancySupport.Hot)
    .AddServerServiceLevel(new LeaderServiceLevelProvider(/* … */)) // real health signal
    .UseServerLoadDirection(options =>
    {
        options.BalancingEndpointUrl = "opc.tcp://ha.example.com:4840"; // the virtual/LB discovery URL
        // options.StalenessWindow = TimeSpan.FromSeconds(15);
        // options.LoadBandSize = 16;             // quantize load to damp oscillation
        // options.HealthSubBandSize = 0;         // 0 = all Healthy is one eligibility tier
    });
// Register an ILoadWeightProvider for load-aware balancing (default is a constant 0 = random spread).

Consistency. The direction inputs follow UseRedundancyConsistency: in Eventual (default) the svc/, load/, and endpoint/ records ride the leaderless CRDT store; in Strong they are linearizable via Raft. The high-churn load/ weight is always coalesced (LoadPublishInterval) so it never issues a write per load tick — a per-write quorum on a rapidly changing value is the overhead to avoid, and a slightly stale load tie-break still picks a healthy peer.

Conformance and client behavior. GetEndpoints (§5.4.2) is defined to describe the server that received the request, so directing to a peer is a deliberate extension; it is kept gated — plain discovery on any other URL returns the local server unchanged, so generic clients and tooling that inspect a specific server are unaffected. A returned EndpointDescription carries the target's own ApplicationUri and certificate, so the CreateSession server-endpoint tamper check (§5.6.2) still holds once the Client connects to the target. The stack's own Client needs no change to follow the redirect (it uses the returned endpoints to connect); point it at the balancing URL and pair it with WithServerRedundancy() so the standard client-driven ServiceLevel selection remains the authoritative complement — it already fails a Client that lands on a non-Healthy server over to the active one and, on a connect failure, retries the normal discovery URL / FindServers/RedundantServerArray path. This is connect-time placement, not continuous balancing — ongoing Failover still relies on the Client monitoring ServiceLevel per §6.6.

Security. The store is treated as an untrusted conduit: every direction and endpoint record is written through the configured IRecordProtector and verified before use, so a forged or tampered record is dropped fail-closed. Configure an authenticated-encryption protector (AesCbcHmacRecordProtector) for any networked backend; the default NullRecordProtector is for single-process/in-memory use only. Because the redirect returns a peer's certificate, Clients must trust every member's certificate (non-transparent redundancy uses per-server certificates). The balancing endpoint is typically unauthenticated discovery, so a rogue replica could bias or deny direction, but it cannot force a Client onto an untrusted server (the Client validates the returned certificate); this is the same rogue-replica trust boundary as the rest of the shared store. Keep the balancing decision damped (load banding + random tie-break + the load coalescing interval) to avoid herding every Client onto a momentary best.

Client-side high availability (replica sets)

OPC 10000-4 §6.6 covers redundant servers and the client behavior for connecting to them. A complementary, beyond-spec capability is a redundant client replica set: two or more client processes that cooperate so that exactly one — the leader — holds the active session and subscriptions while the others stand by (hot, warm, or cold) and take over on leader loss. This is provided by OPCFoundation.NetStandard.Opc.Ua.Client.Redundancy.

flowchart LR
    L["Client 1 (leader)"] <-->|leader election, session secret| KV[("ISharedKeyValueStore<br/>+ IRecordProtector")]
    S["Client 2 (standby)"] <-->|watch, promote| KV
    L -->|active session| Server["OPC UA Server"]
    S -.->|takes over on leader loss| Server

The two client replicas elect a leader and share the leader's session secret through the protected store; on leader loss the standby is promoted and reuses the token to re-activate (or recreates the session and transfers subscriptions).

The simplest way to consume this is AddRedundantClientSession(...) in OPCFoundation.NetStandard.Opc.Ua.Redundancy.Client: register a client shared store and election, then a redundant session, and resolve a single, stable ISession. Failover, leader election, and the underlying session churn are hidden — the application reasons only about ISession, and service calls block until this replica is the leader with a live session (synchronous members throw BadInvalidState until then).

services
    .AddRaftClientSharedStore()                     // ISharedKeyValueStore + ILeaderElection
    .AddRedundantClientSession(options =>
    {
        options.NodeId = replicaId;
        options.Mode = ClientStandbyMode.Hot;
        options.CreateSessionAsync = ct => new ManagedSessionBuilder(configuration, telemetry)
            .UseEndpoint(endpoint)
            .WithServerRedundancy()
            .WithTokenReuseFailover()
            .ConnectAsync(ct);
    });

// A hosted service starts the coordinator; resolve the transparent handle anywhere and use it:
ISession session = provider.GetRequiredService<ISession>();
BrowseResponse response = await session.BrowseAsync(/* … */, ct);   // blocks until this replica leads

Under the hood it reuses the same seams as the server, now shared from Opc.Ua.Core under the Opc.Ua.Redundancy namespace:

• Leader election among the client replicas (ILeaderElection, e.g. SharedStoreLeaseElection), so exactly one replica is the active client at a time.
• Shared session secret. The leader persists its session secrets (AuthenticationToken, nonces) through ISharedKeyValueStore, encrypted and integrity-protected at rest by IRecordProtector. The coordinator is fail-closed: a non-in-memory store with no real protector is rejected. A follower promoted to leader reuses the token to ActivateSession against a HotAndMirrored server, or recreates the session and transfers subscriptions otherwise.
• Standby modes (ClientStandbyMode): Cold (elect only; connect+activate on promotion), Warm (keep a connected session, create subscriptions on promotion), Hot (keep a connected session with subscriptions sampling-only, enable publishing on promotion).

For non-DI callers, RedundantClientSessionBuilder builds the same RedundantClientSession (an ISession) directly. The lower-level ClientReplicaSetBuilder / ClientReplicaCoordinator seam is still available when you want to hold and drive the leader ManagedSession yourself:

var coordinator = new ClientReplicaSetBuilder(telemetry)
    .WithNodeId(replicaId)
    .WithStandbyMode(ClientStandbyMode.Hot)
    .UseRedundancy(election, sharedStore, recordProtector)
    .UseSession(ct => new ManagedSessionBuilder(configuration, telemetry)
        .UseEndpoint(endpoint)
        .WithServerRedundancy()
        .WithTokenReuseFailover()
        .ConnectAsync(ct))
    .ConfigureLeader((session, fastActivated, ct) => ApplySubscriptionsAsync(session, ct))
    .Build();
await coordinator.StartAsync(ct);

The full token-reuse fast-activate from shared secrets and subscription recreate/transfer are exercised by the client replica-set sample and integration tests.

Kubernetes deployment

Use the consolidated Kubernetes High Availability Deployment guide for the Opc.Ua.Redundancy.Kubernetes package. It covers Kubernetes Lease election, EndpointSlice peer discovery, ServiceLevel-driven readiness, StatefulSet/Deployment and Service manifests, RBAC, probes, time synchronization, secrets, and GDS/NTRS registration.

Samples

samples/Redundancy/RedundantServer demonstrates the server-side distributed and redundancy registrations, with docker-compose files for active/active and active/passive replica sets. Its strong active/passive topology historizes a stable Counter and event notifier through UseDistributedHistorian. samples/Redundancy/RedundantClient shows the recommended managed-client pattern: a single ManagedSession with WithServerRedundancy() and token-reuse failover that connects to any server, reads its redundancy metadata, and fails over transparently. Its --history workflow holds raw, event, and processed cursors across active-process loss and verifies post-promotion writes.

samples/Redundancy/RedundantClient/docker-compose.yml runs the client and server in one env-driven compose file that covers the full HA matrix. COMPOSE_PROFILES (defaulted by an adjacent .env) picks one server profile (server-eventual, scalable active/active, or server-strong, a fixed 3-node Raft quorum) and one client profile (client-independent, scalable, or client-coordinated, a fixed 3-node client replica set); the eventual server and independent clients scale with --scale server=N --scale client=M. Both sides log their failover / HA behavior, including data loss: the client turns transparent reconnects and leader handoffs into explicit FAILOVER:, ACTIVE CLIENT:, DATA LOSS:, and HA OK: Counter continued … lines, and the server logs its active/standby role transitions and a per-replica Counter heartbeat. server-eventual shows data loss on failover because each replica owns an independent Counter; server-strong (active/passive Raft) keeps the Counter on a linearizable store and shows no data loss. See the RedundantClient README.

Security considerations

Distributed high-availability deployments protect the shared store as part of the OPC UA trust boundary. Use an authenticated and encrypted channel to the store, protect serialized records with IRecordProtector, provision record-protection keys through a key ring or secret manager, apply TTLs and quotas to session/nonce/subscription keys, and fail closed if a required strongly consistent store is unavailable.

Record context and plaintext ownership

The 2.0 IRecordProtector contract requires a ByteString context on both Protect and TryUnprotect. For stored records, use RecordProtectionContext.Create(recordType, fullStoreKey): strict UTF-8 encoding of recordType + "|" + fullStoreKey. Type labels cannot contain |; keys retain their exact case, escaping and configured prefixes. Readers use the actual key returned by scans and change feeds, not just a type label or a reconstructed identifier. This also binds snapshot chunks to their generation and ordinal, and historical segments to their individual keys, independently of the storage backend.

ByteString binding = RecordProtectionContext.Create("application-record", key);
ByteString envelope = protector.Protect(binding, plaintext);
await store.SetAsync(key, envelope, cancellationToken).ConfigureAwait(false);

(bool found, ByteString stored) = await store.TryGetAsync(key, cancellationToken).ConfigureAwait(false);
if (!found || !protector.TryUnprotect(binding, stored, out ByteString recovered))
{
    throw new ServiceResultException(StatusCodes.BadSecurityChecksFailed);
}

Protect separately when the same plaintext belongs under two different keys; do not copy a ciphertext between a current-record pointer and a generation record. A wrong key, type, context or modified envelope fails closed. Never retry with an empty context after a failed bound read. When binding is genuinely not needed, pass default(ByteString) or ByteString.Empty explicitly; both encode the same empty context. The string convenience extensions only convert to UTF-8 and invoke the required byte-context member once. They do not select a capability or provide a fallback.

AesCbcHmacRecordProtector retains the envelope [version:1][keyId:4 LE][IV:16][ciphertext][HMAC-SHA256:32]. Its MAC authenticates [contextLength:4 LE][context][header][ciphertext], including a zero context length for an explicitly unbound record, and is checked before decryption. KeyRingRecordProtector supplies the same context to active and retired keys.

Private-key consumers require IOwnedRecordProtector.TryUnprotectOwned(context, envelope, out byte[] plaintext). This transfers the original decrypted array, independent of the store input, so the caller can wipe it with CryptoUtils.ZeroMemory in a finally block. Do not implement this by calling TryUnprotect(...).ToArray(): that leaves an unwiped decrypted copy. SharedKeyValuePendingCertificateKeyStore rejects protectors without this ownership contract. Owned key-ring reads skip members that cannot supply owned buffers, without invoking their immutable-plaintext read path. If no capable member authenticates a record, the ring returns false and an empty buffer; pending-key stores treat that record as absent rather than throwing. Custom KMS/HSM providers must implement the owned contract to read pending keys protected by those members, including records retained during key rotation. NullRecordProtector remains a non-authenticating, process-local test/demo option, not a substitute for protection of a networked shared store.

Shared application identity

Transparent redundancy uses one logical application identity. Every replica that serves the transparent virtual endpoint must use the same endpoint URL, ApplicationUri, application instance certificate, and private key so clients can validate one server identity and complete ActivateSession against any replica. The compromise blast radius of that shared private key is the entire transparent set: an attacker can impersonate the virtual server, terminate or redirect client trust, and participate in failover until the certificate is revoked and trust lists are updated. Prefer non-transparent redundancy with per-replica certificates when that shared-key risk is unacceptable.

The samples/Redundancy/RedundantServer sample includes a worked transparent deployment (Transparent/docker-compose.yml): two REDUNDANCY_MODE=transparent replicas that share one ApplicationUri and one certificate and mirror session + address-space state by CRDT gossip, behind an nginx TCP load balancer that publishes a single virtual endpoint. Each replica advertises the load-balancer host, so discovery and CreateSession echo the one endpoint the client uses, and a mirrored session resumes on the survivor when a replica fails. The sample seeds the shared certificate into a volume for convenience; a secured deployment provisions it from a secret to every replica (see docs/Kubernetes.md).

Rotate the transparent shared application certificate/key as a coordinated replica-set operation. Issue the replacement certificate with the same virtual identity and endpoint subject alternative names, distribute the new private key through the certificate store or a secret manager without baking it into images, stage client and GDS trust-list updates so both old and new certificates are trusted during the overlap, restart or hot-reload replicas in a controlled window, verify every replica presents the new certificate at the virtual endpoint, then revoke/remove the old certificate and securely delete the old private key from all nodes and secret versions. Treat an emergency rotation after suspected compromise as a fail-closed event: drain or stop replicas that still hold the old key before restoring client access.

On the client side, a client replica set shares the same trust boundary. The leader persists its session secret (AuthenticationToken, nonces) through ISharedKeyValueStore, so it must be encrypted and integrity-protected at rest by IRecordProtector; the ClientReplicaCoordinator is fail-closed and rejects a non-in-memory store that has no real protector. Because non-transparent redundancy and GetEndpoints load direction return per-server certificates, a failing-over or directed client must trust every member's certificate, and a promoted standby re-validates the server endpoint and certificate on reconnect. Treat the client shared store, like the server's, as an untrusted conduit: authenticate and encrypt the channel to it and scope its keys with TTLs and quotas.