# Description Follow-up to #4390 for distributed address-space ownership, authoritative hydration, and replica-consistent wire NodeIds. Tracks #4444. Identity implementation: `a78d59067` (Server policy/lifecycle/adapter seam) and `07faf5cfd` (replica identity module, runtime wiring, examples and proofs). ## Replica identity - `UseReplicaNodeIdentity` / `ReplicaNodeIdFactory` reserve fixed shared namespace indexes before node-manager construction and reuse `DefaultNodeIdFactory`. ApplicationUri and built-in diagnostics/configuration remain local. - Replicated startup requires an explicit identity contract. Namespace layout, deterministic mode, replica-set identity and writer policy must agree. Standard factory views, replacement and registration preserve the policy. - Active/passive writer-assigned identities are replicated unchanged. Independent creation requires stable keys or explicit IDs. Live IDs and retained tombstones are reserved before writer allocation; no network/storage call runs in `New`. - Protected store descriptors use strong routing, including customized hybrid prefixes. Unknown legacy state and empty eventual scans cannot authorize automatic adoption. Verified-new hybrid provisioning is explicit. - Active/active frames compare the descriptor before CRDT merge. Rejected frames are not reported as applied; incompatible admission remains at ServiceLevel 0 through the standard service-level wiring. - Shared metadata validation covers type definitions, data types, references, permissions, NodeId/ExpandedNodeId/QualifiedName values, arrays/matrices, DataValues and encodeable structures/arguments, without reflection. - Runtime preparation hydrates retained IDs before replacement routes publish. Rebinding keeps the existing store/election or gossip transport/map and attaches to the new node-manager sources. Normal node-manager mutations, including AddNodes, reach the cached local-address-space adapter. - WoT materialization retains the stable configured factory. Samples expose genuinely factory-created nodes and a client `--identity` failover workflow. ## Preserved HA repairs The published `d99995003` and `79da74f14` fixes remain in place: - Shared linearizable sequence/lease coordination; no pretend CAS on CRDT and no private counter substituted for shared coordination. - Strict authoritative record decoding, protected pending reservations, safe snapshot publication and required delta retention. - Accepted topology state controls authoritative membership; stale deletion cannot remove a newer incarnation. - Independent subtree attachment tracking and correct disposal of the final snapshot task. - Values wait for their topology, and parent tombstone horizons prevent older descendant values crossing incarnations. A new deterministic regression also covers writer hydration preparation: temporary reconciliation removals must remain inbound application, not be recaptured as new shared deletion records. ## Validation ### Review feedback fixes `3a682155b` addresses the active-node notification, WoT namespace-index bounds, and retired-manager XML documentation feedback. The address-space adapter now notifies listeners with the registered active node, and both WoT managers reject missing/out-of-range namespace indexes before ownership is established. Targeted feedback validation passes on net10.0 and net48: 8 address-space adapter cases and 74 WoT cases per framework, plus 17 hydration/gossip/live-client integration cases on net10.0. The wrong-instance notification and missing namespace regressions were reproduced before the fixes. ### Identity implementation evidence | Selected scope | net10.0 | net48 | | --- | --- | --- | | HA repair, identity, metadata, peer admission and real-client identity/reload fixtures | 215 passed | 215 passed | | Factory, fluent authoring/import, NodeManagement and address-space adapter fixtures | 213 passed | 213 passed | | WoT node-manager fixtures | 68 passed | 68 passed | - Real UA-TCP clients compare full NodeIds/reference targets across independent servers in all four deterministic modes, with different manager/creation order and local allocations. Saved IDs work after endpoint failover and restart. - Real AddNodes-created IDs remain readable through successive live manager replacements in both replication modes, including nodes created after a reload. - Two process-level cases start three sample replicas, kill the serving process, verify a different serving ServerArray URI, and reuse cached factory-generated IDs for Read and CreateMonitoredItems without remapping or rebrowsing. - The identity/hydration scenario runs from the actual published win-x64 NativeAOT executable in the existing historian companion host. - The focused writer-recapture regression was observed failing without its fix, then passing. The real reload scenario was additionally repeated while diagnosing that race. - Changed-file formatting/style/analyzers were applied. Existing CA1508, CA1823, CA1861 and CA1850 warnings in untouched upstream files were not suppressed. The selected checks above do not replace full-solution CI, coverage gates or maintainer review, and do not claim that the entire solution or every native platform was run. #4444 remains the tracking issue. ## Contract and consistency limits Fixed identity does not make eventual payload reads linearizable. Hybrid views remain non-destructive, use explicit tombstones, and do not infer empty-store bootstrap or partition completion from absence. Compacted snapshots require authoritative storage. Failed/uncertain reservations require reconciliation. There is no automatic namespace-slot allocation, persistent logical-key mapping database, live renumbering, legacy-store wipe/adoption, or new hash algorithm. Opaque shared structures require a registered codec so their namespace-bearing fields can be validated. Application callbacks are still application-owned. ## Related issues - Tracks #4444. - Follow-up to #4390. - Builds on the merged NodeId factory (#4433), configure/seal (#4442) and lifecycle/import (#4427) integration at `aedd3f738`. ## Checklist - [ ] I have signed the CLA and read CONTRIBUTING. - [x] Added targeted regression and integration tests. - [x] Updated directly related API, HA, migration and sample documentation. - [x] Ran scoped formatting and the targeted framework checks listed above. - [ ] Full UA.slnx matrix and coverage gates completed. - [ ] All CI and maintainer review completed. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9dbd9527-d818-47a0-984c-fbb1472d3d6b
31 KiB
High Availability Server
This sample is a minimal Generic Host based OPC UA server that demonstrates the distributed high-availability server building blocks in Opc.Ua.Redundancy.Server and the active/active building blocks in Opc.Ua.Redundancy.Server. It registers an AsyncCustomNodeManager-derived node manager so the local address space participates in replication, and selects its topology with the HA_MODE environment variable: active/passive (ap, the default — leader election so only the active replica writes) or active/active (aa — every replica writes and converges by CRDT gossip). It drives Server.ServiceLevel from the replica state and publishes OPC 10000-4 §6.6 Server.ServerRedundancy metadata, including non-transparent discovery and manual failover.
The bundled configuration uses the default in-memory shared key/value store. That is useful for understanding the DI wiring and for single-process experimentation, but separate OS processes do not share memory. A real multi-process or multi-host deployment needs a shared backend for ISharedKeyValueStore such as Redis; that backend is intentionally deferred from this small sample.
Run one instance
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
Connect an OPC UA client to opc.tcp://localhost:62543/RedundantServer and browse to Objects/High Availability. The Counter variable is writable and is also incremented once per second while this server is leader. ActiveReplica shows the local active writer label.
Both HA modes call UseReplicaNodeIdentity before startup. The sample's shared
namespace occupies index 2 on every replica, while ApplicationUri and built-in
diagnostics remain local. FactoryAssigned/Value and FactoryAssigned/Target
are genuinely factory-created nodes, not preassigned constants; Target contains
the exact NodeId of Value. Use the client's --identity scenario to cache
their Browse results and reuse those same IDs after terminating the serving
replica. See Replica-consistent NodeIds.
The startup output shows the effective OPC UA redundancy value and the ServiceLevel subrange that clients read. For example, with HA_NODE_ID=replica-a:
HA sample node 'replica-a' listening at opc.tcp://localhost:62543/RedundantServer; HA_MODE=ap; REDUNDANCY_MODE=Hot; ServiceLevel=255 (Healthy).
CurrentServerId: replica-a
Redundant peers: (none)
NTRS discovery capability and FindServers peer-set provider are enabled.
RequestServerStateChange is enabled for administrator-driven Maintenance/NoData failover.
Run two instances
Use distinct ports and stable HA_NODE_ID values. REDUNDANCY_MODE selects the OPC UA redundancy model. HA_REDUNDANT_PEERS provides the peer set used for ServerUriArray, RedundantServerArray, NTRS discovery registration, and FindServers peer results. Each peer entry is:
applicationUri|applicationName|discoveryUrl1+discoveryUrl2
$env:HA_NODE_ID = "replica-a"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-b|RedundantServer replica-b|opc.tcp://localhost:62544/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
In a second terminal:
$env:HA_NODE_ID = "replica-b"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-a|RedundantServer replica-a|opc.tcp://localhost:62543/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544
With the default in-memory store, each process elects against its own private store, so this two-terminal setup demonstrates endpoint, node id, service-level, and redundancy metadata configuration rather than cross-process state transfer. To turn it into a real HA pair, register a shared ISharedKeyValueStore that both processes can reach, as shown below.
The second replica prints the configured redundancy value and the peer set:
HA sample node 'replica-b' listening at opc.tcp://localhost:62544/RedundantServer; HA_MODE=ap; REDUNDANCY_MODE=Hot; ServiceLevel=255 (Healthy).
CurrentServerId: replica-b
Redundant peers: urn:localhost:OPCFoundation:RedundantServer:replica-a [opc.tcp://localhost:62543/RedundantServer]
NTRS discovery capability and FindServers peer-set provider are enabled.
RequestServerStateChange is enabled for administrator-driven Maintenance/NoData failover.
Wire up a shared store for real HA
The single-instance defaults stay in effect until you supply a shared backend. The DI wiring below is what makes additions, removals, references, and values replicate across replicas. Replace the placeholder RedisSharedKeyValueStore with any ISharedKeyValueStore reachable by every replica (a Redis adapter is intentionally deferred from this sample):
builder.Services
.AddOpcUa()
.AddServer(o =>
{
o.ApplicationName = "RedundantServer";
o.ApplicationUri = applicationUri; // unique per replica
o.EndpointUrls.Add(endpointUrl);
})
.AddNodeManager<HaSampleNodeManagerFactory>()
.UseReplicaNodeIdentity(
"opcfoundation-ha-sample",
[HaSampleNodeManagerFactory.NamespaceUri],
writerAssignedIds: true)
.UseDistributedAddressSpace(d =>
{
// Shared backend reachable by every replica. The default is an
// in-process InMemorySharedKeyValueStore that is NOT shared across
// processes, so a real deployment must override it.
d.KeyValueStoreFactory = sp => new RedisSharedKeyValueStore(redisConnectionString);
// Authenticated encryption + integrity protection for every record at
// rest. The 32-byte master key comes from a Kubernetes Secret / KMS and
// is shared by all replicas.
d.RecordProtectorFactory = _ => new AesCbcHmacRecordProtector(recordKey);
d.UseLeaderElection = true; // lease-based leader = the single writer
d.NodeId = nodeId; // unique per replica
d.RedundancyMode = RedundancySupport.HotAndMirrored;
})
.UseDistributedSessions(s =>
{
// Mirror encrypted session state so a client can fail over to a standby
// and reconnect with a full ActivateSession (the authentication token is
// only a lookup key). The default is the safe re-auth-on-failover.
s.EnableFastReconnect = true;
})
.AddServerRedundancy(r =>
{
r.Mode = RedundancySupport.HotAndMirrored;
r.CurrentServerId = nodeId;
r.RedundantPeers.Add(new RedundantPeer(
"urn:host-b:OPCFoundation:RedundantServer:replica-b",
new ArrayOf<string>("opc.tcp://host-b:62543/RedundantServer"))
{
ApplicationName = new LocalizedText("RedundantServer replica-b")
});
})
.AddRequestServerStateChange();
Strong consistency with Raft (no external store)
The Redis placeholder above is one way to get a shared ISharedKeyValueStore. This sample also ships an in-package alternative: a strongly-consistent Raft cluster. Set HA_CONSISTENCY=strong and the active/passive path registers UseRedundancyConsistency(RedundancyConsistencyMode.Strong) backed by a multi-node RaftCs cluster over NanoMsg — the shared store, leader election, and single-use session nonce become linearizable and shared across every replica, with no Redis or other external dependency. Configure the cluster from the environment:
| Setting | Example | Description |
|---|---|---|
HA_CONSISTENCY |
strong, eventual |
strong backs the shared store with a Raft cluster; eventual (default) keeps today's behaviour. |
HA_HISTORIAN |
true, false |
Enables the protected shared historian. It is valid only with HA_MODE=ap and HA_CONSISTENCY=strong; strong active/passive defaults to enabled. |
HA_RAFT_ID |
1 |
This replica's unique Raft node id (1..N). |
HA_RAFT_MEMBERS |
3 |
Static cluster size (odd 3/5 for a fault-tolerant quorum). |
HA_RAFT_BIND |
tcp://0.0.0.0:6560 |
Local Raft transport bind address. |
HA_RAFT_PEERS |
tcp://server-b:6560,tcp://server-c:6560 |
The other members' Raft transport addresses. |
See docker-compose.yml with the active-passive.env env file for a runnable 3-node Raft cluster (real cross-container active/passive HA). Set HA_RECORD_KEY to one shared base64 32-byte key before starting it. The topology calls UseDistributedHistorian, historizes HighAvailability.Counter and HighAvailability.HistoryEvents, and keeps history writes plus raw, event, and processed HistoryRead continuation points visible across session takeover. The historian deliberately rejects HA_INSECURE, eventual storage, and active/active multi-writer topology because ordered values, deletes, modification chains, events, and atomic batches have no conflict-free merge contract.
For Kubernetes, UseKubernetesRaftConsensus in Opc.Ua.Redundancy.Kubernetes derives the same wiring from the StatefulSet ordinal and headless-Service DNS, with a file WAL on a PersistentVolume; see docs/Kubernetes.md.
Redundancy mode and discovery settings
| Setting | Values | Description |
|---|---|---|
REDUNDANCY_MODE |
none, cold, warm, hot, hotandmirrored, transparent |
Selects the Server.ServerRedundancy.RedundancySupport value. If unset, active/passive defaults to hot and active/active defaults to hotandmirrored. |
HA_NODE_ID |
stable replica id | Used as this replica's ApplicationUri suffix (unless HA_APPLICATION_URI overrides it) and as CurrentServerId for transparent redundancy. |
HA_REPLICA_SET |
stable shared set id | Defaults to opcfoundation-ha-sample. Must be identical on every replica; it is not the per-process HA_NODE_ID. |
HA_REDUNDANT_PEERS |
`applicationUri | applicationName |
peerServerUris |
comma/semicolon-separated application URIs | Legacy shorthand for RedundantServerArray; prefer HA_REDUNDANT_PEERS when clients must resolve peers through FindServers. |
HA_MODE |
ap, aa |
Chooses active/passive shared-store replication or active/active CRDT gossip. |
HA_PEER_DISCOVERY |
static, dns, lds, k8s |
Peer-discovery mechanism for the client-facing RedundantServerSet (FindServers) and, for active/active, the gossip fabric. static (default) uses the configured peers; dns resolves HA_SERVICE_NAME (one record per replica) and updates both dynamically. lds/k8s are wired via UseLdsPeerDiscovery / UseKubernetesPeerDiscovery. Static configuration is the fallback until discovery finds peers. |
HA_SERVICE_NAME |
a DNS name | The headless-service DNS name resolved by HA_PEER_DISCOVERY=dns (defaults to server). |
HA_CONSISTENCY |
strong, eventual |
Selects the shared-store consistency model; strong backs it with a Raft cluster (see Strong consistency with Raft). |
HA_HISTORIAN |
true, false |
Enables the shared historian and durable HistoryRead continuation store. Supported only by strong active/passive topology; defaults to true there and false elsewhere. |
HA_FAST_RECONNECT |
true, false |
Allows token-reuse reconnect for mirrored sessions. The default requires full ActivateSession re-authentication after failover. |
HA_RECORD_KEY |
base64 32-byte key | Shared record-protection key for the distributed topologies. When set, every mirrored record (session secrets, identity tokens, notifications) is encrypted + integrity-protected at rest with AesCbcHmacRecordProtector. Use the same value on every replica; provision from a Kubernetes Secret / KMS in production. |
HA_INSECURE |
true, false |
Explicit, auditable opt-out that runs an isolated demo without record protection or gossip authentication (prints a warning). Secure by default: without HA_RECORD_KEY and without this flag, a distributed topology fails closed at startup. Never set in production. |
HA_BALANCING_URL |
a discovery URL | Enables GetEndpoints load direction (see below). A GetEndpoints request on this virtual/LB discovery URL is answered with the best replica's endpoints; empty (default) disables it. |
HA_APPLICATION_URI |
a URI | Overrides the per-replica ApplicationUri with a shared one. Required for transparent redundancy so every replica presents one logical server identity (CreateSession validates the client serverUri against it). |
HA_SUBJECT_NAME |
a certificate subject | Sets an explicit, stable certificate subject so replicas sharing a PKI store load one ApplicationInstanceCertificate. |
HA_PKI_ROOT |
a filesystem path | Points the certificate stores at a shared directory. Combined with HA_SUBJECT_NAME, replicas share one ApplicationInstanceCertificate (production provisions this from a secret rather than a shared volume). |
The sample also enables RequestServerStateChange with AddRequestServerStateChange(). An administrator client can call the standard method on Server to request Maintenance or NoData behavior and set Server.EstimatedReturnTime; the server updates Server.ServiceLevel into the appropriate OPC UA subrange so clients back off or fail over.
Mode-specific nodes shown by the sample:
none— leaves redundancy metadata in its single-server defaults.cold,warm,hot,hotandmirrored— publishServerUriArrayfromHA_REDUNDANT_PEERS, retainRedundantServerArray, advertiseNTRS, and return the peerApplicationDescriptionvalues fromFindServers.transparent— publishesCurrentServerIdandRedundantServerArrayfor the transparent set.
Server.ServiceLevel is driven by the sub-range-aware provider: leaders report Healthy, warm standby reports Degraded, hot/hot-and-mirrored replicas report Healthy, and cold standby reports NoData. The console prints the selected mode, server id, peer set, and initial service-level subrange at startup.
GetEndpoints load direction
Setting HA_BALANCING_URL registers UseServerLoadDirection(...): every replica publishes its health ServiceLevel, a load weight, and its endpoints to the shared store, and a GetEndpoints request that arrives on the balancing URL is answered with the best replica's endpoints — the active writer in active/passive, or the least-loaded healthy replica in active/active. Plain discovery on a replica's own URL is unaffected, and this complements (never replaces) the standard client-driven Server.ServiceLevel / RedundantServerArray selection. In a real deployment a load balancer / Kubernetes Service fronts the replicas at the balancing URL; the sample sets StrongEligibility when HA_CONSISTENCY=strong so the eligibility keyspaces are linearizable. It requires a shared store across replicas (use HA_CONSISTENCY=strong / the Raft compose). See docs/HighAvailability.md for the design, conformance, and security notes.
Transparent redundancy (single virtual endpoint)
The load direction above is one endpoint model; the other is transparent redundancy, where every replica presents one logical server behind one virtual endpoint. Unlike the non-transparent modes (the client reads RedundantServerArray and selects a replica), a transparent client sees a single endpoint and never chooses a replica — a load balancer routes it and mirrored session state lets it continue across a replica failure.
To present one logical server, all replicas must share:
- One
ApplicationUri(HA_APPLICATION_URI).CreateSessionvalidates the client-suppliedserverUriagainst the server'sApplicationUri, so replicas that disagree would reject sessions established via discovery on a peer. - One
ApplicationInstanceCertificate(HA_SUBJECT_NAME+HA_PKI_ROOTpointing at a shared store). Under SecurityMode None the certificate is not exchanged, but secured deployments must present the same certificate from every replica. - Mirrored session and address-space state (
HA_MODE=aa,REDUNDANCY_MODE=transparent) so a session created on one replica can be resumed on another.
Each replica advertises the virtual endpoint URL (HA_HOST = the load-balancer host). Because that host is a DNS name, the listener binds to all interfaces in the replica's own container while returning opc.tcp://<lb>:62543/... to clients, so discovery and CreateSession echo the single endpoint the client actually uses (ServerBase.FilterByEndpointUrl matches the client host to the advertised base address). A client therefore connects to one URL, and on a replica failure the load balancer routes the reconnect to the survivor, where the mirrored session resumes with a token-reuse reconnect (the full ActivateSession signature re-check still applies).
Transparent/docker-compose.yml runs this end to end: two REDUNDANCY_MODE=transparent replicas sharing one ApplicationUri and one certificate (seeded into a shared PKI volume by the first replica, then reused by the second) behind an nginx TCP load balancer that publishes the single virtual endpoint opc.tcp://localhost:62543/RedundantServer, plus a client that connects only to that endpoint. In production the shared certificate is provisioned from a Kubernetes Secret / KMS to every replica rather than self-generated (see docs/Kubernetes.md), which also removes the first-start certificate race.
Active/passive vs active/active
The sample selects its redundancy topology with the HA_MODE environment variable — ap (active/passive, the default) or aa (active/active) — and references both Opc.Ua.Redundancy.Server and Opc.Ua.Redundancy.Server:
- Active/passive (
HA_MODE=ap) —UseDistributedAddressSpace+UseDistributedSessionswith leader election. WithHA_CONSISTENCY=strong,UseDistributedHistorianadds one protected, leader-write archive with writer epochs and durable portable HistoryRead continuations. One replica is the active writer; standbys hydrate from the shared store and take over on failover. Redundancy is reported asRedundancySupport.Hot, and clients followServer.ServiceLevel/Server.ServerRedundancyto find the active replica. - Active/active (
HA_MODE=aa) —UseActiveActiveRedundancy(which wiresUseReplicatedAddressSpace+UseReplicatedSessionsfrom one set of gossip options; the individual methods remain for advanced setups). Every replica accepts writes and converges without a leader; redundancy is reported asRedundancySupport.HotAndMirrored. Replicas gossip over TCP: setHA_GOSSIP_PORT(default4840; session entries gossip onport + 1) andHA_GOSSIP_PEERS(a comma/semicolon list ofhost:portfor the other replicas' address-space gossip). A session created on one replica can be resumed on another withManagedSessionBuilder.WithTokenReuseFailover()on the client. Distributed Historical Access is not supported in this multi-writer mode and fails closed when requested.
Set HA_PEER_DISCOVERY=dns (with HA_SERVICE_NAME) to discover peers dynamically instead of a static HA_GOSSIP_PEERS list: the sample registers UseDnsPeerDiscovery, which keeps the client-facing RedundantServerSet (FindServers) and the active/active gossip fabric current as replicas scale up and down. Static configuration (AddServerRedundancy / HA_REDUNDANT_PEERS) is the fallback. HA_PEER_DISCOVERY=lds (UseLdsPeerDiscovery) and Kubernetes EndpointSlice discovery (UseKubernetesPeerDiscovery) are also available — see docs/HighAvailability.md → Dynamic peer discovery.
Run two active/passive replicas
Use HA_MODE=ap (or omit it) for the leader-elected active/passive setup. A real pair needs a shared ISharedKeyValueStore; with the default in-memory store this remains a wiring demonstration.
# replica A
$env:HA_NODE_ID = "replica-a"
$env:HA_MODE = "ap"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-b|RedundantServer replica-b|opc.tcp://localhost:62544/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
In a second terminal:
# replica B
$env:HA_NODE_ID = "replica-b"
$env:HA_MODE = "ap"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-a|RedundantServer replica-a|opc.tcp://localhost:62543/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544
The active replica writes Counter, the standby mirrors state through the shared store, and clients use Server.ServiceLevel, Server.ServerRedundancy.RedundancySupport=Hot, and FindServers peer discovery to select or fail over to the active endpoint.
Run two active/active replicas
# replica A
$env:HA_NODE_ID = "replica-a"
$env:HA_MODE = "aa"
$env:HA_GOSSIP_PORT = "4840"
$env:HA_GOSSIP_PEERS = "127.0.0.1:4842"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
In a second terminal:
# replica B
$env:HA_NODE_ID = "replica-b"
$env:HA_MODE = "aa"
$env:HA_GOSSIP_PORT = "4842"
$env:HA_GOSSIP_PEERS = "127.0.0.1:4840"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544
Both replicas now accept writes to Counter and converge: a write on either endpoint propagates to the other by gossip, and the per-second increment runs on every replica, with CRDT last-writer-wins resolving the concurrent updates. Unlike the active/passive store-backed setup, active/active needs no shared ISharedKeyValueStore between processes — the gossip transport carries the state.
Docker Compose
One compose file — docker-compose.yml — runs the sample as a 3-replica set; the topology is chosen from the environment (the build context is the repository root). Two extra files cover the transparent and load-direction demos.
# Active/active, eventual consistency (CRDT gossip, no shared store) — the default.
docker compose --env-file samples\Redundancy\RedundantServer\active-active-eventual.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml up --build
# Active/active, strong consistency (3-node RaftCs, shared linearizable store).
docker compose --env-file samples\Redundancy\RedundantServer\active-active-strong.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml up --build
# Active/passive, strong consistency (3-node RaftCs; one active writer, two hot standbys).
docker compose --env-file samples\Redundancy\RedundantServer\active-passive.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml up --build
# GetEndpoints load direction over the Raft cluster (sets HA_BALANCING_URL on every replica).
docker compose -f samples\Redundancy\RedundantServer\LoadDirection\docker-compose.yml up --build
# Transparent redundancy: replicas as ONE logical server behind an nginx load balancer
# on a single virtual endpoint (opc.tcp://localhost:62543/RedundantServer).
docker compose -f samples\Redundancy\RedundantServer\Transparent\docker-compose.yml up --build
Each replica exposes its OPC UA endpoint on the host (opc.tcp://localhost:62543/RedundantServer, :62544, :62545). Set HA_HOST to the reachable hostname (the compose file uses the container/service name) so peers and clients connect across the container network. Three replicas run in every topology (Raft needs a 3-node quorum; active/active eventual works with 2 or 3 — remove server-c for a 2-node run, so the replica count is configurable). The Raft topologies are real cross-container HA deployments whose RaftCs cluster is the shared, linearizable store; active/active eventual converges leaderlessly over CRDT gossip with no shared store.
Scale active/active eventual replicas
Scale/docker-compose.yml has a single server service for active/active eventual (CRDT gossip) runs. Docker Compose can scale it to any replica count because the sample discovers peers from the server DNS alias instead of a fixed HA_GOSSIP_PEERS list:
docker compose -f samples\Redundancy\RedundantServer\Scale\docker-compose.yml up --build --scale server=5
The scale file sets HA_PEER_DISCOVERY=dns and HA_SERVICE_NAME=server; every replica resolves the service alias, removes its own container IP, and seeds gossip with the remaining task IPs on HA_GOSSIP_PORT (session gossip uses the next port). No host ports are published, so clients should run inside the same compose network or attach to it. This dynamic scaling path is gossip-only. Dynamic Raft scaling is intentionally unsupported because Raft needs stable replica identities and an odd quorum; use the Kubernetes StatefulSet deployment with UseKubernetesRaftConsensus for strong consistency.
The same file has a clients profile that runs independent managed clients in their own containers — each opts into WithServerRedundancy() and fails over on its own, so it scales to any count alongside the servers (no shared client store needed):
docker compose -f samples\Redundancy\RedundantServer\Scale\docker-compose.yml --profile clients up --build --scale server=3 --scale client=2
For a coordinated single-active client replica set (exactly one client active, the rest hot/warm/cold standbys), use AddRedundantClientSession + a CAS-capable AddRaftClientSharedStore (a fixed Raft quorum, like the server's active/passive) rather than independent clients — see HighAvailability.md. Each client replica is its own process; there is no in-process replica set.
For a client-centric compose that runs the full HA matrix (server consistency × client mode) from one env-driven file and logs failover / HA / data loss on both sides, use samples/Redundancy/RedundantClient/docker-compose.yml: COMPOSE_PROFILES selects server-eventual (shows data loss) or server-strong (Raft, no data loss) crossed with client-independent or client-coordinated. See the RedundantClient README.
Watch replication and failover
The clients (and demo) profile adds one or more RedundantClient instances — one process per replica; set CLIENT_REPLICAS to scale the number of client containers. To see replication and failover end to end:
# 1. Start the server set (active/passive here) plus a client that prints the replicated Counter.
docker compose --env-file samples\Redundancy\RedundantServer\active-passive.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml --profile clients up --build
# 2. In another shell, stop the active/leader replica and watch the client keep going:
docker compose -f samples\Redundancy\RedundantServer\docker-compose.yml stop server-a
The client logs its reconnect/redirect to a surviving replica across the failover. Bring the replica back with docker compose ... start server-a; the Raft cluster re-admits it. Set CLIENT_REPLICAS=3 to run three independent client processes, each of which fails over on its own (for a coordinated single-active client set, use AddRedundantClientSession over a Raft client store as above). On the strong-consistency Raft topology the HighAvailability.Counter continues across the failover: the sample wires it through the distributed value cache (see Sharing values across replicas), so the promoted replica resumes from the last value the former leader shared.
For the broader design, see HighAvailability.md. For an environment-driven replica-set deployment, see Kubernetes.md.
Sharing values across replicas
OPC 10000-4 §6.6 requires identical NodeIds and address spaces across a redundant set but does not standardize how live process values are shared. Opc.Ua.Redundancy.Server adds an opt-in extension for this: a variable's read/write callbacks can participate in a distributed value cache (IDistributedValueCache) so the last value is cached in the shared store and served — within a freshness bound — from any replica.
UseDistributedAddressSpace registers IDistributedValueCache in dependency injection (backed by the distributed node-state store; a consumer that constructs the store directly can build a DistributedValueCache itself). The sample injects it into HaSampleNodeManager and wires the Counter:
// Opt the Counter's read/write callbacks into the distributed value cache.
counter.EnableDistributedValueParticipation(
valueCache,
maxAge: TimeSpan.FromSeconds(10),
liveRead: _ => new ValueTask<DataValue>(ReadLocalCounter()));
// The active replica writes each new value through to the shared store...
await valueCache.CacheAsync(counter.NodeId,
new DataValue(Variant.From(value), StatusCodes.Good, DateTimeUtc.Now));
// ...and a replica that has just been promoted seeds from the last shared value,
// so the Counter continues instead of restarting.
(_, DataValue cached) = await valueCache.TryGetAsync(counter.NodeId, maxAge);
On the strong-consistency (Raft) topology this makes the Counter genuinely shared: standby replicas serve the active replica's value, and after a failover the promoted replica resumes from the last value the former leader wrote (verified end-to-end). Monitored items keep reading through the normal pipeline, so a client that monitors a standby observes the shared value only through the participating read path — consistent with OPC UA monitored-item semantics.
Value sharing uses the shared store, which is protected: supply HA_RECORD_KEY (secure by default) or set HA_INSECURE=true for an isolated non-historian demo — see the settings table above. The distributed historian and its portable continuation records always require HA_RECORD_KEY; they never accept the no-op protector.
Notes
- Active/active eventual convergence over TCP/UDP gossip. State propagation between active/active replicas requires each CRDT snapshot to be sent as a
Crdt.TransportFrameCodecframe (MessageType.State); the TCP/UDP gossip transportsDecodeon send and would otherwise throwSystem.IO.InvalidDataException: Frame length does not match the encoded body length. The stack'sFramingGossipTransportdecorator handles this framing forUseReplicatedAddressSpace/UseReplicatedSessions, so cross-processCounterand session state converge over real gossip (regression-tested with two realTcpGossipTransports). The in-memory transport is a transparent passthrough, so single-process runs behaved correctly regardless. - Value continuity across failover. The
Counteris shared and continues across failover through the distributed value cache: on the strong-consistency (Raft) topology it rides the linearizable Raft store, and on active/active eventual it rides the address-space CRDT gossip. Monitored items read through the normal pipeline, so a monitored item on a standby observes the shared value only through the participating read path (as documented inDistributedValueParticipation).