Files
UA-.NETStandard/samples/Redundancy/RedundantServer
romanettandClaude Opus 5.5 a0d569d756 Fix Server and GDS audit findings, review findings and spec-conformance gaps (#4571)
## Summary

Fixes the findings of a bug audit of `Opc.Ua.Server`,
`Opc.Ua.Gds.Server` and `Opc.Ua.Gds.Client`. Rebased on master after
#4516.

What went in:
- **Audit:** 93 findings, each checked against the OPC UA specification.
- **Code review of those fixes:** 35 further findings.
- **Spec grounding of every fix commit:** 18 conformance gaps, now
closed.

Commits are one per finding, named `Fix <ID>`, `Fix review <ID>` or `Fix
spec gap <name>`. The IDs refer to the audit report (S = Server areas,
G1 = GDS server, G2 = GDS client).

Validation:
- Each finding was traced in code and checked against OPC 10000 (parts
3, 4, 5, 9, 11, 12, 13, 18, 20).
- Every fix commit was checked against the spec text again afterwards: 0
non-conformant, remaining gaps fixed.
- Regression tests were added for most fixes.
- Suites run on net10.0 before the rebase: Server, GDS, History,
Configuration, Features, Core.Security, DI, Redundancy, LDS,
InformationModel, PubSub.Server, Client, Sessions and the three
Subscriptions suites, all green. The same suites on the rebased branch
are still running; results will be posted as a comment.

## Behaviour and user-facing changes

### Secure defaults
- **`Historize()` / `HistorianBuilder`** grant HistoryRead only. History
writes need an explicit `historyAccessLevel: HistoryRead |
HistoryWrite`.
- **File-system mount:**
- Anonymous sessions can no longer create, delete, move or open files
for writing (`Bad_UserAccessDenied`).
  - UserWritable and UserExecutable now reflect the user.
- `AllowAnonymousWrite` restores the old behaviour; the ReferenceServer
sample turns it on.
- **Default PKI root** for the hosted server, hosted GDS, hosted LDS and
`OpcUaApplicationConfigurationProvider`:
- Moved from the shared temp directory to `{LocalApplicationData}/OPC
Foundation/{ApplicationName}/pki`, and startup fails if no per-user
directory exists.
- When certificate stores of an earlier version still exist under temp,
a warning names both paths. Existing deployments that relied on the
default get a new application certificate (and the hosted GDS a new CA)
unless `PkiRoot` is set to the old path.
- **GDS:**
  - `GdsServerOptions.AutoApprove` defaults to `false`.
- The default KeyCredential store grants no roles unless
`AuthorizeRoles` allows them.
- Certificate requests (StartSigningRequest, StartNewKeyPairRequest,
FinishRequest) need an encrypted channel.
- **Hosted server anonymous logins:**
- Anonymous is rejected (`Bad_IdentityTokenRejected`) when the default
authenticator options disable it.
- A custom `AddIdentityAuthenticator<T>()` on its own no longer disables
anonymous.
- With anonymous disabled and no configured token policies, the endpoint
advertises the authenticators' token types instead of Anonymous.
- **Runtime-added roles:** their properties and methods need
SecurityAdmin over an encrypted channel, like the well-known roles.
- **Disabled users:** users disabled through Part 18 UserManagement can
no longer log in through `RegisterDefaultAuthenticators(IUserDatabase)`.

### GDS server (Part 12)
- **Certificate requests:**
- FinishRequest / ReadRequest with another application's RequestId
returns `Bad_InvalidArgument`.
  - Every Start*Request returns a new RequestId.
- A CSR without the ApplicationUri is rejected
(`Bad_CertificateUriInvalid`).
- An abstract or unsupported certificate type is rejected at Start
(`Bad_InvalidArgument`).
- **Private-key password:** no longer persisted in the JSON applications
database. A password saved by an earlier version is read into memory
once and dropped on the next save.
- **CheckRevocationStatus:** validates offline against the GDS's own
trust lists and CRLs. It returns Good for good GDS-issued certificates
and `Bad_CertificateInvalid` for input that is not a certificate.
- **Certificate-group trust lists:**
- Writes need CertificateAuthorityAdmin; SecurityAdmin and application
admins are read-only.
  - Writes are validated with the Part 4 certificate validation.
- **AuthorizationService:**
- The legacy RequestAccessToken applies `AccessControl` and issues the
token for the session identity.
- StartRequestToken passes the session identity to every identity-aware
provider.
  - Unknown ResourceIds return `Bad_NotFound`.
- Expired token requests and refresh tokens are pruned, with at most 10
000 pending requests (`Bad_TooManyOperations`).
- **KeyCredential:** requests with PublicKey + SecurityPolicyUri are
rejected with `Bad_SecurityPolicyRejected` (secret encryption is not
implemented); a PublicKey without SecurityPolicyUri is rejected with
`Bad_InvalidArgument`.
- **Audit events:** access-token audit events no longer contain the
password or token data.
- **RegisterTickets / UnregisterTickets:** need the RegistrarAdmin role
when called from a session.
- **Managed application nodes:** moved from namespace 0 to their own
namespace (`ManagedApplicationsNamespaceUri`); the NodeId is the
ApplicationUri, and the mandatory child properties exist.
- **Hosted GDS:**
- New `GdsServerOptions.CertificateGroups`
(`GdsCertificateGroupOptions`), bindable from configuration.
  - The mandatory DefaultApplicationGroup is always present.
  - The default CA subject uses the resolved application name.
- **JSON applications database:**
- A corrupt file throws instead of being silently replaced by an empty
database.
- Saves are atomic and write in place on bind mounts; on net8+ builds
they also follow symlinks and keep the Unix file mode.
  - Issued certificates and trust-list links survive a reload.

### GDS client
- **Trust-list download:**
  - Reads until an empty chunk (Part 20 short reads).
- A size-limit rejection of the final end-of-file read, as sent by
released servers, counts as end of file.
- **Default certificate group:** a null group means
DefaultApplicationGroup, and vendor groups in other namespaces are no
longer mistaken for well-known groups.
- **Status codes and errors:**
  - Missing directory proxies report `Bad_ServerNotConnected`.
- Short or wrongly typed method outputs throw `BadUnexpectedError` /
`BadTypeMismatch`.
- **GdsAccessTokenProvider:** no longer caches a failed client task, and
one caller's cancellation no longer fails other callers.

### Trust lists, push and ConfigurationFile (Part 12 / Part 20)
- **AddCertificate:**
  - `IsTrustedCertificate=false` returns `Bad_CertificateInvalid`.
- Added certificates are validated (Part 4), and a certificate revoked
by a current CRL returns `Bad_CertificateRevoked`.
- **CloseAndUpdate:** validates the uploaded list against its own
certificates and CRLs, not the stores it replaces. Validation failures
are no longer recorded in the server's Rejected store.
- **RemoveCertificate:** removing a trusted or issuer CA that other
certificates still need returns `Bad_CertificateChainIncomplete`.
- **Handles:**
  - Several read handles may be open at once.
- Any open while a write handle exists returns `Bad_NotReadable` /
`Bad_NotWritable`.
  - A write open while any handle is open returns `Bad_NotWritable`.
  - Handles of closed sessions are released.
  - Idle handles close after ActivityTimeout (default 60 s).
- **Status codes:**
- Read clamps to the remaining bytes and no longer enforces the size
limit.
  - Oversized Write / AddCertificate returns `Bad_RequestTooLarge`.
  - Unsupported open modes return `Bad_NotSupported`.
- **ConfigurationFile.CloseAndUpdate:**
  - Empty Targets return `Bad_InvalidArgument`.
- The ConfigurationUpdatedAuditEvent is raised with the configuration
owner as source.
- With the new `IApplicationConfigurationFileTargetProvider`, providers
receive the Targets, a bad target returns Uncertain with per-target
results, and session-interrupting updates are applied after
RestartDelayTime.
- **Default namespace permissions:** namespaces registered after the
first permission lookup now get their NamespaceMetadata default
RolePermissions and AccessRestrictions (no stale negative cache).

### Sessions, identity and roles (Part 18)
- **Session timeouts:** a failure closing one timed-out session no
longer stops the session-timeout monitor.
- **ChangePassword:** an empty old password returns
`Bad_IdentityTokenInvalid`.
- **Removing or disabling a user:** closes only that user's
UserName-token sessions.
- **Role configuration changes:** roles granted by an `ImpersonateUser`
mapping are kept.
- **X509Subject rules:** a certificate subject value containing a quote
never matches an X509Subject rule.
- **`ImpersonateUser` callback:** no longer runs under the global
session lock.
- **Session diagnostics:** LocaleIds are kept as the client sent them,
while null LocaleIds are ignored for translation.

### Subscriptions and monitored items (Part 4)
- **Acknowledgements:** a failing acknowledgement returns a per-entry
Bad status instead of failing the whole Publish.
- **`DiscardedMessageCount`:** counts messages dropped from the
retransmission queue and unsent overflow messages.
- **Durable subscriptions:** transferring a restored anonymous
subscription works (no NullReferenceException).
- **Transferred subscriptions:** sampling and permission checks use the
item's current owner session.
- **Access denied:** a monitored item whose user may no longer read the
attribute gets a `Bad_UserAccessDenied` notification instead of going
silent. Non-Value attributes need Browse permission (RolePermissions
needs ReadRolePermissions), and User* attributes are evaluated per
subscriber.
- **ModifyMonitoredItems:** a SamplingInterval of -1 uses the
subscription's publishing interval.
- **AggregateFilter priming:** needs HistoryRead / ReadHistory
permission and is bounded (100 000 values, bounded page size).
- **Resource leak:** event monitored items no longer leak a consumer
task per create/delete cycle.

### Node management services (Part 3 / Part 4)
- **Reference types:** AddNodes, AddReferences, DeleteReferences and
Browse reject ids that are not ReferenceTypes with
`Bad_ReferenceTypeIdInvalid`. AddNodes, AddReferences and
DeleteReferences also reject abstract ReferenceTypes; Browse still
accepts them (e.g. `HierarchicalReferences` with includeSubtypes).
- **Concurrent AddNodes:** duplicate BrowseNames under one parent get
`Bad_BrowseNameDuplicated`.
- **TranslateBrowsePathsToNodeIds:**
- Requires Browse permission on intermediate nodes and final targets
(RolePermissions and AccessRestrictions).
  - A long RelativePath no longer rejects the whole request.
- **SemanticChangeEvent:**
- Raised once per actual change of a semantic property, whether or not
it is monitored.
  - Also raised for properties with the SemanticChange access-level bit.
  - Reports the owning node's TypeDefinition as AffectedType.
- **Localized diagnostics:** operation-level text is translated when
OperationLocalizedText is requested.

### History and aggregates (Part 11 / Part 13)
- **DeleteRawModified:** an unspecified or reversed range returns
`Bad_InvalidTimestampArgument`, and StartTime == EndTime deletes the
value at StartTime for every provider.
- **ReadAtTime fallback:** reads only the values around each requested
time. A time whose bound is not found within the search limit returns
`Bad_BoundNotSupported` instead of failing the request.
- **ReadProcessed:** a ProcessingInterval shorter than one tick returns
`Bad_AggregateInvalidInputs`.
- **Delta / DeltaBounds:** widen to the next larger signed type when the
result does not fit the source type (e.g. a decreasing unsigned counter,
or an Int16 overflow).
- **Range / Range2:** return the source data type.

### Alarms and fluent API (Part 9)
- **Suppression:** an alarm stays suppressed while any suppression group
is active, and re-evaluation without a state change does not move
TransitionTime.
- **`MonitorVariable(variable)`:** sets SourceNode, SourceName and
InputNode to the variable and makes it a ConditionSource (HasCondition
to the alarm, HasEventSource from the object).
- **Fluent input/output arguments:** InputArguments / OutputArguments
added through the fluent API to a registered method are readable.
- **Root notifiers:** a failed root-notifier registration no longer
leaves an orphaned event worker.
- **State-machine handlers:** exceptions are logged at Error.

### File system (Part 20)
- **Delete / MoveOrCopy:**
  - Only act on direct children of the directory.
- A NodeId whose kind does not match the entry returns `Bad_NotFound` (a
file NodeId can no longer delete a directory).
- Copying a directory into its own subtree returns
`Bad_InvalidArgument`.
- **Locking:**
- An open file, or a directory containing one, cannot be deleted or
moved (`Bad_InvalidState`).
  - The check is atomic with opens.
- While a Delete or Move runs, read opens get `Bad_NotReadable` and
write opens `Bad_InvalidState`.
- **Open:**
- Reserved mode bits or EraseExisting without Write return
`Bad_InvalidArgument`.
  - Read+Write returns `Bad_NotSupported`.
  - Read while a writer is open returns `Bad_NotReadable`.
- **Client `FileSystemErrors`:** maps `Bad_NotReadable` to
`IOException`.
- **RuntimeNodeSet:** well-formed NodeSets with a stray `Model` element,
or two models in one file where one requires the other, now load.

### Server core, hosting and lifecycle
- **`SessionEstablishmentQueueLimit` > 0:**
CreateSession/ActivateSession now queue for a permit. The permit is
acquired before the request is registered, and a waiting handshake
releases its worker through the request-parking policy.
- **Node manager lifecycle:**
  - Add, reload and remove no longer wait for parked Publish requests.
- A Migrate reload racing with CreateMonitoredItems no longer blocks
later lifecycle operations.
  - Late items on retired or removed managers get `Bad_NodeIdUnknown`.
- **HistoryUpdate:** a batch mixing data and event details is bounded by
both limits.
- **Server stop/start:** stopping the server disposes the configuration
watcher and certificate subscription; starting recreates them.
- **Monitored-item queue factory:** a factory supplied by the caller or
the DI container is no longer disposed by the server.
- **Audit events:** each certificate validation error raises its own
AuditCertificate event. A failed OpenSecureChannel without a service
result is audited as failed.

## New public API (additive)
- `IQueuedSessionEstablishmentLimiter`
- `StandardServer.OwnsMonitoredItemQueueFactory` (protected virtual)
- `DefaultPkiRoot.Get` (Opc.Ua.Configuration)
- `TrustList.SetCertificateValidation`
- `NamespaceMetadataRegistry.Invalidate`
- `IApplicationConfigurationFileTargetProvider` and
`ApplicationConfigurationUpdatePlan`
- `GdsServerOptions.CertificateGroups` / `GdsCertificateGroupOptions`
- `InMemoryKeyCredentialRequestStore.AuthorizeRoles`
-
`DefaultManagedApplicationsNodeManager.ManagedApplicationsNamespaceUri`
- `AuthorizationServiceManager.RequestAccessTokenAsync(...,
IUserIdentity?, ...)`
- `AllowAnonymousWrite` on the file-system node manager and its factory
- `IFileSystemHost.TryBeginMutation` / `EndMutation`, replacing
`HasOpenHandles`

## Not fixed / known limitations
- **KeyCredential:** secret encryption with the client's PublicKey
(requests needing it are rejected).
- **CSR SAN restriction:** restricting subject / SAN values copied from
a CSR (CA policy).
- **Ticket signatures:** ticket signature verification (needs a
trusted-signer configuration).
- **FileType Read+Write open:** needs a read/write stream contract in
`IFileSystemProvider`.
- **Custom `IUserDatabase` implementations:** the disabled-user check
still scans all users per login. The built-in stores do a single lookup.
- **64-bit Delta precision:** limited by the calculator's double
arithmetic.

## Tests with changed expectations
Existing tests were updated where they asserted the old behaviour:
- File-system status codes
- Trust-list handle eviction
- `DiscardedMessageCount`
- Localized diagnostics bits
- Delta on unsigned sources
- SourceNode of fluent alarms
- ReferenceServer transfer permission test (now expects
`Bad_UserAccessDenied` notifications)
- `UserDeactivationShutdownReviewTests` (the revoked session is a
UserName login)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 07:26:05 +02:00
..

High Availability Server

This sample is a minimal Generic Host based OPC UA server that demonstrates the distributed high-availability server building blocks in Opc.Ua.Redundancy.Server and the active/active building blocks in Opc.Ua.Redundancy.Server. It registers an AsyncCustomNodeManager-derived node manager so the local address space participates in replication, and selects its topology with the HA_MODE environment variable: active/passive (ap, the default — leader election so only the active replica writes) or active/active (aa — every replica writes and converges by CRDT gossip). It drives Server.ServiceLevel from the replica state and publishes OPC 10000-4 §6.6 Server.ServerRedundancy metadata, including non-transparent discovery and manual failover.

The bundled configuration uses the default in-memory shared key/value store. That is useful for understanding the DI wiring and for single-process experimentation, but separate OS processes do not share memory. A real multi-process or multi-host deployment needs a shared backend for ISharedKeyValueStore such as Redis; that backend is intentionally deferred from this small sample.

Run one instance

dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543

Connect an OPC UA client to opc.tcp://localhost:62543/RedundantServer and browse to Objects/High Availability. The Counter variable is writable and is also incremented once per second while this server is leader. ActiveReplica shows the local active writer label.

Both HA modes call UseReplicaNodeIdentity before startup. The sample's shared namespace occupies index 2 on every replica, while ApplicationUri and built-in diagnostics remain local. FactoryAssigned/Value and FactoryAssigned/Target are genuinely factory-created nodes, not preassigned constants; Target contains the exact NodeId of Value. Use the client's --identity scenario to cache their Browse results and reuse those same IDs after terminating the serving replica. See Replica-consistent NodeIds.

The startup output shows the effective OPC UA redundancy value and the ServiceLevel subrange that clients read. For example, with HA_NODE_ID=replica-a:

HA sample node 'replica-a' listening at opc.tcp://localhost:62543/RedundantServer; HA_MODE=ap; REDUNDANCY_MODE=Hot; ServiceLevel=255 (Healthy).
CurrentServerId: replica-a
Redundant peers: (none)
NTRS discovery capability and FindServers peer-set provider are enabled.
RequestServerStateChange is enabled for administrator-driven Maintenance/NoData failover.

Run two instances

Use distinct ports and stable HA_NODE_ID values. REDUNDANCY_MODE selects the OPC UA redundancy model. HA_REDUNDANT_PEERS provides the peer set used for ServerUriArray, RedundantServerArray, NTRS discovery registration, and FindServers peer results. Each peer entry is:

applicationUri|applicationName|discoveryUrl1+discoveryUrl2
$env:HA_NODE_ID = "replica-a"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-b|RedundantServer replica-b|opc.tcp://localhost:62544/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543

In a second terminal:

$env:HA_NODE_ID = "replica-b"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-a|RedundantServer replica-a|opc.tcp://localhost:62543/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544

With the default in-memory store, each process elects against its own private store, so this two-terminal setup demonstrates endpoint, node id, service-level, and redundancy metadata configuration rather than cross-process state transfer. To turn it into a real HA pair, register a shared ISharedKeyValueStore that both processes can reach, as shown below.

The second replica prints the configured redundancy value and the peer set:

HA sample node 'replica-b' listening at opc.tcp://localhost:62544/RedundantServer; HA_MODE=ap; REDUNDANCY_MODE=Hot; ServiceLevel=255 (Healthy).
CurrentServerId: replica-b
Redundant peers: urn:localhost:OPCFoundation:RedundantServer:replica-a [opc.tcp://localhost:62543/RedundantServer]
NTRS discovery capability and FindServers peer-set provider are enabled.
RequestServerStateChange is enabled for administrator-driven Maintenance/NoData failover.

Wire up a shared store for real HA

The single-instance defaults stay in effect until you supply a shared backend. The DI wiring below is what makes additions, removals, references, and values replicate across replicas. Replace the placeholder RedisSharedKeyValueStore with any ISharedKeyValueStore reachable by every replica (a Redis adapter is intentionally deferred from this sample):

builder.Services
    .AddOpcUa()
    .AddServer(o =>
    {
        o.ApplicationName = "RedundantServer";
        o.ApplicationUri = applicationUri;   // unique per replica
        o.EndpointUrls.Add(endpointUrl);
    })
    .AddNodeManager<HaSampleNodeManagerFactory>()
    .UseReplicaNodeIdentity(
        "opcfoundation-ha-sample",
        [HaSampleNodeManagerFactory.NamespaceUri],
        writerAssignedIds: true)
    .UseDistributedAddressSpace(d =>
    {
        // Shared backend reachable by every replica. The default is an
        // in-process InMemorySharedKeyValueStore that is NOT shared across
        // processes, so a real deployment must override it.
        d.KeyValueStoreFactory = sp => new RedisSharedKeyValueStore(redisConnectionString);

        // Authenticated encryption + integrity protection for every record at
        // rest. The 32-byte master key comes from a Kubernetes Secret / KMS and
        // is shared by all replicas.
        d.RecordProtectorFactory = _ => new AesCbcHmacRecordProtector(recordKey);

        d.UseLeaderElection = true;   // lease-based leader = the single writer
        d.NodeId = nodeId;            // unique per replica
        d.RedundancyMode = RedundancySupport.HotAndMirrored;
    })
    .UseDistributedSessions(s =>
    {
        // Mirror encrypted session state so a client can fail over to a standby
        // and reconnect with a full ActivateSession (the authentication token is
        // only a lookup key). The default is the safe re-auth-on-failover.
        s.EnableFastReconnect = true;
    })
    .AddServerRedundancy(r =>
    {
        r.Mode = RedundancySupport.HotAndMirrored;
        r.CurrentServerId = nodeId;
        r.RedundantPeers.Add(new RedundantPeer(
            "urn:host-b:OPCFoundation:RedundantServer:replica-b",
            new ArrayOf<string>("opc.tcp://host-b:62543/RedundantServer"))
        {
            ApplicationName = new LocalizedText("RedundantServer replica-b")
        });
    })
    .AddRequestServerStateChange();

Strong consistency with Raft (no external store)

The Redis placeholder above is one way to get a shared ISharedKeyValueStore. This sample also ships an in-package alternative: a strongly-consistent Raft cluster. Set HA_CONSISTENCY=strong and the active/passive path registers UseRedundancyConsistency(RedundancyConsistencyMode.Strong) backed by a multi-node RaftCs cluster over NanoMsg — the shared store, leader election, and single-use session nonce become linearizable and shared across every replica, with no Redis or other external dependency. Configure the cluster from the environment:

Setting Example Description
HA_CONSISTENCY strong, eventual strong backs the shared store with a Raft cluster; eventual (default) keeps today's behaviour.
HA_HISTORIAN true, false Enables the protected shared historian. It is valid only with HA_MODE=ap and HA_CONSISTENCY=strong; strong active/passive defaults to enabled.
HA_RAFT_ID 1 This replica's unique Raft node id (1..N).
HA_RAFT_MEMBERS 3 Static cluster size (odd 3/5 for a fault-tolerant quorum).
HA_RAFT_BIND tcp://0.0.0.0:6560 Local Raft transport bind address.
HA_RAFT_PEERS tcp://server-b:6560,tcp://server-c:6560 The other members' Raft transport addresses.

See docker-compose.yml with the active-passive.env env file for a runnable 3-node Raft cluster (real cross-container active/passive HA). Set HA_RECORD_KEY to one shared base64 32-byte key before starting it. The topology calls UseDistributedHistorian, historizes HighAvailability.Counter and HighAvailability.HistoryEvents, and keeps history writes plus raw, event, and processed HistoryRead continuation points visible across session takeover. The historian deliberately rejects HA_INSECURE, eventual storage, and active/active multi-writer topology because ordered values, deletes, modification chains, events, and atomic batches have no conflict-free merge contract.

For Kubernetes, UseKubernetesRaftConsensus in Opc.Ua.Redundancy.Kubernetes derives the same wiring from the StatefulSet ordinal and headless-Service DNS, with a file WAL on a PersistentVolume; see docs/Kubernetes.md.

Redundancy mode and discovery settings

Setting Values Description
REDUNDANCY_MODE none, cold, warm, hot, hotandmirrored, transparent Selects the Server.ServerRedundancy.RedundancySupport value. If unset, active/passive defaults to hot and active/active defaults to hotandmirrored.
HA_NODE_ID stable replica id Used as this replica's ApplicationUri suffix (unless HA_APPLICATION_URI overrides it) and as CurrentServerId for transparent redundancy.
HA_REPLICA_SET stable shared set id Defaults to opcfoundation-ha-sample. Must be identical on every replica; it is not the per-process HA_NODE_ID.
HA_REDUNDANT_PEERS `applicationUri applicationName
peerServerUris comma/semicolon-separated application URIs Legacy shorthand for RedundantServerArray; prefer HA_REDUNDANT_PEERS when clients must resolve peers through FindServers.
HA_MODE ap, aa Chooses active/passive shared-store replication or active/active CRDT gossip.
HA_PEER_DISCOVERY static, dns, lds, k8s Peer-discovery mechanism for the client-facing RedundantServerSet (FindServers) and, for active/active, the gossip fabric. static (default) uses the configured peers; dns resolves HA_SERVICE_NAME (one record per replica) and updates both dynamically. lds/k8s are wired via UseLdsPeerDiscovery / UseKubernetesPeerDiscovery. Static configuration is the fallback until discovery finds peers.
HA_SERVICE_NAME a DNS name The headless-service DNS name resolved by HA_PEER_DISCOVERY=dns (defaults to server).
HA_CONSISTENCY strong, eventual Selects the shared-store consistency model; strong backs it with a Raft cluster (see Strong consistency with Raft).
HA_HISTORIAN true, false Enables the shared historian and durable HistoryRead continuation store. Supported only by strong active/passive topology; defaults to true there and false elsewhere.
HA_FAST_RECONNECT true, false Allows token-reuse reconnect for mirrored sessions. The default requires full ActivateSession re-authentication after failover.
HA_RECORD_KEY base64 32-byte key Shared record-protection key for the distributed topologies. When set, every mirrored record (session secrets, identity tokens, notifications) is encrypted + integrity-protected at rest with AesCbcHmacRecordProtector. Use the same value on every replica; provision from a Kubernetes Secret / KMS in production.
HA_INSECURE true, false Explicit, auditable opt-out that runs an isolated demo without record protection or gossip authentication (prints a warning). Secure by default: without HA_RECORD_KEY and without this flag, a distributed topology fails closed at startup. Never set in production.
HA_BALANCING_URL a discovery URL Enables GetEndpoints load direction (see below). A GetEndpoints request on this virtual/LB discovery URL is answered with the best replica's endpoints; empty (default) disables it.
HA_APPLICATION_URI a URI Overrides the per-replica ApplicationUri with a shared one. Required for transparent redundancy so every replica presents one logical server identity (CreateSession validates the client serverUri against it).
HA_SUBJECT_NAME a certificate subject Sets an explicit, stable certificate subject so replicas sharing a PKI store load one ApplicationInstanceCertificate.
HA_PKI_ROOT a filesystem path Points the certificate stores at a shared directory. Combined with HA_SUBJECT_NAME, replicas share one ApplicationInstanceCertificate (production provisions this from a secret rather than a shared volume).

The sample also enables RequestServerStateChange with AddRequestServerStateChange(). An administrator client can call the standard method on Server to request Maintenance or NoData behavior and set Server.EstimatedReturnTime; the server updates Server.ServiceLevel into the appropriate OPC UA subrange so clients back off or fail over.

Mode-specific nodes shown by the sample:

  • none — leaves redundancy metadata in its single-server defaults.
  • cold, warm, hot, hotandmirrored — publish ServerUriArray from HA_REDUNDANT_PEERS, retain RedundantServerArray, advertise NTRS, and return the peer ApplicationDescription values from FindServers.
  • transparent — publishes CurrentServerId and RedundantServerArray for the transparent set.

Server.ServiceLevel is driven by the sub-range-aware provider: leaders report Healthy, warm standby reports Degraded, hot/hot-and-mirrored replicas report Healthy, and cold standby reports NoData. The console prints the selected mode, server id, peer set, and initial service-level subrange at startup.

GetEndpoints load direction

Setting HA_BALANCING_URL registers UseServerLoadDirection(...): every replica publishes its health ServiceLevel, a load weight, and its endpoints to the shared store, and a GetEndpoints request that arrives on the balancing URL is answered with the best replica's endpoints — the active writer in active/passive, or the least-loaded healthy replica in active/active. Plain discovery on a replica's own URL is unaffected, and this complements (never replaces) the standard client-driven Server.ServiceLevel / RedundantServerArray selection. In a real deployment a load balancer / Kubernetes Service fronts the replicas at the balancing URL; the sample sets StrongEligibility when HA_CONSISTENCY=strong so the eligibility keyspaces are linearizable. It requires a shared store across replicas (use HA_CONSISTENCY=strong / the Raft compose). See docs/HighAvailability.md for the design, conformance, and security notes.

Transparent redundancy (single virtual endpoint)

The load direction above is one endpoint model; the other is transparent redundancy, where every replica presents one logical server behind one virtual endpoint. Unlike the non-transparent modes (the client reads RedundantServerArray and selects a replica), a transparent client sees a single endpoint and never chooses a replica — a load balancer routes it and mirrored session state lets it continue across a replica failure.

To present one logical server, all replicas must share:

  • One ApplicationUri (HA_APPLICATION_URI). CreateSession validates the client-supplied serverUri against the server's ApplicationUri, so replicas that disagree would reject sessions established via discovery on a peer.
  • One ApplicationInstanceCertificate (HA_SUBJECT_NAME + HA_PKI_ROOT pointing at a shared store). Under SecurityMode None the certificate is not exchanged, but secured deployments must present the same certificate from every replica.
  • Mirrored session and address-space state (HA_MODE=aa, REDUNDANCY_MODE=transparent) so a session created on one replica can be resumed on another.

Each replica advertises the virtual endpoint URL (HA_HOST = the load-balancer host). Because that host is a DNS name, the listener binds to all interfaces in the replica's own container while returning opc.tcp://<lb>:62543/... to clients, so discovery and CreateSession echo the single endpoint the client actually uses (ServerBase.FilterByEndpointUrl matches the client host to the advertised base address). A client therefore connects to one URL, and on a replica failure the load balancer routes the reconnect to the survivor, where the mirrored session resumes with a token-reuse reconnect (the full ActivateSession signature re-check still applies).

Transparent/docker-compose.yml runs this end to end: two REDUNDANCY_MODE=transparent replicas sharing one ApplicationUri and one certificate (seeded into a shared PKI volume by the first replica, then reused by the second) behind an nginx TCP load balancer that publishes the single virtual endpoint opc.tcp://localhost:62543/RedundantServer, plus a client that connects only to that endpoint. In production the shared certificate is provisioned from a Kubernetes Secret / KMS to every replica rather than self-generated (see docs/Kubernetes.md), which also removes the first-start certificate race.

Active/passive vs active/active

The sample selects its redundancy topology with the HA_MODE environment variable — ap (active/passive, the default) or aa (active/active) — and references both Opc.Ua.Redundancy.Server and Opc.Ua.Redundancy.Server:

  • Active/passive (HA_MODE=ap) — UseDistributedAddressSpace + UseDistributedSessions with leader election. With HA_CONSISTENCY=strong, UseDistributedHistorian adds one protected, leader-write archive with writer epochs and durable portable HistoryRead continuations. One replica is the active writer; standbys hydrate from the shared store and take over on failover. Redundancy is reported as RedundancySupport.Hot, and clients follow Server.ServiceLevel / Server.ServerRedundancy to find the active replica.
  • Active/active (HA_MODE=aa) — UseActiveActiveRedundancy (which wires UseReplicatedAddressSpace + UseReplicatedSessions from one set of gossip options; the individual methods remain for advanced setups). Every replica accepts writes and converges without a leader; redundancy is reported as RedundancySupport.HotAndMirrored. Replicas gossip over TCP: set HA_GOSSIP_PORT (default 4840; session entries gossip on port + 1) and HA_GOSSIP_PEERS (a comma/semicolon list of host:port for the other replicas' address-space gossip). A session created on one replica can be resumed on another with ManagedSessionBuilder.WithTokenReuseFailover() on the client. Distributed Historical Access is not supported in this multi-writer mode and fails closed when requested.

Set HA_PEER_DISCOVERY=dns (with HA_SERVICE_NAME) to discover peers dynamically instead of a static HA_GOSSIP_PEERS list: the sample registers UseDnsPeerDiscovery, which keeps the client-facing RedundantServerSet (FindServers) and the active/active gossip fabric current as replicas scale up and down. Static configuration (AddServerRedundancy / HA_REDUNDANT_PEERS) is the fallback. HA_PEER_DISCOVERY=lds (UseLdsPeerDiscovery) and Kubernetes EndpointSlice discovery (UseKubernetesPeerDiscovery) are also available — see docs/HighAvailability.md → Dynamic peer discovery.

Run two active/passive replicas

Use HA_MODE=ap (or omit it) for the leader-elected active/passive setup. A real pair needs a shared ISharedKeyValueStore; with the default in-memory store this remains a wiring demonstration.

# replica A
$env:HA_NODE_ID = "replica-a"
$env:HA_MODE = "ap"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-b|RedundantServer replica-b|opc.tcp://localhost:62544/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543

In a second terminal:

# replica B
$env:HA_NODE_ID = "replica-b"
$env:HA_MODE = "ap"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-a|RedundantServer replica-a|opc.tcp://localhost:62543/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544

The active replica writes Counter, the standby mirrors state through the shared store, and clients use Server.ServiceLevel, Server.ServerRedundancy.RedundancySupport=Hot, and FindServers peer discovery to select or fail over to the active endpoint.

Run two active/active replicas

# replica A
$env:HA_NODE_ID = "replica-a"
$env:HA_MODE = "aa"
$env:HA_GOSSIP_PORT = "4840"
$env:HA_GOSSIP_PEERS = "127.0.0.1:4842"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543

In a second terminal:

# replica B
$env:HA_NODE_ID = "replica-b"
$env:HA_MODE = "aa"
$env:HA_GOSSIP_PORT = "4842"
$env:HA_GOSSIP_PEERS = "127.0.0.1:4840"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544

Both replicas now accept writes to Counter and converge: a write on either endpoint propagates to the other by gossip, and the per-second increment runs on every replica, with CRDT last-writer-wins resolving the concurrent updates. Unlike the active/passive store-backed setup, active/active needs no shared ISharedKeyValueStore between processes — the gossip transport carries the state.

Docker Compose

One compose file — docker-compose.yml — runs the sample as a 3-replica set; the topology is chosen from the environment (the build context is the repository root). Two extra files cover the transparent and load-direction demos.

# Active/active, eventual consistency (CRDT gossip, no shared store) — the default.
docker compose --env-file samples\Redundancy\RedundantServer\active-active-eventual.env `
  -f samples\Redundancy\RedundantServer\docker-compose.yml up --build

# Active/active, strong consistency (3-node RaftCs, shared linearizable store).
docker compose --env-file samples\Redundancy\RedundantServer\active-active-strong.env `
  -f samples\Redundancy\RedundantServer\docker-compose.yml up --build

# Active/passive, strong consistency (3-node RaftCs; one active writer, two hot standbys).
docker compose --env-file samples\Redundancy\RedundantServer\active-passive.env `
  -f samples\Redundancy\RedundantServer\docker-compose.yml up --build

# GetEndpoints load direction over the Raft cluster (sets HA_BALANCING_URL on every replica).
docker compose -f samples\Redundancy\RedundantServer\LoadDirection\docker-compose.yml up --build

# Transparent redundancy: replicas as ONE logical server behind an nginx load balancer
# on a single virtual endpoint (opc.tcp://localhost:62543/RedundantServer).
docker compose -f samples\Redundancy\RedundantServer\Transparent\docker-compose.yml up --build

Each replica exposes its OPC UA endpoint on the host (opc.tcp://localhost:62543/RedundantServer, :62544, :62545). Set HA_HOST to the reachable hostname (the compose file uses the container/service name) so peers and clients connect across the container network. Three replicas run in every topology (Raft needs a 3-node quorum; active/active eventual works with 2 or 3 — remove server-c for a 2-node run, so the replica count is configurable). The Raft topologies are real cross-container HA deployments whose RaftCs cluster is the shared, linearizable store; active/active eventual converges leaderlessly over CRDT gossip with no shared store.

Scale active/active eventual replicas

Scale/docker-compose.yml has a single server service for active/active eventual (CRDT gossip) runs. Docker Compose can scale it to any replica count because the sample discovers peers from the server DNS alias instead of a fixed HA_GOSSIP_PEERS list:

docker compose -f samples\Redundancy\RedundantServer\Scale\docker-compose.yml up --build --scale server=5

The scale file sets HA_PEER_DISCOVERY=dns and HA_SERVICE_NAME=server; every replica resolves the service alias, removes its own container IP, and seeds gossip with the remaining task IPs on HA_GOSSIP_PORT (session gossip uses the next port). No host ports are published, so clients should run inside the same compose network or attach to it. This dynamic scaling path is gossip-only. Dynamic Raft scaling is intentionally unsupported because Raft needs stable replica identities and an odd quorum; use the Kubernetes StatefulSet deployment with UseKubernetesRaftConsensus for strong consistency.

The same file has a clients profile that runs independent managed clients in their own containers — each opts into WithServerRedundancy() and fails over on its own, so it scales to any count alongside the servers (no shared client store needed):

docker compose -f samples\Redundancy\RedundantServer\Scale\docker-compose.yml --profile clients up --build --scale server=3 --scale client=2

For a coordinated single-active client replica set (exactly one client active, the rest hot/warm/cold standbys), use AddRedundantClientSession + a CAS-capable AddRaftClientSharedStore (a fixed Raft quorum, like the server's active/passive) rather than independent clients — see HighAvailability.md. Each client replica is its own process; there is no in-process replica set.

For a client-centric compose that runs the full HA matrix (server consistency × client mode) from one env-driven file and logs failover / HA / data loss on both sides, use samples/Redundancy/RedundantClient/docker-compose.yml: COMPOSE_PROFILES selects server-eventual (shows data loss) or server-strong (Raft, no data loss) crossed with client-independent or client-coordinated. See the RedundantClient README.

Watch replication and failover

The clients (and demo) profile adds one or more RedundantClient instances — one process per replica; set CLIENT_REPLICAS to scale the number of client containers. To see replication and failover end to end:

# 1. Start the server set (active/passive here) plus a client that prints the replicated Counter.
docker compose --env-file samples\Redundancy\RedundantServer\active-passive.env `
  -f samples\Redundancy\RedundantServer\docker-compose.yml --profile clients up --build

# 2. In another shell, stop the active/leader replica and watch the client keep going:
docker compose -f samples\Redundancy\RedundantServer\docker-compose.yml stop server-a

The client logs its reconnect/redirect to a surviving replica across the failover. Bring the replica back with docker compose ... start server-a; the Raft cluster re-admits it. Set CLIENT_REPLICAS=3 to run three independent client processes, each of which fails over on its own (for a coordinated single-active client set, use AddRedundantClientSession over a Raft client store as above). On the strong-consistency Raft topology the HighAvailability.Counter continues across the failover: the sample wires it through the distributed value cache (see Sharing values across replicas), so the promoted replica resumes from the last value the former leader shared.

For the broader design, see HighAvailability.md. For an environment-driven replica-set deployment, see Kubernetes.md.

Sharing values across replicas

OPC 10000-4 §6.6 requires identical NodeIds and address spaces across a redundant set but does not standardize how live process values are shared. Opc.Ua.Redundancy.Server adds an opt-in extension for this: a variable's read/write callbacks can participate in a distributed value cache (IDistributedValueCache) so the last value is cached in the shared store and served — within a freshness bound — from any replica.

UseDistributedAddressSpace registers IDistributedValueCache in dependency injection (backed by the distributed node-state store; a consumer that constructs the store directly can build a DistributedValueCache itself). The sample injects it into HaSampleNodeManager and wires the Counter:

// Opt the Counter's read/write callbacks into the distributed value cache.
counter.EnableDistributedValueParticipation(
    valueCache,
    maxAge: TimeSpan.FromSeconds(10),
    liveRead: _ => new ValueTask<DataValue>(ReadLocalCounter()));

// The active replica writes each new value through to the shared store...
await valueCache.CacheAsync(counter.NodeId,
    new DataValue(Variant.From(value), StatusCodes.Good, DateTimeUtc.Now));

// ...and a replica that has just been promoted seeds from the last shared value,
// so the Counter continues instead of restarting.
(_, DataValue cached) = await valueCache.TryGetAsync(counter.NodeId, maxAge);

On the strong-consistency (Raft) topology this makes the Counter genuinely shared: standby replicas serve the active replica's value, and after a failover the promoted replica resumes from the last value the former leader wrote (verified end-to-end). Monitored items keep reading through the normal pipeline, so a client that monitors a standby observes the shared value only through the participating read path — consistent with OPC UA monitored-item semantics.

Value sharing uses the shared store, which is protected: supply HA_RECORD_KEY (secure by default) or set HA_INSECURE=true for an isolated non-historian demo — see the settings table above. The distributed historian and its portable continuation records always require HA_RECORD_KEY; they never accept the no-op protector.

Notes

  • Active/active eventual convergence over TCP/UDP gossip. State propagation between active/active replicas requires each CRDT snapshot to be sent as a Crdt.Transport FrameCodec frame (MessageType.State); the TCP/UDP gossip transports Decode on send and would otherwise throw System.IO.InvalidDataException: Frame length does not match the encoded body length. The stack's FramingGossipTransport decorator handles this framing for UseReplicatedAddressSpace/UseReplicatedSessions, so cross-process Counter and session state converge over real gossip (regression-tested with two real TcpGossipTransports). The in-memory transport is a transparent passthrough, so single-process runs behaved correctly regardless.
  • Value continuity across failover. The Counter is shared and continues across failover through the distributed value cache: on the strong-consistency (Raft) topology it rides the linearizable Raft store, and on active/active eventual it rides the address-space CRDT gossip. Monitored items read through the normal pipeline, so a monitored item on a standby observes the shared value only through the participating read path (as documented in DistributedValueParticipation).