## Summary Fixes the findings of a bug audit of `Opc.Ua.Server`, `Opc.Ua.Gds.Server` and `Opc.Ua.Gds.Client`. Rebased on master after #4516. What went in: - **Audit:** 93 findings, each checked against the OPC UA specification. - **Code review of those fixes:** 35 further findings. - **Spec grounding of every fix commit:** 18 conformance gaps, now closed. Commits are one per finding, named `Fix <ID>`, `Fix review <ID>` or `Fix spec gap <name>`. The IDs refer to the audit report (S = Server areas, G1 = GDS server, G2 = GDS client). Validation: - Each finding was traced in code and checked against OPC 10000 (parts 3, 4, 5, 9, 11, 12, 13, 18, 20). - Every fix commit was checked against the spec text again afterwards: 0 non-conformant, remaining gaps fixed. - Regression tests were added for most fixes. - Suites run on net10.0 before the rebase: Server, GDS, History, Configuration, Features, Core.Security, DI, Redundancy, LDS, InformationModel, PubSub.Server, Client, Sessions and the three Subscriptions suites, all green. The same suites on the rebased branch are still running; results will be posted as a comment. ## Behaviour and user-facing changes ### Secure defaults - **`Historize()` / `HistorianBuilder`** grant HistoryRead only. History writes need an explicit `historyAccessLevel: HistoryRead | HistoryWrite`. - **File-system mount:** - Anonymous sessions can no longer create, delete, move or open files for writing (`Bad_UserAccessDenied`). - UserWritable and UserExecutable now reflect the user. - `AllowAnonymousWrite` restores the old behaviour; the ReferenceServer sample turns it on. - **Default PKI root** for the hosted server, hosted GDS, hosted LDS and `OpcUaApplicationConfigurationProvider`: - Moved from the shared temp directory to `{LocalApplicationData}/OPC Foundation/{ApplicationName}/pki`, and startup fails if no per-user directory exists. - When certificate stores of an earlier version still exist under temp, a warning names both paths. Existing deployments that relied on the default get a new application certificate (and the hosted GDS a new CA) unless `PkiRoot` is set to the old path. - **GDS:** - `GdsServerOptions.AutoApprove` defaults to `false`. - The default KeyCredential store grants no roles unless `AuthorizeRoles` allows them. - Certificate requests (StartSigningRequest, StartNewKeyPairRequest, FinishRequest) need an encrypted channel. - **Hosted server anonymous logins:** - Anonymous is rejected (`Bad_IdentityTokenRejected`) when the default authenticator options disable it. - A custom `AddIdentityAuthenticator<T>()` on its own no longer disables anonymous. - With anonymous disabled and no configured token policies, the endpoint advertises the authenticators' token types instead of Anonymous. - **Runtime-added roles:** their properties and methods need SecurityAdmin over an encrypted channel, like the well-known roles. - **Disabled users:** users disabled through Part 18 UserManagement can no longer log in through `RegisterDefaultAuthenticators(IUserDatabase)`. ### GDS server (Part 12) - **Certificate requests:** - FinishRequest / ReadRequest with another application's RequestId returns `Bad_InvalidArgument`. - Every Start*Request returns a new RequestId. - A CSR without the ApplicationUri is rejected (`Bad_CertificateUriInvalid`). - An abstract or unsupported certificate type is rejected at Start (`Bad_InvalidArgument`). - **Private-key password:** no longer persisted in the JSON applications database. A password saved by an earlier version is read into memory once and dropped on the next save. - **CheckRevocationStatus:** validates offline against the GDS's own trust lists and CRLs. It returns Good for good GDS-issued certificates and `Bad_CertificateInvalid` for input that is not a certificate. - **Certificate-group trust lists:** - Writes need CertificateAuthorityAdmin; SecurityAdmin and application admins are read-only. - Writes are validated with the Part 4 certificate validation. - **AuthorizationService:** - The legacy RequestAccessToken applies `AccessControl` and issues the token for the session identity. - StartRequestToken passes the session identity to every identity-aware provider. - Unknown ResourceIds return `Bad_NotFound`. - Expired token requests and refresh tokens are pruned, with at most 10 000 pending requests (`Bad_TooManyOperations`). - **KeyCredential:** requests with PublicKey + SecurityPolicyUri are rejected with `Bad_SecurityPolicyRejected` (secret encryption is not implemented); a PublicKey without SecurityPolicyUri is rejected with `Bad_InvalidArgument`. - **Audit events:** access-token audit events no longer contain the password or token data. - **RegisterTickets / UnregisterTickets:** need the RegistrarAdmin role when called from a session. - **Managed application nodes:** moved from namespace 0 to their own namespace (`ManagedApplicationsNamespaceUri`); the NodeId is the ApplicationUri, and the mandatory child properties exist. - **Hosted GDS:** - New `GdsServerOptions.CertificateGroups` (`GdsCertificateGroupOptions`), bindable from configuration. - The mandatory DefaultApplicationGroup is always present. - The default CA subject uses the resolved application name. - **JSON applications database:** - A corrupt file throws instead of being silently replaced by an empty database. - Saves are atomic and write in place on bind mounts; on net8+ builds they also follow symlinks and keep the Unix file mode. - Issued certificates and trust-list links survive a reload. ### GDS client - **Trust-list download:** - Reads until an empty chunk (Part 20 short reads). - A size-limit rejection of the final end-of-file read, as sent by released servers, counts as end of file. - **Default certificate group:** a null group means DefaultApplicationGroup, and vendor groups in other namespaces are no longer mistaken for well-known groups. - **Status codes and errors:** - Missing directory proxies report `Bad_ServerNotConnected`. - Short or wrongly typed method outputs throw `BadUnexpectedError` / `BadTypeMismatch`. - **GdsAccessTokenProvider:** no longer caches a failed client task, and one caller's cancellation no longer fails other callers. ### Trust lists, push and ConfigurationFile (Part 12 / Part 20) - **AddCertificate:** - `IsTrustedCertificate=false` returns `Bad_CertificateInvalid`. - Added certificates are validated (Part 4), and a certificate revoked by a current CRL returns `Bad_CertificateRevoked`. - **CloseAndUpdate:** validates the uploaded list against its own certificates and CRLs, not the stores it replaces. Validation failures are no longer recorded in the server's Rejected store. - **RemoveCertificate:** removing a trusted or issuer CA that other certificates still need returns `Bad_CertificateChainIncomplete`. - **Handles:** - Several read handles may be open at once. - Any open while a write handle exists returns `Bad_NotReadable` / `Bad_NotWritable`. - A write open while any handle is open returns `Bad_NotWritable`. - Handles of closed sessions are released. - Idle handles close after ActivityTimeout (default 60 s). - **Status codes:** - Read clamps to the remaining bytes and no longer enforces the size limit. - Oversized Write / AddCertificate returns `Bad_RequestTooLarge`. - Unsupported open modes return `Bad_NotSupported`. - **ConfigurationFile.CloseAndUpdate:** - Empty Targets return `Bad_InvalidArgument`. - The ConfigurationUpdatedAuditEvent is raised with the configuration owner as source. - With the new `IApplicationConfigurationFileTargetProvider`, providers receive the Targets, a bad target returns Uncertain with per-target results, and session-interrupting updates are applied after RestartDelayTime. - **Default namespace permissions:** namespaces registered after the first permission lookup now get their NamespaceMetadata default RolePermissions and AccessRestrictions (no stale negative cache). ### Sessions, identity and roles (Part 18) - **Session timeouts:** a failure closing one timed-out session no longer stops the session-timeout monitor. - **ChangePassword:** an empty old password returns `Bad_IdentityTokenInvalid`. - **Removing or disabling a user:** closes only that user's UserName-token sessions. - **Role configuration changes:** roles granted by an `ImpersonateUser` mapping are kept. - **X509Subject rules:** a certificate subject value containing a quote never matches an X509Subject rule. - **`ImpersonateUser` callback:** no longer runs under the global session lock. - **Session diagnostics:** LocaleIds are kept as the client sent them, while null LocaleIds are ignored for translation. ### Subscriptions and monitored items (Part 4) - **Acknowledgements:** a failing acknowledgement returns a per-entry Bad status instead of failing the whole Publish. - **`DiscardedMessageCount`:** counts messages dropped from the retransmission queue and unsent overflow messages. - **Durable subscriptions:** transferring a restored anonymous subscription works (no NullReferenceException). - **Transferred subscriptions:** sampling and permission checks use the item's current owner session. - **Access denied:** a monitored item whose user may no longer read the attribute gets a `Bad_UserAccessDenied` notification instead of going silent. Non-Value attributes need Browse permission (RolePermissions needs ReadRolePermissions), and User* attributes are evaluated per subscriber. - **ModifyMonitoredItems:** a SamplingInterval of -1 uses the subscription's publishing interval. - **AggregateFilter priming:** needs HistoryRead / ReadHistory permission and is bounded (100 000 values, bounded page size). - **Resource leak:** event monitored items no longer leak a consumer task per create/delete cycle. ### Node management services (Part 3 / Part 4) - **Reference types:** AddNodes, AddReferences, DeleteReferences and Browse reject ids that are not ReferenceTypes with `Bad_ReferenceTypeIdInvalid`. AddNodes, AddReferences and DeleteReferences also reject abstract ReferenceTypes; Browse still accepts them (e.g. `HierarchicalReferences` with includeSubtypes). - **Concurrent AddNodes:** duplicate BrowseNames under one parent get `Bad_BrowseNameDuplicated`. - **TranslateBrowsePathsToNodeIds:** - Requires Browse permission on intermediate nodes and final targets (RolePermissions and AccessRestrictions). - A long RelativePath no longer rejects the whole request. - **SemanticChangeEvent:** - Raised once per actual change of a semantic property, whether or not it is monitored. - Also raised for properties with the SemanticChange access-level bit. - Reports the owning node's TypeDefinition as AffectedType. - **Localized diagnostics:** operation-level text is translated when OperationLocalizedText is requested. ### History and aggregates (Part 11 / Part 13) - **DeleteRawModified:** an unspecified or reversed range returns `Bad_InvalidTimestampArgument`, and StartTime == EndTime deletes the value at StartTime for every provider. - **ReadAtTime fallback:** reads only the values around each requested time. A time whose bound is not found within the search limit returns `Bad_BoundNotSupported` instead of failing the request. - **ReadProcessed:** a ProcessingInterval shorter than one tick returns `Bad_AggregateInvalidInputs`. - **Delta / DeltaBounds:** widen to the next larger signed type when the result does not fit the source type (e.g. a decreasing unsigned counter, or an Int16 overflow). - **Range / Range2:** return the source data type. ### Alarms and fluent API (Part 9) - **Suppression:** an alarm stays suppressed while any suppression group is active, and re-evaluation without a state change does not move TransitionTime. - **`MonitorVariable(variable)`:** sets SourceNode, SourceName and InputNode to the variable and makes it a ConditionSource (HasCondition to the alarm, HasEventSource from the object). - **Fluent input/output arguments:** InputArguments / OutputArguments added through the fluent API to a registered method are readable. - **Root notifiers:** a failed root-notifier registration no longer leaves an orphaned event worker. - **State-machine handlers:** exceptions are logged at Error. ### File system (Part 20) - **Delete / MoveOrCopy:** - Only act on direct children of the directory. - A NodeId whose kind does not match the entry returns `Bad_NotFound` (a file NodeId can no longer delete a directory). - Copying a directory into its own subtree returns `Bad_InvalidArgument`. - **Locking:** - An open file, or a directory containing one, cannot be deleted or moved (`Bad_InvalidState`). - The check is atomic with opens. - While a Delete or Move runs, read opens get `Bad_NotReadable` and write opens `Bad_InvalidState`. - **Open:** - Reserved mode bits or EraseExisting without Write return `Bad_InvalidArgument`. - Read+Write returns `Bad_NotSupported`. - Read while a writer is open returns `Bad_NotReadable`. - **Client `FileSystemErrors`:** maps `Bad_NotReadable` to `IOException`. - **RuntimeNodeSet:** well-formed NodeSets with a stray `Model` element, or two models in one file where one requires the other, now load. ### Server core, hosting and lifecycle - **`SessionEstablishmentQueueLimit` > 0:** CreateSession/ActivateSession now queue for a permit. The permit is acquired before the request is registered, and a waiting handshake releases its worker through the request-parking policy. - **Node manager lifecycle:** - Add, reload and remove no longer wait for parked Publish requests. - A Migrate reload racing with CreateMonitoredItems no longer blocks later lifecycle operations. - Late items on retired or removed managers get `Bad_NodeIdUnknown`. - **HistoryUpdate:** a batch mixing data and event details is bounded by both limits. - **Server stop/start:** stopping the server disposes the configuration watcher and certificate subscription; starting recreates them. - **Monitored-item queue factory:** a factory supplied by the caller or the DI container is no longer disposed by the server. - **Audit events:** each certificate validation error raises its own AuditCertificate event. A failed OpenSecureChannel without a service result is audited as failed. ## New public API (additive) - `IQueuedSessionEstablishmentLimiter` - `StandardServer.OwnsMonitoredItemQueueFactory` (protected virtual) - `DefaultPkiRoot.Get` (Opc.Ua.Configuration) - `TrustList.SetCertificateValidation` - `NamespaceMetadataRegistry.Invalidate` - `IApplicationConfigurationFileTargetProvider` and `ApplicationConfigurationUpdatePlan` - `GdsServerOptions.CertificateGroups` / `GdsCertificateGroupOptions` - `InMemoryKeyCredentialRequestStore.AuthorizeRoles` - `DefaultManagedApplicationsNodeManager.ManagedApplicationsNamespaceUri` - `AuthorizationServiceManager.RequestAccessTokenAsync(..., IUserIdentity?, ...)` - `AllowAnonymousWrite` on the file-system node manager and its factory - `IFileSystemHost.TryBeginMutation` / `EndMutation`, replacing `HasOpenHandles` ## Not fixed / known limitations - **KeyCredential:** secret encryption with the client's PublicKey (requests needing it are rejected). - **CSR SAN restriction:** restricting subject / SAN values copied from a CSR (CA policy). - **Ticket signatures:** ticket signature verification (needs a trusted-signer configuration). - **FileType Read+Write open:** needs a read/write stream contract in `IFileSystemProvider`. - **Custom `IUserDatabase` implementations:** the disabled-user check still scans all users per login. The built-in stores do a single lookup. - **64-bit Delta precision:** limited by the calculator's double arithmetic. ## Tests with changed expectations Existing tests were updated where they asserted the old behaviour: - File-system status codes - Trust-list handle eviction - `DiscardedMessageCount` - Localized diagnostics bits - Delta on unsigned sources - SourceNode of fluent alarms - ReferenceServer transfer permission test (now expects `Bad_UserAccessDenied` notifications) - `UserDeactivationShutdownReviewTests` (the revoked session is a UserName login) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
High Availability Server
This sample is a minimal Generic Host based OPC UA server that demonstrates the distributed high-availability server building blocks in Opc.Ua.Redundancy.Server and the active/active building blocks in Opc.Ua.Redundancy.Server. It registers an AsyncCustomNodeManager-derived node manager so the local address space participates in replication, and selects its topology with the HA_MODE environment variable: active/passive (ap, the default — leader election so only the active replica writes) or active/active (aa — every replica writes and converges by CRDT gossip). It drives Server.ServiceLevel from the replica state and publishes OPC 10000-4 §6.6 Server.ServerRedundancy metadata, including non-transparent discovery and manual failover.
The bundled configuration uses the default in-memory shared key/value store. That is useful for understanding the DI wiring and for single-process experimentation, but separate OS processes do not share memory. A real multi-process or multi-host deployment needs a shared backend for ISharedKeyValueStore such as Redis; that backend is intentionally deferred from this small sample.
Run one instance
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
Connect an OPC UA client to opc.tcp://localhost:62543/RedundantServer and browse to Objects/High Availability. The Counter variable is writable and is also incremented once per second while this server is leader. ActiveReplica shows the local active writer label.
Both HA modes call UseReplicaNodeIdentity before startup. The sample's shared
namespace occupies index 2 on every replica, while ApplicationUri and built-in
diagnostics remain local. FactoryAssigned/Value and FactoryAssigned/Target
are genuinely factory-created nodes, not preassigned constants; Target contains
the exact NodeId of Value. Use the client's --identity scenario to cache
their Browse results and reuse those same IDs after terminating the serving
replica. See Replica-consistent NodeIds.
The startup output shows the effective OPC UA redundancy value and the ServiceLevel subrange that clients read. For example, with HA_NODE_ID=replica-a:
HA sample node 'replica-a' listening at opc.tcp://localhost:62543/RedundantServer; HA_MODE=ap; REDUNDANCY_MODE=Hot; ServiceLevel=255 (Healthy).
CurrentServerId: replica-a
Redundant peers: (none)
NTRS discovery capability and FindServers peer-set provider are enabled.
RequestServerStateChange is enabled for administrator-driven Maintenance/NoData failover.
Run two instances
Use distinct ports and stable HA_NODE_ID values. REDUNDANCY_MODE selects the OPC UA redundancy model. HA_REDUNDANT_PEERS provides the peer set used for ServerUriArray, RedundantServerArray, NTRS discovery registration, and FindServers peer results. Each peer entry is:
applicationUri|applicationName|discoveryUrl1+discoveryUrl2
$env:HA_NODE_ID = "replica-a"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-b|RedundantServer replica-b|opc.tcp://localhost:62544/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
In a second terminal:
$env:HA_NODE_ID = "replica-b"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-a|RedundantServer replica-a|opc.tcp://localhost:62543/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544
With the default in-memory store, each process elects against its own private store, so this two-terminal setup demonstrates endpoint, node id, service-level, and redundancy metadata configuration rather than cross-process state transfer. To turn it into a real HA pair, register a shared ISharedKeyValueStore that both processes can reach, as shown below.
The second replica prints the configured redundancy value and the peer set:
HA sample node 'replica-b' listening at opc.tcp://localhost:62544/RedundantServer; HA_MODE=ap; REDUNDANCY_MODE=Hot; ServiceLevel=255 (Healthy).
CurrentServerId: replica-b
Redundant peers: urn:localhost:OPCFoundation:RedundantServer:replica-a [opc.tcp://localhost:62543/RedundantServer]
NTRS discovery capability and FindServers peer-set provider are enabled.
RequestServerStateChange is enabled for administrator-driven Maintenance/NoData failover.
Wire up a shared store for real HA
The single-instance defaults stay in effect until you supply a shared backend. The DI wiring below is what makes additions, removals, references, and values replicate across replicas. Replace the placeholder RedisSharedKeyValueStore with any ISharedKeyValueStore reachable by every replica (a Redis adapter is intentionally deferred from this sample):
builder.Services
.AddOpcUa()
.AddServer(o =>
{
o.ApplicationName = "RedundantServer";
o.ApplicationUri = applicationUri; // unique per replica
o.EndpointUrls.Add(endpointUrl);
})
.AddNodeManager<HaSampleNodeManagerFactory>()
.UseReplicaNodeIdentity(
"opcfoundation-ha-sample",
[HaSampleNodeManagerFactory.NamespaceUri],
writerAssignedIds: true)
.UseDistributedAddressSpace(d =>
{
// Shared backend reachable by every replica. The default is an
// in-process InMemorySharedKeyValueStore that is NOT shared across
// processes, so a real deployment must override it.
d.KeyValueStoreFactory = sp => new RedisSharedKeyValueStore(redisConnectionString);
// Authenticated encryption + integrity protection for every record at
// rest. The 32-byte master key comes from a Kubernetes Secret / KMS and
// is shared by all replicas.
d.RecordProtectorFactory = _ => new AesCbcHmacRecordProtector(recordKey);
d.UseLeaderElection = true; // lease-based leader = the single writer
d.NodeId = nodeId; // unique per replica
d.RedundancyMode = RedundancySupport.HotAndMirrored;
})
.UseDistributedSessions(s =>
{
// Mirror encrypted session state so a client can fail over to a standby
// and reconnect with a full ActivateSession (the authentication token is
// only a lookup key). The default is the safe re-auth-on-failover.
s.EnableFastReconnect = true;
})
.AddServerRedundancy(r =>
{
r.Mode = RedundancySupport.HotAndMirrored;
r.CurrentServerId = nodeId;
r.RedundantPeers.Add(new RedundantPeer(
"urn:host-b:OPCFoundation:RedundantServer:replica-b",
new ArrayOf<string>("opc.tcp://host-b:62543/RedundantServer"))
{
ApplicationName = new LocalizedText("RedundantServer replica-b")
});
})
.AddRequestServerStateChange();
Strong consistency with Raft (no external store)
The Redis placeholder above is one way to get a shared ISharedKeyValueStore. This sample also ships an in-package alternative: a strongly-consistent Raft cluster. Set HA_CONSISTENCY=strong and the active/passive path registers UseRedundancyConsistency(RedundancyConsistencyMode.Strong) backed by a multi-node RaftCs cluster over NanoMsg — the shared store, leader election, and single-use session nonce become linearizable and shared across every replica, with no Redis or other external dependency. Configure the cluster from the environment:
| Setting | Example | Description |
|---|---|---|
HA_CONSISTENCY |
strong, eventual |
strong backs the shared store with a Raft cluster; eventual (default) keeps today's behaviour. |
HA_HISTORIAN |
true, false |
Enables the protected shared historian. It is valid only with HA_MODE=ap and HA_CONSISTENCY=strong; strong active/passive defaults to enabled. |
HA_RAFT_ID |
1 |
This replica's unique Raft node id (1..N). |
HA_RAFT_MEMBERS |
3 |
Static cluster size (odd 3/5 for a fault-tolerant quorum). |
HA_RAFT_BIND |
tcp://0.0.0.0:6560 |
Local Raft transport bind address. |
HA_RAFT_PEERS |
tcp://server-b:6560,tcp://server-c:6560 |
The other members' Raft transport addresses. |
See docker-compose.yml with the active-passive.env env file for a runnable 3-node Raft cluster (real cross-container active/passive HA). Set HA_RECORD_KEY to one shared base64 32-byte key before starting it. The topology calls UseDistributedHistorian, historizes HighAvailability.Counter and HighAvailability.HistoryEvents, and keeps history writes plus raw, event, and processed HistoryRead continuation points visible across session takeover. The historian deliberately rejects HA_INSECURE, eventual storage, and active/active multi-writer topology because ordered values, deletes, modification chains, events, and atomic batches have no conflict-free merge contract.
For Kubernetes, UseKubernetesRaftConsensus in Opc.Ua.Redundancy.Kubernetes derives the same wiring from the StatefulSet ordinal and headless-Service DNS, with a file WAL on a PersistentVolume; see docs/Kubernetes.md.
Redundancy mode and discovery settings
| Setting | Values | Description |
|---|---|---|
REDUNDANCY_MODE |
none, cold, warm, hot, hotandmirrored, transparent |
Selects the Server.ServerRedundancy.RedundancySupport value. If unset, active/passive defaults to hot and active/active defaults to hotandmirrored. |
HA_NODE_ID |
stable replica id | Used as this replica's ApplicationUri suffix (unless HA_APPLICATION_URI overrides it) and as CurrentServerId for transparent redundancy. |
HA_REPLICA_SET |
stable shared set id | Defaults to opcfoundation-ha-sample. Must be identical on every replica; it is not the per-process HA_NODE_ID. |
HA_REDUNDANT_PEERS |
`applicationUri | applicationName |
peerServerUris |
comma/semicolon-separated application URIs | Legacy shorthand for RedundantServerArray; prefer HA_REDUNDANT_PEERS when clients must resolve peers through FindServers. |
HA_MODE |
ap, aa |
Chooses active/passive shared-store replication or active/active CRDT gossip. |
HA_PEER_DISCOVERY |
static, dns, lds, k8s |
Peer-discovery mechanism for the client-facing RedundantServerSet (FindServers) and, for active/active, the gossip fabric. static (default) uses the configured peers; dns resolves HA_SERVICE_NAME (one record per replica) and updates both dynamically. lds/k8s are wired via UseLdsPeerDiscovery / UseKubernetesPeerDiscovery. Static configuration is the fallback until discovery finds peers. |
HA_SERVICE_NAME |
a DNS name | The headless-service DNS name resolved by HA_PEER_DISCOVERY=dns (defaults to server). |
HA_CONSISTENCY |
strong, eventual |
Selects the shared-store consistency model; strong backs it with a Raft cluster (see Strong consistency with Raft). |
HA_HISTORIAN |
true, false |
Enables the shared historian and durable HistoryRead continuation store. Supported only by strong active/passive topology; defaults to true there and false elsewhere. |
HA_FAST_RECONNECT |
true, false |
Allows token-reuse reconnect for mirrored sessions. The default requires full ActivateSession re-authentication after failover. |
HA_RECORD_KEY |
base64 32-byte key | Shared record-protection key for the distributed topologies. When set, every mirrored record (session secrets, identity tokens, notifications) is encrypted + integrity-protected at rest with AesCbcHmacRecordProtector. Use the same value on every replica; provision from a Kubernetes Secret / KMS in production. |
HA_INSECURE |
true, false |
Explicit, auditable opt-out that runs an isolated demo without record protection or gossip authentication (prints a warning). Secure by default: without HA_RECORD_KEY and without this flag, a distributed topology fails closed at startup. Never set in production. |
HA_BALANCING_URL |
a discovery URL | Enables GetEndpoints load direction (see below). A GetEndpoints request on this virtual/LB discovery URL is answered with the best replica's endpoints; empty (default) disables it. |
HA_APPLICATION_URI |
a URI | Overrides the per-replica ApplicationUri with a shared one. Required for transparent redundancy so every replica presents one logical server identity (CreateSession validates the client serverUri against it). |
HA_SUBJECT_NAME |
a certificate subject | Sets an explicit, stable certificate subject so replicas sharing a PKI store load one ApplicationInstanceCertificate. |
HA_PKI_ROOT |
a filesystem path | Points the certificate stores at a shared directory. Combined with HA_SUBJECT_NAME, replicas share one ApplicationInstanceCertificate (production provisions this from a secret rather than a shared volume). |
The sample also enables RequestServerStateChange with AddRequestServerStateChange(). An administrator client can call the standard method on Server to request Maintenance or NoData behavior and set Server.EstimatedReturnTime; the server updates Server.ServiceLevel into the appropriate OPC UA subrange so clients back off or fail over.
Mode-specific nodes shown by the sample:
none— leaves redundancy metadata in its single-server defaults.cold,warm,hot,hotandmirrored— publishServerUriArrayfromHA_REDUNDANT_PEERS, retainRedundantServerArray, advertiseNTRS, and return the peerApplicationDescriptionvalues fromFindServers.transparent— publishesCurrentServerIdandRedundantServerArrayfor the transparent set.
Server.ServiceLevel is driven by the sub-range-aware provider: leaders report Healthy, warm standby reports Degraded, hot/hot-and-mirrored replicas report Healthy, and cold standby reports NoData. The console prints the selected mode, server id, peer set, and initial service-level subrange at startup.
GetEndpoints load direction
Setting HA_BALANCING_URL registers UseServerLoadDirection(...): every replica publishes its health ServiceLevel, a load weight, and its endpoints to the shared store, and a GetEndpoints request that arrives on the balancing URL is answered with the best replica's endpoints — the active writer in active/passive, or the least-loaded healthy replica in active/active. Plain discovery on a replica's own URL is unaffected, and this complements (never replaces) the standard client-driven Server.ServiceLevel / RedundantServerArray selection. In a real deployment a load balancer / Kubernetes Service fronts the replicas at the balancing URL; the sample sets StrongEligibility when HA_CONSISTENCY=strong so the eligibility keyspaces are linearizable. It requires a shared store across replicas (use HA_CONSISTENCY=strong / the Raft compose). See docs/HighAvailability.md for the design, conformance, and security notes.
Transparent redundancy (single virtual endpoint)
The load direction above is one endpoint model; the other is transparent redundancy, where every replica presents one logical server behind one virtual endpoint. Unlike the non-transparent modes (the client reads RedundantServerArray and selects a replica), a transparent client sees a single endpoint and never chooses a replica — a load balancer routes it and mirrored session state lets it continue across a replica failure.
To present one logical server, all replicas must share:
- One
ApplicationUri(HA_APPLICATION_URI).CreateSessionvalidates the client-suppliedserverUriagainst the server'sApplicationUri, so replicas that disagree would reject sessions established via discovery on a peer. - One
ApplicationInstanceCertificate(HA_SUBJECT_NAME+HA_PKI_ROOTpointing at a shared store). Under SecurityMode None the certificate is not exchanged, but secured deployments must present the same certificate from every replica. - Mirrored session and address-space state (
HA_MODE=aa,REDUNDANCY_MODE=transparent) so a session created on one replica can be resumed on another.
Each replica advertises the virtual endpoint URL (HA_HOST = the load-balancer host). Because that host is a DNS name, the listener binds to all interfaces in the replica's own container while returning opc.tcp://<lb>:62543/... to clients, so discovery and CreateSession echo the single endpoint the client actually uses (ServerBase.FilterByEndpointUrl matches the client host to the advertised base address). A client therefore connects to one URL, and on a replica failure the load balancer routes the reconnect to the survivor, where the mirrored session resumes with a token-reuse reconnect (the full ActivateSession signature re-check still applies).
Transparent/docker-compose.yml runs this end to end: two REDUNDANCY_MODE=transparent replicas sharing one ApplicationUri and one certificate (seeded into a shared PKI volume by the first replica, then reused by the second) behind an nginx TCP load balancer that publishes the single virtual endpoint opc.tcp://localhost:62543/RedundantServer, plus a client that connects only to that endpoint. In production the shared certificate is provisioned from a Kubernetes Secret / KMS to every replica rather than self-generated (see docs/Kubernetes.md), which also removes the first-start certificate race.
Active/passive vs active/active
The sample selects its redundancy topology with the HA_MODE environment variable — ap (active/passive, the default) or aa (active/active) — and references both Opc.Ua.Redundancy.Server and Opc.Ua.Redundancy.Server:
- Active/passive (
HA_MODE=ap) —UseDistributedAddressSpace+UseDistributedSessionswith leader election. WithHA_CONSISTENCY=strong,UseDistributedHistorianadds one protected, leader-write archive with writer epochs and durable portable HistoryRead continuations. One replica is the active writer; standbys hydrate from the shared store and take over on failover. Redundancy is reported asRedundancySupport.Hot, and clients followServer.ServiceLevel/Server.ServerRedundancyto find the active replica. - Active/active (
HA_MODE=aa) —UseActiveActiveRedundancy(which wiresUseReplicatedAddressSpace+UseReplicatedSessionsfrom one set of gossip options; the individual methods remain for advanced setups). Every replica accepts writes and converges without a leader; redundancy is reported asRedundancySupport.HotAndMirrored. Replicas gossip over TCP: setHA_GOSSIP_PORT(default4840; session entries gossip onport + 1) andHA_GOSSIP_PEERS(a comma/semicolon list ofhost:portfor the other replicas' address-space gossip). A session created on one replica can be resumed on another withManagedSessionBuilder.WithTokenReuseFailover()on the client. Distributed Historical Access is not supported in this multi-writer mode and fails closed when requested.
Set HA_PEER_DISCOVERY=dns (with HA_SERVICE_NAME) to discover peers dynamically instead of a static HA_GOSSIP_PEERS list: the sample registers UseDnsPeerDiscovery, which keeps the client-facing RedundantServerSet (FindServers) and the active/active gossip fabric current as replicas scale up and down. Static configuration (AddServerRedundancy / HA_REDUNDANT_PEERS) is the fallback. HA_PEER_DISCOVERY=lds (UseLdsPeerDiscovery) and Kubernetes EndpointSlice discovery (UseKubernetesPeerDiscovery) are also available — see docs/HighAvailability.md → Dynamic peer discovery.
Run two active/passive replicas
Use HA_MODE=ap (or omit it) for the leader-elected active/passive setup. A real pair needs a shared ISharedKeyValueStore; with the default in-memory store this remains a wiring demonstration.
# replica A
$env:HA_NODE_ID = "replica-a"
$env:HA_MODE = "ap"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-b|RedundantServer replica-b|opc.tcp://localhost:62544/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
In a second terminal:
# replica B
$env:HA_NODE_ID = "replica-b"
$env:HA_MODE = "ap"
$env:REDUNDANCY_MODE = "hot"
$env:HA_REDUNDANT_PEERS = "urn:localhost:OPCFoundation:RedundantServer:replica-a|RedundantServer replica-a|opc.tcp://localhost:62543/RedundantServer"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544
The active replica writes Counter, the standby mirrors state through the shared store, and clients use Server.ServiceLevel, Server.ServerRedundancy.RedundancySupport=Hot, and FindServers peer discovery to select or fail over to the active endpoint.
Run two active/active replicas
# replica A
$env:HA_NODE_ID = "replica-a"
$env:HA_MODE = "aa"
$env:HA_GOSSIP_PORT = "4840"
$env:HA_GOSSIP_PEERS = "127.0.0.1:4842"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62543
In a second terminal:
# replica B
$env:HA_NODE_ID = "replica-b"
$env:HA_MODE = "aa"
$env:HA_GOSSIP_PORT = "4842"
$env:HA_GOSSIP_PEERS = "127.0.0.1:4840"
dotnet run --project samples\Redundancy\RedundantServer\RedundantServer.csproj -- --port 62544
Both replicas now accept writes to Counter and converge: a write on either endpoint propagates to the other by gossip, and the per-second increment runs on every replica, with CRDT last-writer-wins resolving the concurrent updates. Unlike the active/passive store-backed setup, active/active needs no shared ISharedKeyValueStore between processes — the gossip transport carries the state.
Docker Compose
One compose file — docker-compose.yml — runs the sample as a 3-replica set; the topology is chosen from the environment (the build context is the repository root). Two extra files cover the transparent and load-direction demos.
# Active/active, eventual consistency (CRDT gossip, no shared store) — the default.
docker compose --env-file samples\Redundancy\RedundantServer\active-active-eventual.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml up --build
# Active/active, strong consistency (3-node RaftCs, shared linearizable store).
docker compose --env-file samples\Redundancy\RedundantServer\active-active-strong.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml up --build
# Active/passive, strong consistency (3-node RaftCs; one active writer, two hot standbys).
docker compose --env-file samples\Redundancy\RedundantServer\active-passive.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml up --build
# GetEndpoints load direction over the Raft cluster (sets HA_BALANCING_URL on every replica).
docker compose -f samples\Redundancy\RedundantServer\LoadDirection\docker-compose.yml up --build
# Transparent redundancy: replicas as ONE logical server behind an nginx load balancer
# on a single virtual endpoint (opc.tcp://localhost:62543/RedundantServer).
docker compose -f samples\Redundancy\RedundantServer\Transparent\docker-compose.yml up --build
Each replica exposes its OPC UA endpoint on the host (opc.tcp://localhost:62543/RedundantServer, :62544, :62545). Set HA_HOST to the reachable hostname (the compose file uses the container/service name) so peers and clients connect across the container network. Three replicas run in every topology (Raft needs a 3-node quorum; active/active eventual works with 2 or 3 — remove server-c for a 2-node run, so the replica count is configurable). The Raft topologies are real cross-container HA deployments whose RaftCs cluster is the shared, linearizable store; active/active eventual converges leaderlessly over CRDT gossip with no shared store.
Scale active/active eventual replicas
Scale/docker-compose.yml has a single server service for active/active eventual (CRDT gossip) runs. Docker Compose can scale it to any replica count because the sample discovers peers from the server DNS alias instead of a fixed HA_GOSSIP_PEERS list:
docker compose -f samples\Redundancy\RedundantServer\Scale\docker-compose.yml up --build --scale server=5
The scale file sets HA_PEER_DISCOVERY=dns and HA_SERVICE_NAME=server; every replica resolves the service alias, removes its own container IP, and seeds gossip with the remaining task IPs on HA_GOSSIP_PORT (session gossip uses the next port). No host ports are published, so clients should run inside the same compose network or attach to it. This dynamic scaling path is gossip-only. Dynamic Raft scaling is intentionally unsupported because Raft needs stable replica identities and an odd quorum; use the Kubernetes StatefulSet deployment with UseKubernetesRaftConsensus for strong consistency.
The same file has a clients profile that runs independent managed clients in their own containers — each opts into WithServerRedundancy() and fails over on its own, so it scales to any count alongside the servers (no shared client store needed):
docker compose -f samples\Redundancy\RedundantServer\Scale\docker-compose.yml --profile clients up --build --scale server=3 --scale client=2
For a coordinated single-active client replica set (exactly one client active, the rest hot/warm/cold standbys), use AddRedundantClientSession + a CAS-capable AddRaftClientSharedStore (a fixed Raft quorum, like the server's active/passive) rather than independent clients — see HighAvailability.md. Each client replica is its own process; there is no in-process replica set.
For a client-centric compose that runs the full HA matrix (server consistency × client mode) from one env-driven file and logs failover / HA / data loss on both sides, use samples/Redundancy/RedundantClient/docker-compose.yml: COMPOSE_PROFILES selects server-eventual (shows data loss) or server-strong (Raft, no data loss) crossed with client-independent or client-coordinated. See the RedundantClient README.
Watch replication and failover
The clients (and demo) profile adds one or more RedundantClient instances — one process per replica; set CLIENT_REPLICAS to scale the number of client containers. To see replication and failover end to end:
# 1. Start the server set (active/passive here) plus a client that prints the replicated Counter.
docker compose --env-file samples\Redundancy\RedundantServer\active-passive.env `
-f samples\Redundancy\RedundantServer\docker-compose.yml --profile clients up --build
# 2. In another shell, stop the active/leader replica and watch the client keep going:
docker compose -f samples\Redundancy\RedundantServer\docker-compose.yml stop server-a
The client logs its reconnect/redirect to a surviving replica across the failover. Bring the replica back with docker compose ... start server-a; the Raft cluster re-admits it. Set CLIENT_REPLICAS=3 to run three independent client processes, each of which fails over on its own (for a coordinated single-active client set, use AddRedundantClientSession over a Raft client store as above). On the strong-consistency Raft topology the HighAvailability.Counter continues across the failover: the sample wires it through the distributed value cache (see Sharing values across replicas), so the promoted replica resumes from the last value the former leader shared.
For the broader design, see HighAvailability.md. For an environment-driven replica-set deployment, see Kubernetes.md.
Sharing values across replicas
OPC 10000-4 §6.6 requires identical NodeIds and address spaces across a redundant set but does not standardize how live process values are shared. Opc.Ua.Redundancy.Server adds an opt-in extension for this: a variable's read/write callbacks can participate in a distributed value cache (IDistributedValueCache) so the last value is cached in the shared store and served — within a freshness bound — from any replica.
UseDistributedAddressSpace registers IDistributedValueCache in dependency injection (backed by the distributed node-state store; a consumer that constructs the store directly can build a DistributedValueCache itself). The sample injects it into HaSampleNodeManager and wires the Counter:
// Opt the Counter's read/write callbacks into the distributed value cache.
counter.EnableDistributedValueParticipation(
valueCache,
maxAge: TimeSpan.FromSeconds(10),
liveRead: _ => new ValueTask<DataValue>(ReadLocalCounter()));
// The active replica writes each new value through to the shared store...
await valueCache.CacheAsync(counter.NodeId,
new DataValue(Variant.From(value), StatusCodes.Good, DateTimeUtc.Now));
// ...and a replica that has just been promoted seeds from the last shared value,
// so the Counter continues instead of restarting.
(_, DataValue cached) = await valueCache.TryGetAsync(counter.NodeId, maxAge);
On the strong-consistency (Raft) topology this makes the Counter genuinely shared: standby replicas serve the active replica's value, and after a failover the promoted replica resumes from the last value the former leader wrote (verified end-to-end). Monitored items keep reading through the normal pipeline, so a client that monitors a standby observes the shared value only through the participating read path — consistent with OPC UA monitored-item semantics.
Value sharing uses the shared store, which is protected: supply HA_RECORD_KEY (secure by default) or set HA_INSECURE=true for an isolated non-historian demo — see the settings table above. The distributed historian and its portable continuation records always require HA_RECORD_KEY; they never accept the no-op protector.
Notes
- Active/active eventual convergence over TCP/UDP gossip. State propagation between active/active replicas requires each CRDT snapshot to be sent as a
Crdt.TransportFrameCodecframe (MessageType.State); the TCP/UDP gossip transportsDecodeon send and would otherwise throwSystem.IO.InvalidDataException: Frame length does not match the encoded body length. The stack'sFramingGossipTransportdecorator handles this framing forUseReplicatedAddressSpace/UseReplicatedSessions, so cross-processCounterand session state converge over real gossip (regression-tested with two realTcpGossipTransports). The in-memory transport is a transparent passthrough, so single-process runs behaved correctly regardless. - Value continuity across failover. The
Counteris shared and continues across failover through the distributed value cache: on the strong-consistency (Raft) topology it rides the linearizable Raft store, and on active/active eventual it rides the address-space CRDT gossip. Monitored items read through the normal pipeline, so a monitored item on a standby observes the shared value only through the participating read path (as documented inDistributedValueParticipation).