# Description Follow-up to #4390 for distributed address-space ownership, authoritative hydration, and replica-consistent wire NodeIds. Tracks #4444. Identity implementation: `a78d59067` (Server policy/lifecycle/adapter seam) and `07faf5cfd` (replica identity module, runtime wiring, examples and proofs). ## Replica identity - `UseReplicaNodeIdentity` / `ReplicaNodeIdFactory` reserve fixed shared namespace indexes before node-manager construction and reuse `DefaultNodeIdFactory`. ApplicationUri and built-in diagnostics/configuration remain local. - Replicated startup requires an explicit identity contract. Namespace layout, deterministic mode, replica-set identity and writer policy must agree. Standard factory views, replacement and registration preserve the policy. - Active/passive writer-assigned identities are replicated unchanged. Independent creation requires stable keys or explicit IDs. Live IDs and retained tombstones are reserved before writer allocation; no network/storage call runs in `New`. - Protected store descriptors use strong routing, including customized hybrid prefixes. Unknown legacy state and empty eventual scans cannot authorize automatic adoption. Verified-new hybrid provisioning is explicit. - Active/active frames compare the descriptor before CRDT merge. Rejected frames are not reported as applied; incompatible admission remains at ServiceLevel 0 through the standard service-level wiring. - Shared metadata validation covers type definitions, data types, references, permissions, NodeId/ExpandedNodeId/QualifiedName values, arrays/matrices, DataValues and encodeable structures/arguments, without reflection. - Runtime preparation hydrates retained IDs before replacement routes publish. Rebinding keeps the existing store/election or gossip transport/map and attaches to the new node-manager sources. Normal node-manager mutations, including AddNodes, reach the cached local-address-space adapter. - WoT materialization retains the stable configured factory. Samples expose genuinely factory-created nodes and a client `--identity` failover workflow. ## Preserved HA repairs The published `d99995003` and `79da74f14` fixes remain in place: - Shared linearizable sequence/lease coordination; no pretend CAS on CRDT and no private counter substituted for shared coordination. - Strict authoritative record decoding, protected pending reservations, safe snapshot publication and required delta retention. - Accepted topology state controls authoritative membership; stale deletion cannot remove a newer incarnation. - Independent subtree attachment tracking and correct disposal of the final snapshot task. - Values wait for their topology, and parent tombstone horizons prevent older descendant values crossing incarnations. A new deterministic regression also covers writer hydration preparation: temporary reconciliation removals must remain inbound application, not be recaptured as new shared deletion records. ## Validation ### Review feedback fixes `3a682155b` addresses the active-node notification, WoT namespace-index bounds, and retired-manager XML documentation feedback. The address-space adapter now notifies listeners with the registered active node, and both WoT managers reject missing/out-of-range namespace indexes before ownership is established. Targeted feedback validation passes on net10.0 and net48: 8 address-space adapter cases and 74 WoT cases per framework, plus 17 hydration/gossip/live-client integration cases on net10.0. The wrong-instance notification and missing namespace regressions were reproduced before the fixes. ### Identity implementation evidence | Selected scope | net10.0 | net48 | | --- | --- | --- | | HA repair, identity, metadata, peer admission and real-client identity/reload fixtures | 215 passed | 215 passed | | Factory, fluent authoring/import, NodeManagement and address-space adapter fixtures | 213 passed | 213 passed | | WoT node-manager fixtures | 68 passed | 68 passed | - Real UA-TCP clients compare full NodeIds/reference targets across independent servers in all four deterministic modes, with different manager/creation order and local allocations. Saved IDs work after endpoint failover and restart. - Real AddNodes-created IDs remain readable through successive live manager replacements in both replication modes, including nodes created after a reload. - Two process-level cases start three sample replicas, kill the serving process, verify a different serving ServerArray URI, and reuse cached factory-generated IDs for Read and CreateMonitoredItems without remapping or rebrowsing. - The identity/hydration scenario runs from the actual published win-x64 NativeAOT executable in the existing historian companion host. - The focused writer-recapture regression was observed failing without its fix, then passing. The real reload scenario was additionally repeated while diagnosing that race. - Changed-file formatting/style/analyzers were applied. Existing CA1508, CA1823, CA1861 and CA1850 warnings in untouched upstream files were not suppressed. The selected checks above do not replace full-solution CI, coverage gates or maintainer review, and do not claim that the entire solution or every native platform was run. #4444 remains the tracking issue. ## Contract and consistency limits Fixed identity does not make eventual payload reads linearizable. Hybrid views remain non-destructive, use explicit tombstones, and do not infer empty-store bootstrap or partition completion from absence. Compacted snapshots require authoritative storage. Failed/uncertain reservations require reconciliation. There is no automatic namespace-slot allocation, persistent logical-key mapping database, live renumbering, legacy-store wipe/adoption, or new hash algorithm. Opaque shared structures require a registered codec so their namespace-bearing fields can be validated. Application callbacks are still application-owned. ## Related issues - Tracks #4444. - Follow-up to #4390. - Builds on the merged NodeId factory (#4433), configure/seal (#4442) and lifecycle/import (#4427) integration at `aedd3f738`. ## Checklist - [ ] I have signed the CLA and read CONTRIBUTING. - [x] Added targeted regression and integration tests. - [x] Updated directly related API, HA, migration and sample documentation. - [x] Ran scoped formatting and the targeted framework checks listed above. - [ ] Full UA.slnx matrix and coverage gates completed. - [ ] All CI and maintainer review completed. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9dbd9527-d818-47a0-984c-fbb1472d3d6b
11 KiB
Managed client sample
This console sample shows the recommended way to connect an OPC UA client: build a single ManagedSession with WithServerRedundancy(). The same code works whether or not the target server is configured for redundancy, because a client is almost never aware of the server topology until it has connected.
- Against a redundant server, the managed session reads
Server.ServerRedundancy/Server.ServiceLevel, discovers the redundant set from the connected server, and fails over transparently. There is no client-side failover-mode selection and no hand-maintained seed list — the peer set comes from the server. - Against a server that is not configured for redundancy, the same session simply runs as a resilient, automatically reconnecting client (
RedundancySupport=None).
Run
dotnet run --project samples\Redundancy\RedundantClient\RedundantClient.csproj -- `
--server opc.tcp://localhost:62543/RedundantServer --autoaccept --nosecurity --duration 00:05:00
| Option | Default | Description |
|---|---|---|
--server, -s |
opc.tcp://localhost:62543/RedundantServer |
Discovery URL of any server in the (optionally) redundant set. |
--nosecurity |
off | Select endpoints with MessageSecurityMode.None. |
--autoaccept |
off | Automatically accept untrusted server certificates (sample only). |
--duration, -d |
00:02:00 |
How long to monitor before exiting; 00:00:00 runs until Ctrl+C. |
--suite |
off | Run a browse/read/subscribe workload against the redundant ISession. |
--history |
off | Open raw, event, and processed HistoryRead continuations, pause for active-server loss, then resume all three through the promoted replica. Requires an independent client and the strong active/passive historian sample. |
--identity |
off | Cache factory-created shared NodeIds from Browse, wait for server failover, then Read and create a monitored item with the exact saved IDs. Requires an independent client. |
--history-failover-delay |
00:00:15 |
Pause after the first page so an operator or process test can terminate the active server. |
The sample connects, logs the server's reported RedundancySupport (or notes that the server is not redundant), subscribes to Server.ServerStatus.CurrentTime, and logs the values together with any transparent connection-state changes (reconnect or failover). To observe failover, lower the active server's service level (for example with the RedundantServer sample's manual failover support) or stop the active server; the managed session reconnects to a healthy peer on its own.
For the historical failover workflow, start the three-node strong server topology with one shared HA_RECORD_KEY, then run:
dotnet run --project samples\Redundancy\RedundantClient\RedundantClient.csproj -- `
--server opc.tcp://localhost:62543/RedundantServer --autoaccept --nosecurity `
--history --history-failover-delay 00:00:15
The workflow first writes a distinctive raw-history marker and reads it back through the active replica. When the client prints HISTORY: portable continuations ready, stop the active server. The managed session uses token-reuse takeover, resumes the raw, event, and processed cursors without duplicates or gaps, reads the same marker through the promoted replica, verifies that the promoted writer adds new shared raw and event history, and prints HISTORY HA OK.
See HighAvailability.md for the redundancy design and the RedundantServer sample for the server side.
For the NodeId identity workflow, run the client with --identity against a
replica set. When it prints IDENTITY: cached shared NodeIds, terminate its
serving server. It verifies the new serving ApplicationUri through ServerArray,
reads FactoryAssigned/Value and its NodeId-valued Target with the saved IDs,
and uses the saved value ID in CreateMonitoredItems. No post-failover browse or
namespace remapping is used. Success is reported as IDENTITY HA OK.
Run with docker compose (one file, the full HA matrix, env-driven)
A single docker-compose.yml in this folder runs the whole matrix — a redundant server set and one or more clients — where server consistency and client mode are chosen independently by an environment variable, and both sides log their failover / HA behavior, including data loss. The adjacent .env sets the default cell (eventual servers + independent clients), so a bare up needs no flags. docker compose auto-loads that .env even when you run from the repository root.
Selection is by Docker compose profiles (the COMPOSE_PROFILES environment variable) — pick one server profile and one client profile:
| Profile | Topology | Scales? |
|---|---|---|
server-eventual |
Active/active servers, CRDT gossip + DNS discovery (the default) | yes (--scale server=N) |
server-strong |
Fixed 3-node active/passive Raft servers — no data loss | no (fixed odd quorum) |
client-independent |
N managed clients, each fails over on its own (the default) | yes (--scale client=N) |
client-coordinated |
Fixed 3-node client replica set — exactly one active client | no (fixed odd quorum) |
For client-coordinated, CLIENT_CONSISTENCY (eventual | strong, default eventual) selects the shared-store flavor used to hand the session over to the promoted client. Only the eventual server and the independent clients are --scale-able; the strong server and the coordinated client are Raft quorums that need stable identities, so they are fixed 3-node named replicas. SERVER_REPLICAS (default 3) and CLIENT_REPLICAS (default 2) set the scalable counts.
Run from the repository root:
# DEFAULT: 3 eventual servers + 2 independent clients (failover + data loss demoable out of the box).
docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build
# Scale the eventual server and the independent clients independently:
docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build --scale server=5 --scale client=3
# STRONG server (no data loss) x independent clients:
COMPOSE_PROFILES=server-strong,client-independent \
docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build
# Any matrix cell, e.g. STRONG server x EVENTUAL coordinated client:
COMPOSE_PROFILES=server-strong,client-coordinated CLIENT_CONSISTENCY=eventual \
docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build
To trigger a server failover, stop the replica a client is on — for example scale the eventual servers down (… up -d --scale server=1) or docker kill one server container (for server-strong, stop the Raft leader, e.g. … stop server-a). Then watch the client log, which turns the transparent failover and its data impact into explicit lines:
| Client log line | Meaning |
|---|---|
Connection state: Connected -> Reconnecting / Failover |
The session lost its server and is selecting a healthy replica (independent client). |
CONNECTED: session (re)connected to … |
The session re-established a channel (transparent reconnect/failover). |
ACTIVE CLIENT: replica '<id>' is now the active client |
This coordinated replica was elected leader and (re)established monitoring. |
STANDBY: this replica is no longer the active client |
This coordinated replica lost leadership; a peer client took over. |
FAILOVER: now served by replica '<id>' (was '<other>') |
The HighAvailability.ActiveReplica value changed — a different server is now serving the session. |
DATA LOSS: CurrentTime jumped 6.0s (5 update(s) missed during failover) |
Gap in ServerStatus.CurrentTime: samples were missed while the subscription was re-established on the new server (or while a standby client was being promoted). |
DATA LOSS: Counter … across failover (state did not carry over) |
The replicated Counter regressed or diverged — the value was not preserved (see below). |
HA OK: Counter continued <n> -> <n+k> across failover (no data loss) |
The Counter value was preserved across the failover. |
The server log shows the other side of the same events: HA: replica <id> became ACTIVE writer / … became STANDBY on role changes, and a periodic HA: replica <id> ACTIVE, Counter=<n> heartbeat so you can see which replica produces which values.
Whether the Counter shows data loss or continuity depends on the server profile:
server-eventual(active/active, eventual consistency, the default). Every replica writes its own independentCounter, so on failover the value does not carry over and the client logs data loss (DATA LOSS: Counter jumped …). This is the scalable topology.server-strong(active/passive, strong Raft consistency). TheCounterand its Historical Access archive ride a linearizable Raft store, so a promoted standby continues both live state and portable raw/event/processed cursors. The client logs no data loss (HA OK: Counter continued …). Strong consistency needs a fixed odd quorum (3 nodes), so it is not--scale-able and requires one sharedHA_RECORD_KEY.
Client replica set (high availability)
Run the client image in multiple containers so each replica is its own process — a single-process demo cannot coordinate a real deployment. The client mode is chosen with CLIENT_MODE (which the compose sets per profile); two process-per-replica models are supported:
- Independent managed clients (
CLIENT_MODE=independent, the compose default, 2, scalable): every container builds aManagedSessionwithWithServerRedundancy()and reconnects to a healthy peer independently. Scale the client image with--scale client=N, or use theclientsprofile inRedundantServer/Scale/docker-compose.yml. - Coordinated single-active replica set (
CLIENT_MODE=eventual|strong, theclient-coordinatedprofile, fixed odd quorum, default 3): each container builds aRedundantClientSessionover a real Raft cluster among the client replicas (RaftLeaderElection+ aRaftSharedKeyValueStore, or aHybridSharedKeyValueStoreover CRDT gossip for eventual) — the same building blocks the server uses, mirrored on the client. The processes elect one leader that holds the session and share its protected session secrets, so a follower takes over on leader loss and resumes monitoring. Each replica exposes a transparentRedundantClientSession(ISession). Because the election is Raft-quorum-based in both eventual and strong, a coordinated set needs an odd ≥ 3 quorum (a 2-member set is degenerate); the compose ships a fixed 3-nodeclient-a/b/cset. A coordinated set mirrors session secrets through a networked store, so it requires a record protector: setCLIENT_RECORD_KEYto a shared base64 32-byte key in production, orCLIENT_INSECURE=truefor an isolated demo (a well-known, non-secret demo key). See HighAvailability.md.