Files
Marc Schier 4633aa7a34 Harden HA address-space hydration and replica-consistent NodeIds (#4445)
# Description

Follow-up to #4390 for distributed address-space ownership,
authoritative
hydration, and replica-consistent wire NodeIds. Tracks #4444.

Identity implementation: `a78d59067` (Server policy/lifecycle/adapter
seam) and
`07faf5cfd` (replica identity module, runtime wiring, examples and
proofs).

## Replica identity

- `UseReplicaNodeIdentity` / `ReplicaNodeIdFactory` reserve fixed shared
namespace
indexes before node-manager construction and reuse
`DefaultNodeIdFactory`.
  ApplicationUri and built-in diagnostics/configuration remain local.
- Replicated startup requires an explicit identity contract. Namespace
layout,
  deterministic mode, replica-set identity and writer policy must agree.
Standard factory views, replacement and registration preserve the
policy.
- Active/passive writer-assigned identities are replicated unchanged.
Independent
creation requires stable keys or explicit IDs. Live IDs and retained
tombstones
are reserved before writer allocation; no network/storage call runs in
`New`.
- Protected store descriptors use strong routing, including customized
hybrid
prefixes. Unknown legacy state and empty eventual scans cannot authorize
  automatic adoption. Verified-new hybrid provisioning is explicit.
- Active/active frames compare the descriptor before CRDT merge.
Rejected frames
are not reported as applied; incompatible admission remains at
ServiceLevel 0
  through the standard service-level wiring.
- Shared metadata validation covers type definitions, data types,
references,
permissions, NodeId/ExpandedNodeId/QualifiedName values,
arrays/matrices,
  DataValues and encodeable structures/arguments, without reflection.
- Runtime preparation hydrates retained IDs before replacement routes
publish.
Rebinding keeps the existing store/election or gossip transport/map and
attaches
to the new node-manager sources. Normal node-manager mutations,
including
  AddNodes, reach the cached local-address-space adapter.
- WoT materialization retains the stable configured factory. Samples
expose
genuinely factory-created nodes and a client `--identity` failover
workflow.

## Preserved HA repairs

The published `d99995003` and `79da74f14` fixes remain in place:

- Shared linearizable sequence/lease coordination; no pretend CAS on
CRDT and no
  private counter substituted for shared coordination.
- Strict authoritative record decoding, protected pending reservations,
safe
  snapshot publication and required delta retention.
- Accepted topology state controls authoritative membership; stale
deletion
  cannot remove a newer incarnation.
- Independent subtree attachment tracking and correct disposal of the
final
  snapshot task.
- Values wait for their topology, and parent tombstone horizons prevent
older
  descendant values crossing incarnations.

A new deterministic regression also covers writer hydration preparation:
temporary reconciliation removals must remain inbound application, not
be
recaptured as new shared deletion records.

## Validation

### Review feedback fixes

`3a682155b` addresses the active-node notification, WoT namespace-index
bounds,
and retired-manager XML documentation feedback. The address-space
adapter now
notifies listeners with the registered active node, and both WoT
managers reject
missing/out-of-range namespace indexes before ownership is established.

Targeted feedback validation passes on net10.0 and net48: 8
address-space adapter
cases and 74 WoT cases per framework, plus 17
hydration/gossip/live-client
integration cases on net10.0. The wrong-instance notification and
missing
namespace regressions were reproduced before the fixes.

### Identity implementation evidence

| Selected scope | net10.0 | net48 |
| --- | --- | --- |
| HA repair, identity, metadata, peer admission and real-client
identity/reload fixtures | 215 passed | 215 passed |
| Factory, fluent authoring/import, NodeManagement and address-space
adapter fixtures | 213 passed | 213 passed |
| WoT node-manager fixtures | 68 passed | 68 passed |

- Real UA-TCP clients compare full NodeIds/reference targets across
independent
servers in all four deterministic modes, with different manager/creation
order
and local allocations. Saved IDs work after endpoint failover and
restart.
- Real AddNodes-created IDs remain readable through successive live
manager
replacements in both replication modes, including nodes created after a
reload.
- Two process-level cases start three sample replicas, kill the serving
process,
verify a different serving ServerArray URI, and reuse cached
factory-generated
  IDs for Read and CreateMonitoredItems without remapping or rebrowsing.
- The identity/hydration scenario runs from the actual published win-x64
NativeAOT
  executable in the existing historian companion host.
- The focused writer-recapture regression was observed failing without
its fix,
  then passing. The real reload scenario was additionally repeated while
  diagnosing that race.
- Changed-file formatting/style/analyzers were applied. Existing CA1508,
CA1823,
CA1861 and CA1850 warnings in untouched upstream files were not
suppressed.

The selected checks above do not replace full-solution CI, coverage
gates or
maintainer review, and do not claim that the entire solution or every
native
platform was run. #4444 remains the tracking issue.

## Contract and consistency limits

Fixed identity does not make eventual payload reads linearizable. Hybrid
views
remain non-destructive, use explicit tombstones, and do not infer
empty-store
bootstrap or partition completion from absence. Compacted snapshots
require
authoritative storage. Failed/uncertain reservations require
reconciliation.

There is no automatic namespace-slot allocation, persistent logical-key
mapping
database, live renumbering, legacy-store wipe/adoption, or new hash
algorithm.
Opaque shared structures require a registered codec so their
namespace-bearing
fields can be validated. Application callbacks are still
application-owned.

## Related issues

- Tracks #4444.
- Follow-up to #4390.
- Builds on the merged NodeId factory (#4433), configure/seal (#4442)
and
  lifecycle/import (#4427) integration at `aedd3f738`.

## Checklist

- [ ] I have signed the CLA and read CONTRIBUTING.
- [x] Added targeted regression and integration tests.
- [x] Updated directly related API, HA, migration and sample
documentation.
- [x] Ran scoped formatting and the targeted framework checks listed
above.
- [ ] Full UA.slnx matrix and coverage gates completed.
- [ ] All CI and maintainer review completed.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9dbd9527-d818-47a0-984c-fbb1472d3d6b
2026-09-09 13:26:45 +02:00

11 KiB

Managed client sample

This console sample shows the recommended way to connect an OPC UA client: build a single ManagedSession with WithServerRedundancy(). The same code works whether or not the target server is configured for redundancy, because a client is almost never aware of the server topology until it has connected.

  • Against a redundant server, the managed session reads Server.ServerRedundancy / Server.ServiceLevel, discovers the redundant set from the connected server, and fails over transparently. There is no client-side failover-mode selection and no hand-maintained seed list — the peer set comes from the server.
  • Against a server that is not configured for redundancy, the same session simply runs as a resilient, automatically reconnecting client (RedundancySupport=None).

Run

dotnet run --project samples\Redundancy\RedundantClient\RedundantClient.csproj -- `
  --server opc.tcp://localhost:62543/RedundantServer --autoaccept --nosecurity --duration 00:05:00
Option Default Description
--server, -s opc.tcp://localhost:62543/RedundantServer Discovery URL of any server in the (optionally) redundant set.
--nosecurity off Select endpoints with MessageSecurityMode.None.
--autoaccept off Automatically accept untrusted server certificates (sample only).
--duration, -d 00:02:00 How long to monitor before exiting; 00:00:00 runs until Ctrl+C.
--suite off Run a browse/read/subscribe workload against the redundant ISession.
--history off Open raw, event, and processed HistoryRead continuations, pause for active-server loss, then resume all three through the promoted replica. Requires an independent client and the strong active/passive historian sample.
--identity off Cache factory-created shared NodeIds from Browse, wait for server failover, then Read and create a monitored item with the exact saved IDs. Requires an independent client.
--history-failover-delay 00:00:15 Pause after the first page so an operator or process test can terminate the active server.

The sample connects, logs the server's reported RedundancySupport (or notes that the server is not redundant), subscribes to Server.ServerStatus.CurrentTime, and logs the values together with any transparent connection-state changes (reconnect or failover). To observe failover, lower the active server's service level (for example with the RedundantServer sample's manual failover support) or stop the active server; the managed session reconnects to a healthy peer on its own.

For the historical failover workflow, start the three-node strong server topology with one shared HA_RECORD_KEY, then run:

dotnet run --project samples\Redundancy\RedundantClient\RedundantClient.csproj -- `
  --server opc.tcp://localhost:62543/RedundantServer --autoaccept --nosecurity `
  --history --history-failover-delay 00:00:15

The workflow first writes a distinctive raw-history marker and reads it back through the active replica. When the client prints HISTORY: portable continuations ready, stop the active server. The managed session uses token-reuse takeover, resumes the raw, event, and processed cursors without duplicates or gaps, reads the same marker through the promoted replica, verifies that the promoted writer adds new shared raw and event history, and prints HISTORY HA OK.

See HighAvailability.md for the redundancy design and the RedundantServer sample for the server side.

For the NodeId identity workflow, run the client with --identity against a replica set. When it prints IDENTITY: cached shared NodeIds, terminate its serving server. It verifies the new serving ApplicationUri through ServerArray, reads FactoryAssigned/Value and its NodeId-valued Target with the saved IDs, and uses the saved value ID in CreateMonitoredItems. No post-failover browse or namespace remapping is used. Success is reported as IDENTITY HA OK.

Run with docker compose (one file, the full HA matrix, env-driven)

A single docker-compose.yml in this folder runs the whole matrix — a redundant server set and one or more clients — where server consistency and client mode are chosen independently by an environment variable, and both sides log their failover / HA behavior, including data loss. The adjacent .env sets the default cell (eventual servers + independent clients), so a bare up needs no flags. docker compose auto-loads that .env even when you run from the repository root.

Selection is by Docker compose profiles (the COMPOSE_PROFILES environment variable) — pick one server profile and one client profile:

Profile Topology Scales?
server-eventual Active/active servers, CRDT gossip + DNS discovery (the default) yes (--scale server=N)
server-strong Fixed 3-node active/passive Raft servers — no data loss no (fixed odd quorum)
client-independent N managed clients, each fails over on its own (the default) yes (--scale client=N)
client-coordinated Fixed 3-node client replica set — exactly one active client no (fixed odd quorum)

For client-coordinated, CLIENT_CONSISTENCY (eventual | strong, default eventual) selects the shared-store flavor used to hand the session over to the promoted client. Only the eventual server and the independent clients are --scale-able; the strong server and the coordinated client are Raft quorums that need stable identities, so they are fixed 3-node named replicas. SERVER_REPLICAS (default 3) and CLIENT_REPLICAS (default 2) set the scalable counts.

Run from the repository root:

# DEFAULT: 3 eventual servers + 2 independent clients (failover + data loss demoable out of the box).
docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build

# Scale the eventual server and the independent clients independently:
docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build --scale server=5 --scale client=3

# STRONG server (no data loss) x independent clients:
COMPOSE_PROFILES=server-strong,client-independent \
  docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build

# Any matrix cell, e.g. STRONG server x EVENTUAL coordinated client:
COMPOSE_PROFILES=server-strong,client-coordinated CLIENT_CONSISTENCY=eventual \
  docker compose -f samples/Redundancy/RedundantClient/docker-compose.yml up --build

To trigger a server failover, stop the replica a client is on — for example scale the eventual servers down (… up -d --scale server=1) or docker kill one server container (for server-strong, stop the Raft leader, e.g. … stop server-a). Then watch the client log, which turns the transparent failover and its data impact into explicit lines:

Client log line Meaning
Connection state: Connected -> Reconnecting / Failover The session lost its server and is selecting a healthy replica (independent client).
CONNECTED: session (re)connected to … The session re-established a channel (transparent reconnect/failover).
ACTIVE CLIENT: replica '<id>' is now the active client This coordinated replica was elected leader and (re)established monitoring.
STANDBY: this replica is no longer the active client This coordinated replica lost leadership; a peer client took over.
FAILOVER: now served by replica '<id>' (was '<other>') The HighAvailability.ActiveReplica value changed — a different server is now serving the session.
DATA LOSS: CurrentTime jumped 6.0s (5 update(s) missed during failover) Gap in ServerStatus.CurrentTime: samples were missed while the subscription was re-established on the new server (or while a standby client was being promoted).
DATA LOSS: Counter … across failover (state did not carry over) The replicated Counter regressed or diverged — the value was not preserved (see below).
HA OK: Counter continued <n> -> <n+k> across failover (no data loss) The Counter value was preserved across the failover.

The server log shows the other side of the same events: HA: replica <id> became ACTIVE writer / … became STANDBY on role changes, and a periodic HA: replica <id> ACTIVE, Counter=<n> heartbeat so you can see which replica produces which values.

Whether the Counter shows data loss or continuity depends on the server profile:

  • server-eventual (active/active, eventual consistency, the default). Every replica writes its own independent Counter, so on failover the value does not carry over and the client logs data loss (DATA LOSS: Counter jumped …). This is the scalable topology.
  • server-strong (active/passive, strong Raft consistency). The Counter and its Historical Access archive ride a linearizable Raft store, so a promoted standby continues both live state and portable raw/event/processed cursors. The client logs no data loss (HA OK: Counter continued …). Strong consistency needs a fixed odd quorum (3 nodes), so it is not --scale-able and requires one shared HA_RECORD_KEY.

Client replica set (high availability)

Run the client image in multiple containers so each replica is its own process — a single-process demo cannot coordinate a real deployment. The client mode is chosen with CLIENT_MODE (which the compose sets per profile); two process-per-replica models are supported:

  • Independent managed clients (CLIENT_MODE=independent, the compose default, 2, scalable): every container builds a ManagedSession with WithServerRedundancy() and reconnects to a healthy peer independently. Scale the client image with --scale client=N, or use the clients profile in RedundantServer/Scale/docker-compose.yml.
  • Coordinated single-active replica set (CLIENT_MODE=eventual | strong, the client-coordinated profile, fixed odd quorum, default 3): each container builds a RedundantClientSession over a real Raft cluster among the client replicas (RaftLeaderElection + a RaftSharedKeyValueStore, or a HybridSharedKeyValueStore over CRDT gossip for eventual) — the same building blocks the server uses, mirrored on the client. The processes elect one leader that holds the session and share its protected session secrets, so a follower takes over on leader loss and resumes monitoring. Each replica exposes a transparent RedundantClientSession (ISession). Because the election is Raft-quorum-based in both eventual and strong, a coordinated set needs an odd ≥ 3 quorum (a 2-member set is degenerate); the compose ships a fixed 3-node client-a/b/c set. A coordinated set mirrors session secrets through a networked store, so it requires a record protector: set CLIENT_RECORD_KEY to a shared base64 32-byte key in production, or CLIENT_INSECURE=true for an isolated demo (a well-known, non-secret demo key). See HighAvailability.md.