mirror of
https://github.com/OPCFoundation/UA-.NETStandard.git
synced 2026-10-21 00:56:28 +08:00
master
23 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 romanett
|
5f6faa60d7 |
Server: fewer allocations per single-value Read (isolation reservations, scope reuse, routing capture without restore) (#4856)
Removes per-request allocations on the server's single-value Read path
with small, internal changes. No wire-format change, no public API
change, no change of the execution model (the receive loop is
untouched). Exactly-once semantics of the isolation leases,
cancellation, timeouts, diagnostics and auditing are unchanged.
## Effect
Allocated bytes per single-value Read (perf harness, `native-read`, 10
clients x 100 in flight, `AsyncCustomNodeManager`, Server GC;
deterministic):
| | 1 core | 4 cores |
|---|---|---|
| master
|
||
|
Marc Schier
|
9bfc780e14 |
Update NuGet dependencies and migrate tests to NUnit 5 (#4840)
# Description Update 55 NuGet dependencies to the latest stable releases available on 2026-10-19, preserving documented compatibility pins and framework-specific version bands. Existing OpenUSD prereleases remain unchanged because no newer releases are available. Migrate existing tests to NUnit 5: await asynchronous assertions, propagate async through helpers, handle nullable predicates, and replace obsolete `Order` attributes with sequencing dependencies. Sequencing dependencies use `AllowFailure = true`; filtering a dependent test also runs prerequisites. Adjust fixture-level parallelism where NUnit dependency scheduling requires it. Keep Roslyn API/runtime pins intact; select newer Immutable/Metadata runtime packages only for test and profiling hosts requiring them. Update the approved SDK pin to 10.0.402 across local, CI, and container builds. Document NUnit 5 conventions. |
||
|
romanett
|
33d6d29e12 |
Server: wake fair queue workers through a LIFO signal, let a looking worker take new requests (#4833)
Request queue workers of the `FairRequestQueue` waited on a bounded `Channel<bool>(1)` for the coalesced wake token. With many cores, its monitor became the server's largest lock contention point, and each wake cost a channel operation, an allocated wait and a thread-pool hand-off. This PR replaces that signal and lets a worker that just finished a request keep looking for the next one for a few microseconds before it parks. The concept is unchanged: - requests are admitted into the fair queue; - `MinRequestThreadCount`/`MaxRequestThreadCount` workers dequeue and run them; - the receive loop is not touched; - weighted fairness, the per-owner and global limits, leases, admission and `BadServerTooBusy`, cancellation with an exactly-once lease release, stop and drain, and parked publish requests behave as before. There are no wire, public API or configuration changes. ## Results Paired A/B against `master` |
||
|
romanett
|
19a97da52e |
Server: confirm unchanged isolation classifications at dispatch without the full lookup (#4787)
Roadmap item R3 of the o6 benchmark gap analysis (master `5b03e9afa` profile). Each request was revalidated twice by the resource-isolation provider: at dequeue, and again before execution. Each revalidation repeated: - the owner table lookup (a `ConditionalWeakTable`); - the session binding lookup (token and session dictionaries, the session's expiry clock); - the classifier; - the owner key checks. Now the classification also returns the evidence it was derived from. The queue confirms that evidence at dispatch and runs the full revalidation only when it no longer holds. No wire, configuration or public API change. Every rejection status code, and the order of the checks relative to cancellation and session close, is unchanged: the confirmation only ever short-circuits to Good, and the full revalidation decides every other outcome. ## Results Paired A/B against master `af601ebca` (= `5b03e9afa` + #4778). The builds alternate per round, with the order rotated each round, on the same machine. The table shows medians of 5 rounds, with the throughput range of the rounds in brackets. The machine was shared with other work; allocated bytes per request are deterministic. - **o6read** uses the native open62541 client workers of the o6-automation benchmark, in the o6 2.0.1 configuration: `AsyncCustomNodeManager`, Server GC, queue = clients × outstanding + 3. - **"4 cores"** pins the server to 4 cores (`--server-affinity 0xF`, `DOTNET_PROCESSOR_COUNT=4`). - **The other rows** use the harness's .NET client, 20 sessions unless noted. | Scenario | req/s | Server CPU µs/req | Alloc B/req | |---|---|---|---| | o6read 10×100, 4 cores | 130,300 → 135,400 (+4 %) [129–135k → 132–140k] | 30.3 → 28.6 (**−6 %**) | 4,706 → 4,709 | | o6read 10×100 encrypted, 4 cores | 116,300 → 122,500 (**+5 %**) [113–119k → 118–126k] | 33.4 → 31.5 (**−6 %**) | 4,747 → 4,756 | | o6read 10×100, 24 cores | 344,500 → 347,400 (+1 %) [328–350k → 331–362k] | 53.6 → 52.2 (−3 %) | 4,710 → 4,716 | | o6read 10×1000, 24 cores | 310,800 → 307,300 (−1 %) [297–332k → 307–323k] | 55.6 → 57.5 (+4 %, noise) | 4,719 → 4,730 | | read --nodes 1, 20 sessions | 189,500 → 198,900 (+5 %) | 49.7 → 48.2 (−3 %) | 4,624 → 4,623 | | read --nodes 1, 200 sessions | 174,500 → 179,100 (+3 %) | 50.2 → 44.7 (**−11 %**) | 4,510 → 4,521 | | browse | 36,490 → 35,750 (−2 %) | 179 → 175 (−3 %) | 39,004 → 39,016 | | sub (5 × 100 items, 100 ms) | 94,020 → 93,180 notif/s | 8.2 → 9.2 (rounds 4.5–12.7) | 145 → 145 | **Where it helps:** CPU per request falls by 6 % on 4 cores, plain and encrypted, and by 3–11 % in the read scenarios with the .NET client. **Where it does not:** on 24 cores, CPU per request is within the spread of the rounds. There the cost is dominated by workers of one channel waiting for each other (the request hop and the send-order lock), which this PR does not change. ### Profile cpu trace, 24 cores, 10×100, quiet runs at 433–439k req/s. The trace was taken with this PR and #4788 (R7) together; the frames below belong to this PR only. | Frame | master | this PR | |---|---|---| | `FairRequestQueue.GetRevalidationStatus`, inclusive | 15.4 % | 14.0 % | | of which the provider and session lookups | 15.4 % | 3.9 % (`IsAdmissionCurrent`, mostly the session's expiry clock) | | `SessionManager.TryGetSessionContext`, inclusive | 14.6 % | 0.2 % (admission only) | The lookups are gone, but the dequeue-time check still accounts for about 10 % of the samples as its own time. That is the first touch of the request (entry, request object, channel context) on the worker thread, right after the receive thread wrote it on another core. On master the same cache misses showed up inside `TryGetSessionContext`, which is why the profile share (16.5 %) overstated what this change could save. Removing the request hop (separate work) is what removes those misses. ## Changes ### Evidence with the classification - **Internal interface.** `DefaultServerResourceIsolationProvider` implements a new internal interface, `IResourceIsolationAdmissionEvidence`. Its `Classify` also returns the evidence: the issued owner identity, which holds the channel snapshot and the session binding. - **Kept on the entry.** `FairRequestQueue` keeps the evidence, an internal `ResourceIsolationAdmissionEvidence` record that only the issuing provider interprets, on the queue entry. - **Confirmation.** Before the full revalidation, the queue asks the provider whether the evidence still holds. This applies to both checks, at dequeue and before execution. The provider: - compares the channel snapshot with the request's channel context, as before, so mutated channel evidence is still detected; - asks the binding's liveness check. Only a confirmed Good is short-circuited. - **No evidence, so every revalidation runs in full:** - a classifier is configured, because it may map the same inputs differently at any time; - the request has a token but no binding, because an activation may bind it while the request waits; - the binding comes from another binding provider, which has no liveness check. ### Binding liveness (`SessionManager`) A `SessionBindingContext` issued by `SessionManager` carries an internal liveness check. It runs the checks of `TryGetSessionContext` that can change while the binding and the request's channel context stay the same, without the token and session dictionary lookups: - **The dispatch binding is still this binding.** A removal, a new activation, an identity change, a transfer and an activation in commit all replace it. A commit always creates a new instance. - **The session's channel certificate was not replaced.** `SetRestoredSessionTransferSecurityState` can replace it without a new binding, so the certificate has a version. - **The manager is not shutting down.** Shutdown empties the token index without replacing the bindings. - **The session itself** is activated, not closing or claimed, not expired (the clock-based `HasExpired`, including expiry before the monitor closes the session), and still valid for the channel. These checks run in the same order as in `TryGetSessionContext`. A replaced binding releases its session, so owners that still reference the binding do not keep a closed session alive. ## Testing - **`RuntimeResourceIsolationTests` (new):** - the evidence confirms only an unchanged binding and channel, and a mutated certificate ends it without consulting the session; - it is withheld with a classifier, for a binding without liveness, and for a token without a binding; - a disposed provider throws; - the check allocates nothing. - **`SessionBindingTests` (new):** - the liveness check agrees with a fresh `TryGetSessionContext` through re-activation, transfer and close, an identity change, expiry before the monitor closes the session, an activation in commit, a session that starts closing, and shutdown; - replacing the transfer-security certificate with `SetRestoredSessionTransferSecurityState` keeps the binding but ends the liveness check (fails without the certificate version); - a race of 1000 transfers and a close against 4 readers never confirms a replaced binding, or any binding after the close. - **`FairRequestQueueTests` (new):** - current evidence replaces both full revalidations; - stale evidence at dequeue, or before execution, falls back to the full revalidation and its status (`BadSessionIdInvalid`, `BadServerTooBusy`), and the request does not run. - **Local, net10.0**, with both PRs applied, all 0 failed: - Opc.Ua.Core.Tests (Stack.Server, Stack.Tcp, Stack.Transport): 2753 passed; - Opc.Ua.Server.Tests: 8750 passed; - Opc.Ua.Sessions.Tests: 857 passed (379 skipped as on master); - Opc.Ua.Redundancy.Server.Tests: 884 passed; Opc.Ua.Redundancy.Client.Tests: 134 passed; - Opc.Ua.Stress.Tests Contract + Integration: 45 passed. net48 builds. `dotnet format whitespace --verify-no-changes` passes on the changed files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Marc Schier
|
d764161a53 |
Extract core/types fixes for schemas, replay, and source generation (#4801)
# Description Extract the reusable core/types layer from #4462 without including UaLens or the other planned extraction layers. - Preserve null versus empty in `ByteString.Copy`, and provide older-framework path, random-byte, and SHA-256 convenience APIs. - Encode abstract numeric structure fields (`Number`, `Integer`, and `UInteger`) as Variants in JSON, XSD, and BSD schemas, with regression coverage and schema documentation. - Add opt-in complete-file validation for finalized PCAP replay while retaining the existing tolerant reader default. - Keep placeholder references on generated type templates, not runtime instances, while preserving ordinary external references. **Initial extraction:** commit `8881e096fb5e4eaa36f3c13f8caa57733373bd19` preserves the assigned changes from source `390c4a7a51b71da40caec2d22ef74518bbe670ce`, based on pinned `master` commit `5b03e9afa492cdb36049ee52ae844f630f89d8db`: 20 exact full-file extractions and only the generic replay paragraph in `docs/Diagnostics.md`. The UaLens capture-window paragraph remains with the source PR. No UaLens projects, UI, native-rendering paths, dependencies, or workflows are included. **Review follow-up:** `814c0f2dc6056ea5eb898611565b3e96cfa65104` fixes direct-placeholder references that bypassed filtering through `TargetPath` and emitted an undefined `forInstance` guard in instance-only factories. Filtering now follows target resolution. Compilation/reference-shape regressions cover ObjectType/VariableType, both placeholder rules, and direct-child/external targets; `docs/NodeManagers.md` documents the behavior. The review thread was resolved after publication and verification. **CI follow-up:** `93ec3928bd5a9cb2ea86110085e79131db17d443` addresses both causes found in [CI run 38700770996](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/38700770996): - `MODELGEN012` treated an identical payload-less transitive dependency declaration as a competing model producer after the SDK Programs model appeared in newer `master`. Matching closure rows no longer emit that diagnostic. Producer ordering is unchanged; actual producers, differing declarations, and malformed payload handling retain their existing diagnostics. New regression cases exercise both reference orders, and `docs/ModelDependencies.md` records the distinction. - The queue-fairness test sampled OS scheduling and failed on macOS. This reuses the exact one-file correction from `84d2c34d3902bfe700a87f6c30708b435500878f` in #4765: hold the existing heavy-owner publication hook, prove concurrent work-conserving progress, then assert exact recovered 1:3 service, all 960 unique requests, and complete queue/token/owner/lease cleanup. No production scheduler or fairness bound was changed. The PR now changes 26 paths. Its pinned extraction ancestry is preserved: no rebase, force push, or merge of newer `master` was performed. This remains the bottom of native stack 4807, ordered 4801 -> 4802 -> 4803 -> 4804 -> 4806. ### Current CI-repair validation Windows, existing .NET SDK **10.0.401**, **Release**, with analyzers enabled. CI tested merge `9a2d252793da12aeb99ed379dfc9b03e89c912b7` (master `d40843c23a5fe8116ffe380cf937d88759f5218b` plus PR head `814c0f2dc6056ea5eb898611565b3e96cfa65104`), which includes newer tests absent from the pinned extraction branch. That exact merge was reproduced in a disposable detached worktree, then the proposed patch was applied. Git patch-id verified that the tested patch and committed branch patch were identical (`b7ed3ba2b61d1862825dd33d93c50c64d3a759ea`). Before the fix, all four failing client-accessor cases reproduced `MODELGEN012`, and both new matching-reexport cases failed. After the fix: | Context / scope | Framework | Passed | Failed | Skipped | | --- | --- | ---: | ---: | ---: | | Exact CI merge + patch: client-accessor and dependency fixtures | net10.0 | 32 | 0 | 0 | | Same fixtures with CI coverage instrumentation | net10.0 | 32 | 0 | 0 | | Exact CI merge + patch: entire SourceGeneration.Tests project | net10.0 | 295 | 0 | 0 | | Exact CI merge + patch: entire SourceGeneration.Tests project | net48 | 295 | 0 | 0 | | Preserved PR branch: dependency and placeholder fixtures | net10.0 | 30 | 0 | 0 | | Preserved PR branch: dependency and placeholder fixtures | net48 | 30 | 0 | 0 | | Preserved PR branch: entire FairRequestQueueTests fixture, with coverage | net10.0 | 93 | 0 | 0 | | Preserved PR branch: entire FairRequestQueueTests fixture | net48 | 93 | 0 | 0 | | Exact CI merge + patch: entire FairRequestQueueTests fixture, with coverage | net10.0 | 93 | 0 | 0 | All nine commands exited **0**. The repository TRX verdict helpers verified **993 passing test executions, 0 failures, and 0 fixture setup/teardown failures** across these overlapping runs. No build/analyzer warnings were reported. Normal restore resolved missing assets/target entries in the new checkout and when switching TFMs; no dependency declaration or SDK pin was changed. Exact successful test argument sets follow. Variables abbreviate the executable, project paths, and results directory only; output was also captured to separate local logs. ```powershell $env:DOTNET_ROOT = 'D:\git\UA-.NETStandard_master\TestResults\ualens-528-deep-repro\sdk' $env:PATH = "$env:DOTNET_ROOT;$env:PATH" $dotnet = "$env:DOTNET_ROOT\dotnet.exe" $generator = 'tests\Opc.Ua.SourceGeneration.Tests\Opc.Ua.SourceGeneration.Tests.csproj' $coreTests = 'tests\Opc.Ua.Core.Tests\Opc.Ua.Core.Tests.csproj' $results = 'C:\Users\mschier\.copilot\session-state\8f497e89-7fc2-4f48-880b-4803d97c1c12\files\ci-validation' # In the disposable exact-CI-merge checkout, with the identical proposed patch applied: Set-Location 'D:\Temp\ua-ci4801-8f497e89' # Passed: 32, Failed: 0, Skipped: 0 & $dotnet test $generator --no-restore --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~ClientAccessorGeneratorTests|FullyQualifiedName~ModelDependencyScannerTests' --logger 'trx;LogFileName=ci-metadata-green-net10.trx' --results-directory $results --verbosity minimal # Passed: 295, Failed: 0, Skipped: 0; reuses the analyzer-enabled build above & $dotnet test $generator --no-build --no-restore --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 --logger 'trx;LogFileName=ci-sourcegeneration-merge-net10.trx' --results-directory $results --verbosity quiet # Passed: 295, Failed: 0, Skipped: 0 & $dotnet test $generator --configuration Release --framework net48 -p:CustomTestTarget=net48 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --logger 'trx;LogFileName=ci-sourcegeneration-merge-net48.trx' --results-directory $results --verbosity minimal # Passed: 93, Failed: 0, Skipped: 0 & $dotnet test $coreTests --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true -p:CollectCoverage=true --collect 'XPlat Code Coverage' --settings 'tests\coverlet.runsettings.xml' --filter 'FullyQualifiedName~FairRequestQueueTests' --logger 'trx;LogFileName=ci-fairqueue-merge-net10.trx' --results-directory $results --verbosity minimal # Passed: 32, Failed: 0, Skipped: 0 & $dotnet test $generator --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true -p:CollectCoverage=true --collect 'XPlat Code Coverage' --settings 'tests\coverlet.runsettings.xml' --filter 'FullyQualifiedName~ClientAccessorGeneratorTests|FullyQualifiedName~ModelDependencyScannerTests' --logger 'trx;LogFileName=ci-metadata-coverage-net10.trx' --results-directory $results --verbosity minimal # In the preserved PR branch worktree: Set-Location 'D:\git\copilot-worktrees\UA-wot-d3-final\marcschier-congenial-umbrella' # Passed: 30, Failed: 0, Skipped: 0 & $dotnet test $generator --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~ModelDependencyScannerTests|FullyQualifiedName~PlaceholderReferenceGeneratorTests' --logger 'trx;LogFileName=ci-sourcegeneration-branch-net10.trx' --results-directory $results --verbosity minimal # Passed: 93, Failed: 0, Skipped: 0 & $dotnet test $coreTests --no-restore --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true -p:CollectCoverage=true --collect 'XPlat Code Coverage' --settings 'tests\coverlet.runsettings.xml' --filter 'FullyQualifiedName~FairRequestQueueTests' --logger 'trx;LogFileName=ci-fairqueue-branch-net10.trx' --results-directory $results --verbosity minimal # Passed: 93, Failed: 0, Skipped: 0 & $dotnet test $coreTests --configuration Release --framework net48 -p:CustomTestTarget=net48 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~FairRequestQueueTests' --logger 'trx;LogFileName=ci-fairqueue-branch-net48.trx' --results-directory $results --verbosity minimal # Passed: 30, Failed: 0, Skipped: 0 & $dotnet test $generator --configuration Release --framework net48 -p:CustomTestTarget=net48 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~ModelDependencyScannerTests|FullyQualifiedName~PlaceholderReferenceGeneratorTests' --logger 'trx;LogFileName=ci-sourcegeneration-branch-net48.trx' --results-directory $results --verbosity minimal ``` `git diff --cached --check` passed. The queue-test blob was verified byte-for-byte against the existing published correction. The upstream branch was verified as `93ec3928bd5a9cb2ea86110085e79131db17d443` after push. **Remote CI is pending, not claimed green:** [new CI run 38712127138](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/38712127138). Cross-platform confirmation, CodeQL, coverage gates, and the complete `UA.slnx` suites on this new head are not established by these local checks. The source PR's checks are not claimed as evidence for this branch. ### Earlier validation The original extraction had 1,407 passing scoped test executions across net10.0/net48. The placeholder-review follow-up had six passing focused cases and 269 passing tests in the full source-generator project on each framework, with analyzer-enabled builds. Those are historical results for their respective commits, not fresh full-suite evidence for the CI-repair head. Earlier command details remain in this PR's edit history. ## Related Issues - Related to #4461 (existing tracking issue; not auto-closing). - Extracted from #4462 (source PR; remains open and is not auto-closing). - Reuses the bounded queue-test correction from #4765; does not incorporate that PR's other changes. _If this is a relatively large or complex change, a design must have been discussed in the related tracking issue and signed off (which becomes the Architectural Decision Record (ADR))._ This began as an exact extraction of existing behavior. The subsequent commits address the reported placeholder-generation defect and observed CI failures; they are not a new design or a claim of design sign-off. ## Checklist _Put an `x` in the boxes that apply. You can complete these step by step after opening the PR._ - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [ ] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [x] I have addressed **all** PR feedback received. |
||
|
romanett
|
6cfb8cf196 |
Server: remove per-request context copies and lock hot spots from the request path (−16 % allocation per Read) (#4767)
## Description Five small changes to the per-request server path, found in the CPU and allocation profiles of the o6 read benchmark (`AsyncCustomNodeManager`, Server GC, 10 clients × 100 Reads in flight) on master |
||
|
romanett
|
40288fa7d8 |
Tests: weighted FairRequestQueue test asserts the median of three windows (#4763)
# Description `FairRequestQueueTests.WeightedServiceHoldsUnderConcurrentWorkersAsync` failed once on GitHub Actions macOS net10.0 (#4753, run 38349863236): light=124 heavy=276, ratio 2.23 against an expected 2.5-3.5. It passed on every other CI run and 30/30 locally. **Root cause.** `TrySelect` dequeues an owner's turn token from the lock-free ready queue. On the last token of a turn, `TakeTurnToken` CASes the turn and calls `PushTurn` for the next one. A worker preempted between those two steps leaves the owner with no queued token while it is stalled. The work-conserving dispatch serves the other owner in the meantime, and that share is never made up. Delay injection confirms this: sleeping 1 ms on every 7th turn publish collapses 1:3 to roughly 1:1 (light=199 heavy=201). Publishing a turn ahead of time would only cover stalls shorter than one round (microseconds), while real preemption lasts milliseconds. Making up the lost share (deficit or virtual time) would mean redesigning the scheduler. So this PR documents the bounded skew and makes the test robust instead. **Changes** - `FairRequestQueue` remarks: the weighting holds over time, not in every window, and a preempted worker can cost an owner its share for the length of the stall. - `WeightedServiceHoldsUnderConcurrentWorkersAsync` now measures 3 independent windows through `MeasureWeightedServiceAsync` and asserts that the **median** heavy/light ratio is in 2.5-3.5. The failure message lists all three windows. - Each window uses perOwner = 480 and measured = 560, with queue limits of `2 * perOwner`. That stays under the TestProvider's 1000-request cap and keeps both owners backlogged. - Each window still checks that both owners stay backlogged, then drains the queue and checks that everything is back to zero. The median tolerates one skewed window and still fails when the weighting is lost systematically. With stalls injected into one window only, the median is 3.0 and the test passes (e.g. `light=238 heavy=322; light=140 heavy=420; light=140 heavy=420`). With stalls injected into every window, it still fails. **Validation:** the `FairRequestQueueTests` fixture passes 93/93 on net10.0 Release. The weighted test passed 30/30 runs pinned to 2 cores (`start /affinity 3`). `dotnet format whitespace --verify-no-changes` is clean, and the build has 0 warnings. ## Related Issues - Flaky test seen on #4753 after #4750 (FairRequestQueue without a queue-wide lock) ## Checklist - [x] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [x] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [ ] I have addressed **all** PR feedback received. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
romanett
|
a14b23e7dd |
Server: cheaper symmetric crypto path, FairRequestQueue without a queue-wide lock (#4750)
# Server: cheaper symmetric crypto path, FairRequestQueue without a queue-wide lock Roadmap items E1 and 5b of the o6 benchmark gap analysis, one commit each. - **E1:** Windows one-shot `Aes.EncryptCbc`/`DecryptCbc` import the key (`BCryptImportKey`) on every message, although #4720 caches one `Aes` per channel token. The AES-GCM and ChaCha20-Poly1305 policies also created a new keyed instance per message. - **5b:** with `AsyncCustomNodeManager`, which is what the o6 benchmark's 2.0.1 worker uses, the `FairRequestQueue` gate became the largest waiting point. `TryEnqueue`, `TrySelect` and `TryDequeue` spun on one `Lock` for every request. No wire, configuration or public API changes. ## Results These are paired A/B runs against `origin/master`, five rounds per scenario. The two harness builds alternate per round, starting side swapped every round, on the same shared machine (Ryzen AI 9 HX 370, 12 cores / 24 threads, Windows 11). The tables show medians, with ranges in brackets where they matter. The server is configured like the o6 2.0.1 worker: `AsyncCustomNodeManager`, Server GC, queue sized clients × outstanding + 3. o6read uses the benchmark's native open62541 client workers; `read` uses the harness's .NET client. On this machine throughput moves between rounds by up to 2×, so CPU and bytes per request are the reliable signals. ### E1: symmetric crypto (measured on |
||
|
Marc Schier
|
84d6387a4f |
Make request cancellation own exactly one terminal state (#4744)
# Description `RequestLifetime` now publishes the winning cancellation status and terminal ownership in a single atomic state, so concurrent client Cancel, Session close and timeout can never overwrite each other's status or double-own the terminal transition. External token registrations are owned by the lifetime and released on completion, and disposal is deferred until running cancellation callbacks have finished (no use-after-dispose of the CancellationTokenSource). Adds regression tests: 16384-iteration race tests asserting status/owner/callback outcomes with a finite 2 min watchdog (`RequestLifetimeTests`, `RequestManagerTests`) and `WaitAsync` cancellation tests (`TaskWaitAsyncTests`; the polyfill change is only 2 XML doc lines, no runtime change). Files: `RequestLifetime.cs`, `System.Threading.Tasks.cs` (docs), and the three test files. `StandardServer.cs` is intentionally untouched (already on master). ## Related Issues - None. ## Checklist - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [x] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [ ] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [ ] I have addressed **all** PR feedback received. Validation actually run (net10.0 and net48, all passed): Types.Tests `TaskWaitAsync` 23/23; Core.Tests `RequestLifetime` 16/16; Server.Tests `RequestManager` 56/56; `dotnet format whitespace --verify-no-changes` on owned files. Full UA.slnx not run. |
||
|
romanett
|
25ebf5a4d8 |
Server: remove the server-wide locks from the request pipeline, bounded transport buffer pool (#4742)
# Server: remove the server-wide locks from the request pipeline,
bounded transport buffer pool
Quick wins #5 and #6 of the o6 benchmark gap analysis. Under 10 channels
× 100 outstanding single-value Reads on 24 cores, about 59 % of the
server's on-CPU time spun in `Monitor`/`Lock` slow paths, and roughly 20
of the ~30 lock acquisitions per Read were on server-wide locks in the
request pipeline. Separately, on hosts with one or two processors every
response allocated a fresh 64 KiB chunk, because
`ArrayPool<byte>.Shared` runs out with about a thousand responses in
flight.
No wire or configuration changes. The only public API change is an
additive `RequestLifetime` constructor. The admission *sizing*
(`MaxQueuedRequestCount` → shared capacity and reserved floors) is
untouched; this PR only changes how the accounting is locked.
## Results
Paired A/B against `origin/master` (
|
||
|
romanett
|
f5062be349 |
Complete each pooled service response at most once (#4729) (#4736)
## Proposed changes Fixes the cross-talk part of #4729 on master. `ServiceResponsePooledValueTaskSource` comes from one pool shared by the whole process, and `ManualResetValueTaskSourceCore.SetResult` never checks the version token. A second completion of an `EndpointIncomingRequest` therefore either threw `InvalidOperationException` or, once the source had been read and recycled, answered whichever request rented it next. That request could belong to another server instance in the same process. master's request queue (Channel + `FairRequestQueue`) already completes each request exactly once. But `ServerBase.ScheduleIncomingRequest` and `ProcessRequestAsync` are virtual, so a single override that completes and then throws (the `ProcessAsync` catch completes again) was enough to corrupt unrelated requests. - Each rental of the pooled source now gets a lease number, which is never reused for a later rental of the same instance. - `EndpointIncomingRequest` captures the lease when it is constructed and completes through `TrySetResult(lease, response)`. That is a single CAS on the pending lease, so only the first completion of the current rental is accepted. Late or stale completions are discarded and can never reach the next owner. - `SetResult`/`SetException`, which had no callers outside this path, are replaced by `TrySetResult`. The master378 backport #4738 also fixes the double completions in the old `RequestQueue` itself (snapshot in `Dispose`, and enqueue followed by a throw). ## Tests - `ServiceResponsePooledValueTaskSourceTests`: a second completion of the same rental is ignored, and a late completion after the source is recycled does not complete the next rental. Both use a private pool, so the result is deterministic. - `EndpointIncomingRequestTests.ProcessAsync_ScheduleCompletesThenThrows_KeepsFirstCompletionAsync`: a scheduler that completes the request and then throws. Without the fix it fails with `InvalidOperationException`. - `Opc.Ua.Core.Tests` Stack.Server + Stack.Tcp (net10.0): 1871 passed, 0 failed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
romanett
|
28f1401324 |
Server: cut request-path lock contention and allocation; add load harness (#4707)
Profiling the reference server under load showed most request-path CPU spent spinning on contended locks, plus several avoidable allocations per request or per node. This PR removes them, switches the reference server to Server GC, and adds the load harness used to measure it. ## Results Paired A/B against `master` (3 alternating rounds, medians, 20 sessions, workstation GC on both sides, 24-core machine): | Scenario | Throughput | p50 | p90 | Server CPU / op | Server alloc / op | Lock contention | |---|---|---|---|---|---|---| | Read, 1 value | **+45%** (94k → 137k req/s) | −20% | −46% | −37% | −39% | −94% | | Read, 100 values | **+40%** | −26% | −17% | −24% | −25% | +130% ¹ | | Read, 100 values, SignAndEncrypt | **+48%** | −36% | −20% | −30% | −26% | +106% ¹ | | Browse (Server object) | **+33%** | −31% | −12% | −13% | −14% | −40% | | Write, 100 values | +13% | −12% | −4% | −10% | −19% | −23% | | Data-change subscriptions | ±0 ² | ≈ | ≈ | −14% | −17% | −29% | ¹ The faster server spends more time in the remaining per-node locks. ² Throughput is bounded by the writer rate in this scenario. Separately, Server GC instead of workstation GC doubles Read-100 throughput (21k → 44k req/s) and cuts p90 from 2.1 ms to 0.6 ms, because workstation GC paused the server for about half the wall time under load. The working set grows from ~210 MB to ~430 MB. ## Changes - **`ObjectPool<T>`**: lock-free slot array. `ConcurrentBag.Count` took every per-thread lock on each `Return` of the pooled response value-task source, once per request. - **`FairRequestQueue`**: - drops redundant gate entries: the fail-fast pre-checks and the stopped check read volatile fields and are re-checked under the gate; the owner count is folded into selection; - coalesces wake pulses with a pending flag, so the bounded signal channel's lock is only taken when no token is outstanding. Wake-count semantics in the tests are unchanged. - **`DefaultServerResourceIsolationProvider`**: - compares peer addresses without `GetAddressBytes()` allocations; - builds the unsecured verification context only when a classifier is configured. - **TCP listener / channel**: caches the certificate DER copies per channel, keyed on the certificate instance. `X509Certificate2.RawData` returns a fresh copy on every access, and the listener read it for every request. - **`EndpointBase`**: resolves the request `ActivitySource` once instead of `Assembly.GetCallingAssembly()` per request. - **Node managers**: - one `ServerSystemContext` copy per request (cached on the `OperationContext`) for per-node callbacks: permission metadata, node metadata, view membership, method lookup; - removes a discarded copy in `GetReferenceDescriptionAsync`; - per-request operation caches are plain `Dictionary` instead of a `ConcurrentDictionary` with 24 lock objects; - the validation cache is built in one pass; - fixes zero-capacity error lists. - **`ResourceManager`**: translation lookups read an immutable snapshot that is rebuilt after writes. Browse serialised on the global lock for every display name. - **`MonitoredItem`**: shares the default `DataChangeFilter` instead of allocating one per sampled value. - **Reference server**: `ServerGarbageCollection=true`; new "Garbage collector mode" section in `docs/ServerScalability.md`. - **`perf/`**: `ServerLoadHarness` (out-of-process server plus load driver, server-side CPU/GC/alloc/contention deltas, optional `dotnet-trace` capture), `TraceAnalyzer` (TraceEvent-based CPU/allocation/contention aggregation), and paired A/B scripts. Not part of `UA.slnx`. ### Behaviour notes for custom node managers - The `IDictionary<NodeId, NodeState>` operation cache handed to overrides is now a `Dictionary`. It was a `ConcurrentDictionary`, but only one request flow uses it. - Within one request, per-node callbacks of a node manager share one `ServerSystemContext` copy. ### Later commits on this PR - **Permission checks without `NodeMetadata`** (`9068684c3`): the role-permission and access-restriction validators run on an internal `PermissionMetadata` struct; `AsyncCustomNodeManager` fills it on the stack from the same attributes and namespace defaults. Browse's reference filter reads 3 attributes per target instead of 13. Read 100 values: 120.6 → 95.7 KB/request, ~21k → 26k req/s; Browse 67.6 → 51.8 KB, 26.7k → 29.8k req/s. - **Read-only channel evidence** (`936b28d76`, `dc638ab9f`): `SecureChannelContext.ClientChannelCertificate`, `ServerChannelCertificate` and `ChannelThumbprint` are `ByteString`, so the cached per-channel certificate buffer is shared without a copy and cannot be altered by handlers or classifiers. **Public API change**, see the new section in `docs/MigrationGuide.md`. The isolation provider's revalidation snapshot still copies. - **Session lookup** (`936b28d76`): `SessionManager.TryGetSessionContext` takes the candidate snapshot without the bindings lock; the validating check under the lock is unchanged (2 instead of 4 global lock acquisitions per request). - Unit tests for `ObjectPool`, the per-request system context cache, the translation snapshot, `SecureChannelContext` null semantics and a stopped `FairRequestQueue`; `perf/` follows the repository conventions. ## Testing - `Opc.Ua.Server.Tests` (net10.0): 8386 passed before the `ByteString` change. After it, 13 secure-session and isolation tests failed; both causes are fixed in `dc638ab9f` and those tests (25 with their variants) pass. A full rerun of the Server, Sessions, Core, Redundancy, LDS and GDS suites on the final commit is in progress. - Full solution builds for net10.0 and net48; `dotnet format whitespace` clean. ### o6-automation opcua-benchmarks (sampling, deep pipelines) Replicated both tests of the [o6-automation benchmark](https://github.com/o6-automation/opcua-benchmarks) in the harness (`--scenario counters`, and `read` with 10 clients × N outstanding), paired against `origin/master`: | Test | master | this PR | |---|---|---| | Subscription sampling, 100 Hz, every item ≥ 99 % of samples, 1 item | FAIL (75–94 %) | PASS (100 %) | | same, 1024 items | FAIL (64–73 %) | PASS (99.9 %) | | same, capacity | — | between 4096 and 6144 items (~410–520k samples/s) | | Read, 10 clients × 100 outstanding, None | stalls (0–1.7k req/s) | 121–131k req/s | | Read, 10 clients × 100 outstanding, SignAndEncrypt | stalls (0 req/s) | 89–91k req/s | | Read, 10 clients × 1000 outstanding | channels faulted (`BadTcpNotEnoughResources`) | 0 errors, ~45–54k req/s | - **Fixed-rate sampling** (`30b769d13`): the sampling loop waited one interval after each sample, adding timer overshoot and sampling time to every cycle; samples now have absolute deadlines. - **Async response send** (`cdac5c5ca`): `SendResponse` blocked thread-pool workers on the channel gate; with many requests in flight the receive loop that was handed the gate waited for a pool thread and the channel stalled. The TCP, Kestrel TCP and WebSocket listeners now use `SendResponseAsync`. - **Backpressure instead of channel fault** (`191cd2ba7`): server channels stop reading while more than 100 responses are queued and resume at 50, instead of faulting the channel. Client channels keep reading. At 10,000 outstanding requests throughput is ~50k req/s with or without the limit engaged, so the remaining gap to the 100-per-client case is not the backpressure. ## Not changed - `SentMessageQueue` deep-clones every `NotificationMessage`. Retained messages return their pooled payloads on acknowledgement while the response copy may still be queued for send, and a client can acknowledge a sequence number before receiving it; the clone keeps a recycled payload from reaching another session's response. Removing it needs a send-completion hook from the transport. - Generated responses allocate a default `ResponseHeader` that is replaced (~2-3 % of per-request allocation). Fixing it means making structure fields of every generated type lazily initialized, a generator-wide change to default values, equality and encoding. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
romanett
|
9b034f8291 |
Enable nullable reference types centrally; migrate all unset and annotations-only projects (#4700)
# Description Nullable reference types are now enabled once, in `common.props`, and every project in the repository builds warning-clean with them. **Configuration** - `<Nullable>enable</Nullable>` lives in `common.props`. All per-project `<Nullable>` settings are removed: 178 project files (enable, annotations, and the `disable` in the three Aot test projects). - 419 per-file `#nullable enable`/`restore` directives and the 4 `#nullable disable` directives are removed. - Kept on purpose: - The 5 `<auto-generated>` schema files (`UANodeSet.cs`, `UA Type Dictionary.cs`, `OPCBinarySchema.cs`, `SecuredApplication.cs`, `PInvokeHelper.cs`) keep `#nullable enable`. The compiler ignores the project setting in generated code, so the directive is required there. - `tests/Opc.Ua.Interop.LegacyPeer` keeps `disable`. It builds against the 1.5.378 packages and does not import the repository props. - `docs/DeveloperGuide.md` and `.github/copilot-instructions.md` now describe the central setting. **Migration** (the 43 unset and 13 annotations-only projects, roughly 10k diagnostics) - **Production code: annotated at the declarations.** Covers `tools/Opc.Ua.SourceGeneration.Core`, `src/Opc.Ua.Lds.Server` and the LDS sample. - Members, parameters and returns are nullable where null actually flows. - Null-check helpers carry `[NotNullWhen]` (`IsNull`, `IsValidXmlName`, `IsMethodTypeDesign`). - `CoreUtils.Format`, `Utils.Format`, `ITemplateWriter.WriteLine` take `params object?[]`. `Template.AddReplacement` takes `string?` (rendering already skips null replacements). - `IRegisteredServerStore.RegisterAsync` returns `Task<RegistrationEntry?>`: it returns null when a registration is removed. - Dereferences backed by an invariant (for example, the validated model) use `!`, so runtime behaviour is unchanged. - **Behavioural changes in production code** (small, deliberate): - `NodeSetToModelDesign` throws `InvalidOperationException` when the file is not a NodeSet. Before, it threw a NullReferenceException on the next line. - `ModelDesignValidator` starts its lookup tables empty instead of null. A redundant `if (m_nodes == null) return;` guard is gone. - **Test code: minimal annotations.** - `?` on values that can be null, `!` where the test asserts non-null. - `= null!` for fixture members assigned in SetUp. - `TryGetStructure<T>(out T? x)` where the inferred `T?` broke the `IEncodeable` constraint. - A few redundant `== null` checks on structs (`NodeId`, `ArrayOf<T>`) folded into the adjacent `IsNull`/`Count` checks. **Notes for reviewers** - The diff is large (~1,500 files) but almost entirely mechanical: `?`, `!` and directive removal. The non-annotation changes are listed above. - Known follow-up, not done here: `IModelDesign.Namespaces/TargetNamespace/NamespaceUris` stay non-nullable even though the validator returns null before it loads a dictionary. Making them nullable would ripple through every generator call site. ## Related Issues - Fixes #4439 ## Checklist - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [ ] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [ ] I have addressed **all** PR feedback received. The "no new warnings" box covers net10.0 and net48: full non-incremental builds of `UA.slnx`, `tools/SourceGeneration.slnx`, `tools/MigrationAnalyzer.slnx` and the Aot test projects finished with 0 warnings and 0 errors under the repository's warnings-as-errors settings. net9.0 and net8.0 also had no nullable diagnostics, but they were not rebuilt after the final fixes. Tests have not been run locally yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
romanett
|
d98eb6b28c |
Fix server session defects found by a spec-checked review of CreateSession, ActivateSession, session lifecycle and continuation points (#4632)
# Description
Fixes defects found by a spec-checked review of the server session
implementation: CreateSession, ActivateSession, the session lifecycle
(per-request validation, CloseSession, Cancel, timeout, shutdown) and
session continuation points. Each finding was checked against OPC
10000-4 (§5.7, §7.9, §7.32), OPC 10000-5 (session diagnostics and audit
events) and OPC 10000-6 §6.8.2.
There is one commit per finding, and each commit message cites the spec
section. A max-effort code review of the fixes then found 15 more
issues, mostly regressions the first round introduced; they are fixed in
the `revfix` merges.
## What changed
**CreateSession**
- At MaxSessionCount the oldest session that was never activated is
closed to make room (§5.7.2.1). Sessions that are activated, already
closing, or have an activation in flight are never evicted. The victim
is claimed atomically, so a late activation commit on it is rejected.
- `requestedSessionTimeout` NaN is revised. A malformed `endpointUrl` no
longer fails the call with Bad_UnexpectedError. `serverUri` is ignored,
as §5.7.2.2 requires.
- Unnamed sessions get distinct names.
- Session diagnostics now report `MaxResponseMessageSize` and
`ServerUri`, and `SessionSecurityDiagnostics.Encoding` uses the
specified names ("UA Binary", "XML", "JSON").
- AuditCreateSessionEvent carries SecureChannelId and the request
certificate, also on failure.
- Every rejected CreateSession is counted, and only once. A rejection
during server startup no longer turns into Bad_UnexpectedError.
- If an `OnApplicationCertificateError` override accepts an error, the
certificate is now kept and the remaining checks still run. A
certificate accepted this way does not grant TrustedApplication or
application-based roles.
**ActivateSession**
- EphemeralKey handling follows OPC 10000-6 §6.8.2:
- A key used to decrypt a UserName or IssuedToken secret is retired and
replaced in the same commit.
- A key that was not used is kept, and nothing is returned.
- ECDHPolicyUri is honoured; an unsupported policy gets
Bad_SecurityPolicyRejected.
- Request parameters are no longer echoed back.
- After the activation commits, a failure or cancellation no longer
faults the response, which would hide the new server nonce. Post-commit
callbacks run uncancellably.
- An activation that overlaps a session close or timeout fails with
Bad_SessionClosed instead of returning Good.
- The PolicyId of an AnonymousIdentityToken is validated.
- Status codes now follow §5.7.3.3:
- Bad_ApplicationSignatureInvalid for a mismatched client-signature
algorithm
- Bad_UserSignatureInvalid for a mismatched X509 user-token signature
algorithm
- Bad_IdentityTokenInvalid for an undecodable token or an unknown policy
- `ClientUserIdOfSession` keeps the user of the first activation.
**Lifecycle and requests**
- Requests on an expired or closing session are rejected with
Bad_SessionClosed (§5.7.2.1). A CloseSession on an expired session that
is not yet closing is still processed as the client's close and honours
`deleteSubscriptions`.
- Closing a session aborts its outstanding requests with
Bad_SessionClosed. A request that registers after that abort pass is
rejected too.
- Cancel completes the matching requests with
Bad_RequestCancelledByClient (§5.7.5.1). It also reaches requests still
waiting in the request queue, but only when both timestamps prove the
request was sent earlier.
- There is one AuditCancelEvent per Cancel call, carrying the client's
audit entry id and user.
- Every close path goes through a single close claim, so
SessionAbortCount, SessionTimeoutCount and the "Session/Terminated" and
"Session/Timeout" audits are each counted once.
- Teardown that started cannot be cancelled. Requests rejected by
session validation are counted.
**Continuation points**
- HistoryRead returns Bad_NoContinuationPoints instead of evicting
points of the same request (§7.9). Only the request that saved a point
pins it, so points from prior requests can still be freed.
- Points saved by a HistoryRead that faults are released.
- Session-less Browse and BrowseNext no longer dereference a null
session.
## Behaviour changes reviewers should know about
- At the session cap, CreateSession evicts a non-activated session
instead of returning Bad_TooManySessions. The eviction is counted in
SessionAbortCount.
- Clients see Bad_SessionClosed for requests on expired or closing
sessions, and Bad_RequestCancelledByClient for cancelled requests.
- `ActivateSessionProcessAdditionalParameters` is now called on every
successful activation, with or without an AdditionalHeader.
- If an `OnApplicationCertificateError` override returns instead of
throwing, CreateSession now accepts the certificate.
- Timeout and termination audits are now raised after the session
teardown.
- The only public API changes are additive overloads in `AuditEvents`
and `RequestManager`.
**Known partial fixes:**
- The session's `maxResponseMessageSize` is reported but not enforced;
only the channel limit applies.
- With overlapping ActivateSession calls on one session, an earlier
response can still carry a key that a later activation has already
replaced. Closing that fully needs a new `ISession` member.
## Related Issues
- The Redundancy.Server session-mirroring findings from the same review
are tracked separately in #4627.
## Tests
- Regression tests cover each finding. The new files are:
- `CreateSessionSpecRegressionTests`
- `ActivateSessionValidationRegressionTests`
- `StandardServerActivateSessionRegressionTests`
- `SessionLifecycleRegressionTests`
- `SessionRequestAdmissionRegressionTests`
- `HistoryReadContinuationDispatchTests`
Existing test classes were extended too.
- Local results on net10.0:
- `Opc.Ua.Server.Tests`: 7883 passed / 0 failed / 5 skipped
- `Opc.Ua.Sessions.Tests`: 790 passed / 0 failed / 379 skipped (same as
master)
- net48 Release build of `Opc.Ua.Server.Tests`: 0 errors / 0 warnings.
## Checklist
- [x] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [ ] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [ ] I have addressed **all** PR feedback received.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
romanett
|
ce88955f70 |
Harden encoders/decoders against untrusted input and align them with Part 6 / Part 14 (#4599)
# Description
Hardens the encoders and decoders against untrusted input and aligns
them with OPC 10000-6 (1.05) and OPC 10000-14. This comes from an audit
of every codec reachable from a peer:
- Binary, JSON and XML decoders and encoders;
- built-in and complex types;
- UA-TCP/UASC framing;
- PubSub UADP and PubSub JSON.
The audit found 97 issues: 1 critical, 7 high, 39 medium and 50 low.
They are all fixed here, one commit per finding (IDs B1-, J1-, X1-, T1-,
S1-, U1-, P1-, E1-, E2- in the commit subjects). Every fix was then
checked against the spec a second time, and the deviations that re-check
found were fixed as well (`*-spec` commits).
## Security (decoders reachable from unauthenticated peers)
- **Allocation from length prefixes (B1-1, B1-5, U1-1, U1-3).**
- Array, string and byte-string lengths are checked against the bytes
left in the message before anything is allocated.
- Arrays read element by element grow as elements are decoded, so nested
length prefixes can no longer multiply memory.
- UADP discovery counts and chunk reassembly buffers are bounded the
same way.
- **Stack exhaustion.** Nesting is checked for remaining stack
(`RuntimeHelpers.EnsureSufficientExecutionStack` mapped to
`BadEncodingLimitsExceeded`), and the large Variant read/write methods
were split into scalar, array and matrix methods. Measured on 1 MB and
256 KB threads, no codec overflows at the default limits any more.
- **XML element depth (X1-1).** XmlElement values and raw
ExtensionObject bodies are copied node by node with a depth limit,
instead of with the recursive `InnerXml`, `ImportNode`, `InnerText` and
`DeepEquals` calls.
- **Limit bypasses:**
- JSON NodeId, QualifiedName and opaque strings now respect
MaxStringLength and MaxByteStringLength (J1-1).
- Matrix rank is limited to 32.
- Decimal size is bounded (T1-3).
- XML base64 is checked before it is decoded.
- XML arrays are checked before each element is read.
- **CPU and algorithmic cost:**
- UADP reassembly uses an O(1) overlap check with a per-message chunk
cap (U1-2).
- PubSub JSON matches fields to metadata through a name index (P1-2).
- LocalizedText "mul" text is parsed lazily (T1-5).
- Unknown SecurityTokenIds trigger a throttled background SKS refresh
instead of an inline call per datagram (U1-9).
- **PubSub metadata poisoning (P1-1).** Inbound metadata is registered
only for identities a configured reader expects, is capped, and an
unauthenticated message can no longer pin a version.
- **Exception contract.** Decoders fail with `ServiceResultException`
(BadDecodingError / BadEncodingLimitsExceeded) instead of letting
ArgumentException, InvalidOperationException, TypeLoadException and
similar escape.
- **UASC (S1-*):**
- OPN and CLO must be single final chunks.
- The certificate of an unsigned OPN is ignored.
- The padding and ExtraPadding bytes are validated.
- A renewed token is activated only after a chunk has been verified with
it.
- A policy change on renewal is rejected before the channel certificate
is touched.
- Secured chunk length arithmetic is bounded.
## Wire-visible behaviour changes
All follow the spec exactly; there is no reader for the old forms unless
noted.
- **UA Binary:**
- Variant type ids 26-31 decode as ByteString.
- Picoseconds without their timestamp are ignored, and values of 10000
and above are clamped to 9999.
- A Null-type Variant with the array flag consumes its length and
dimensions.
- **UA JSON:**
- Messages are ExtensionObjects with the body inline; there is no
`UaBody` envelope any more (5.4.9 / 5.4.2.16).
- The Variant `Value` is omitted exactly for NULLs of nullable types
(Table 41).
- RawData keeps nested UaType/UaTypeId and the StatusCode Symbol.
- Duplicate member names are rejected.
- `{}` decodes as a null ExtensionObject.
- Numbers and identifiers are parsed strictly and culture-independently.
- QualifiedName follows 5.4.2.14.
- Opaque bodies of known types carry the DataType id.
- Enumerations without a literal are written as the number.
- **UA XML:**
- The earliest DateTime is written as `0001-01-01T00:00:00Z`.
- Null typed scalars and null array elements are written with `xsi:nil`.
- An ExtensionObject without a body has no `Body` element.
- Strings XML cannot represent are rejected.
- Only the xs:dateTime and xs:date forms are accepted; a value without a
zone is still read as UTC, because published model files use it.
- Only the Part 6 Guid form is accepted.
- Booleans are case-sensitive in XmlDecoder.
- **UA-TCP:**
- OPNC/OPNA/CLOC/CLOA fault the channel. Older releases that split a
long-chain OPN into chunks are rejected, so this needs a release note.
- The sender truncates the certificate chain to fit a single chunk.
- The received certificate chain is sized from the chunk size.
- Hello buffer sizes down to 1024 bytes are honoured; below that, an ERR
with Bad_TcpInternalError is sent.
- A 4096-byte Error reason is accepted, and the SecurityPolicyUri limit
is 255 bytes.
- **PubSub UADP:**
- Chunk NetworkMessages use the Part 14 7.2.4.4.4 layout, and each chunk
is secured on its own with its own sequence number.
- The Sizes array is part of the secured payload.
- Padded RawData keeps its length prefix, and null values are written as
-1.
- DataSetMessages with the valid bit clear are skipped.
- Null discovery arrays are accepted.
- **PubSub JSON:**
- Variants use `UaType`/`Value`, with the 7.2.5.4.2 top-level collapse.
The deprecated `Type`/`Body` form is still read.
- Status is written as a `{Code,Symbol}` object; a numeric Status is
still accepted.
- Discovery messages follow Tables 188-192, with GUID MessageIds.
- Enumeration fields are written as `Name_Value`.
- Fields appended with a MinorVersion bump are accepted.
## Known deviations and limitations (documented, not changed)
- A known type whose ExtensionObject XML body fails to decode fails the
message, like a failing binary body. 5.2.2.15 only says the decoder
*should* continue, and continuing reopens the log-flooding path.
- Reserved Variant type ids 26-31 are passed to the application as
ByteString without the raw type id.
- JSON UaTypeId of an opaque body of an *unknown* type stays the
encoding id, because no type system is available to map it.
- PubSub JSON encoder:
- Structure fields without a subtype are still wrapped with a UaType,
and concrete multi-dimensional or abstract-rank fields still carry a
UaType (Table 186).
- Namespace-0 enumerations without EnumDataTypes are written as a
numeric string.
- Chunked transcoded frames have no writer group, so their sequence
numbers continue from the source number.
- Pre-existing and out of scope: the HTTPS JSON binding uses
Content-Type `application/opcua+uajson`, while 1.05 7.4.5 requires
`application/json`.
## Tests
- Every fix has a regression test.
- Tests that pinned non-conformant behaviour were updated, and their
commit messages say so.
- Fuzz seeds were regenerated where the wire changed.
Local results:
net10.0, 27 suites green (Types 11354, Core.Encoders 2933, Encoders.Fuzz
8112, Core 5955 + 1 known intermittent ActivityTraceFlags metrics test
that passes alone, Client 2732, Server 7802, Sessions 790, GDS 1234,
Subscriptions 641, SourceGeneration.Core 4266, PubSub 1490 and all
PubSub transport suites). Full UA.slnx build 0 warnings / 0 errors on
net10.0 and net48.
## Related Issues
- Follow-up: #4598 (possible throughput regression of per-element binary
array reads).
## Checklist
- [x] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [x] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed. (net10.0
suites run; net48 build only)
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [ ] I have addressed **all** PR feedback received.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
romanett
|
20c2d77d57 |
Fix Core/Types audit findings and conform them to the OPC UA spec (stacked on #4569) (#4573)
> **Stacked on #4569.** The base branch is `romanett/practical-murdock-fd9c56`, so this diff shows only the commits of this PR. Retarget to `master` after #4569 merges. ## Summary Bug audit of `Opc.Ua.Core`, `Opc.Ua.Core.Types`, `Opc.Ua.Types` and `Opc.Ua.Core.Schema`, with the findings fixed and the fixes grounded against the OPC UA specification (reference.opcfoundation.org). 133 commits in four groups: 1. **Audit fixes** (`Fix <ID>: ...`): 116 findings from a read-only audit, each with a regression test. IDs follow the audit scopes: T1-T6 `Opc.Ua.Types`, C1-C4 `Opc.Ua.Core`, S1 `Opc.Ua.Core.Types` + `Opc.Ua.Core.Schema`. 2. **Review fixes** (`Fix review <ID>: ...`): issues a code review of group 1 found, plus lower-severity leftovers. 3. **Spec fixes** (`Spec: #<n> ...`): the 18 places where an earlier fix only partly matched the spec text. 4. Test and analyzer follow-ups. Where `master` (#4516, #4522, #4540, #4564) or #4569 already fixed the same issue, their version was kept and this PR only adds what they didn't cover. ## Behaviour and user-facing changes ### Wire format and interoperability (breaking) - **PubSub AES-CTR (Aes128-CTR / Aes256-CTR)** now follows Part 14 Tables 156/157: - The counter block is `KeyNonce[4] | MessageNonce[8] | BlockCounter[4]`, and the block counter starts at 1. - The SecurityHeader nonce is 8 bytes: `Random[4] | UInt32 SequenceNumber`, with the sequence number starting at 1. - SKS key data is split with a 4-byte KeyNonce, giving 52 / 68 bytes. - **Not interoperable with earlier releases of this stack**, but matches the spec. There is a test against the independent RFC 3686 vector. - **User token signature** (Part 4 6.1.8): - The signed data is now built from the user token policy's SecurityPolicy on both client and server. - It also supports the SecurityMode None layout `ServerNonce | HASH(ServerCertificate) | ClientNonce`. - New overload `SecurityPolicyInfo.GetUserTokenSignatureData(..., MessageSecurityMode)`. - **Secure channel:** - The ECC AEAD `LastSequenceNumber` is correct across the 32-bit sequence wrap (Part 6 6.8.1). - The incoming chunk counts against MaxChunkCount/MaxMessageSize. - The sender no longer lets a message through that is 24 bytes over MaxMessageSize. - The OPN response signature is verified without overrunning the buffer (RSA_DH with differing key sizes). ### Sessions (server) - **CreateSession nonce:** - The client nonce is kept on None channels, because the None-mode user token signature covers it. - A nonce over 128 bytes returns `Bad_NonceInvalid` on every channel (Part 4 5.7.2.3). - The duplicate-nonce check runs only on secured channels. - A non-empty nonce under 32 bytes is still accepted on None, for compatibility. - **ActivateSession:** an embedded `EccEncryptedSecret` signing certificate is validated with the server's peer certificate validator (Part 6 6.8.3). Byte equality with the client certificate is only the fallback when no validator is available. New opt-in interface `ICertificateValidatorProvider`. - **FindServers / GetEndpoints:** `GatewayServerUri` and `DiscoveryProfileUri` are no longer swapped. ### Write, Read and Call (NodeState / node managers) - **UserWriteMask:** - Enforced for non-value attributes and for VariableType Value writes (`ValueForVariableType`), returning `Bad_UserAccessDenied`. - An unconfigured UserWriteMask (0, no handler) is not enforced, and reads back as the WriteMask so Read and Write agree (Part 3 8.60). - A throwing or Bad `OnReadUserWriteMask` handler fails only that item. - **Index-range writes:** - Slices are merged into the cached value under the attribute lock, also for `OnWriteValueAsync` handlers. - A data-change notification is still produced when the cache cannot absorb the slice. - The slice must match the unclipped range, otherwise `Bad_IndexRangeDataMismatch` (new status code in `Opc.Ua.Types`). - A range past the target returns `Bad_IndexRangeNoData`. - A handler without index-range support returns `Bad_WriteNotSupported` (Part 4 7.27, 5.11.4.4). - **Index-range reads on matrices** return partial results clipped to the source, instead of allocating the requested size (a Read with `"0:40000,0:40000"` no longer allocates about 1.6e9 elements). - **DataTypeDefinition writes:** - A value that isn't a decoded DataTypeDefinition, or whose kind doesn't match the DataType category, returns `Bad_TypeMismatch`. - Null is rejected for concrete Structure/Enumeration/OptionSet types. - WriteMask is checked first (Part 3 5.8.3). - **AnalogItem:** a null write skips the InstrumentRange check again. - **Method call:** the async fallback passes the ObjectId to the method handlers. - **Redundancy:** `DistributedValueParticipation` merges index-range writes and applies the index range on reads. - **Change masks:** `ClearChangeMasks` takes the bits atomically, so concurrent changes are no longer lost. - **`AccessLevelEx`:** preserved by Clone/Export. ### Alarms & Conditions (Part 9) - **Acknowledge / Confirm / AddComment:** - Accept any EventId the condition or its branches reported while that state is still pending (5.7.3). - An unknown id returns `Bad_EventIdUnknown`. - An already handled state returns `Bad_ConditionBranchAlreadyAcked` / `...AlreadyConfirmed`. - AddComment on a branch no longer changes the trunk, and the branch table is re-keyed when a branch's EventId changes. - **Shelving:** - OneShotShelve without MaxTimeShelved no longer overflows the timer. - A timed shelve is measured on the monotonic clock, so wall-clock steps don't delay the unshelve. - The state is named `OneShotShelved`. - **Audit events:** - They report failure for rejected method calls. - `SelectedResponse` is a UInt32. - A rejected negative Respond leaves SelectedResponse unset. - **`ContentFilterElement.Validate`:** an unknown FilterOperator is reported per element as `Bad_FilterOperatorInvalid`, instead of failing the whole service call. ### ContentFilter evaluation (Part 4 7.7.3, on top of #4564) - **Bitwise operators:** - The result takes the size of the larger operand. - Boolean, numeric String and one-element array operands are converted to integers implicitly. - Types without an implicit integer conversion give NULL. - **Strings:** ordering is ordinal, no longer culture dependent. - **Cast:** restricted to the Table 121 conversions, and QualifiedName to LocalizedText is added. The general `Variant.ConvertTo` keeps its wider set. - **Null and empty operands:** - `Like` with a NULL operand is NULL. - Empty arrays of the same DataType compare equal. - Null-string operands are NULL everywhere. ### Encoders and built-in types - **XML:** - Strings keep leading, trailing and whitespace-only content (Part 6 5.3.1.5). NodeSet import opts back into treating whitespace-only strings as empty with the new `XmlDecoder.TreatWhitespaceOnlyStringsAsEmpty`. - Layout whitespace around `<Identifier>` values is ignored, and trailing control characters are stripped from `s=` ids. - `XmlElement` ignores DTDs without expanding entities. - **JSON:** - A DateTime without an offset is read as UTC. - An unmappable `nsu=` NodeId and an unknown `svu=` ExpandedNodeId decode to a namespace-0 String NodeId (5.4.2.10 / 5.4.2.11). - `svr=` is rejected in a NodeId field. - Bad matrix dimensions return `BadDecodingError`. - The non-strict body stack stays balanced after an error. - The nesting limits match between encoder and decoder. - **Binary:** - `MaxByteStringLength` no longer applies to strings. - ExtensionObject bodies count toward the nesting limit on seekable streams (and in the XML encoder). - **Parsing:** - The NodeId parser requires `i=`/`s=`/`g=`/`b=`. - `ExpandedNodeId.Parse` rejects an empty `nsu=`/`svu=`. - NumericRange rejects a trailing `,` with `Bad_IndexRangeInvalid`. - RelativePath escapes `#`. - **Status codes:** `StatusCode.WithSubCode`/`WithCodeBits` keep the severity bits and look up the symbolic id for the new code. - **Variant / VariantHelper:** - `TryCastTo` returns false on a type mismatch. So `DataValue.GetValue<T>(default)` returns the default, and `GetValueOrDefault` / `CastTo` throw `BadTypeMismatch` as documented. - `System.DateTime` casts are supported. - Concrete encodeable arrays (e.g. `Argument[]`) no longer throw. - `ConvertTo` keeps one-element and empty arrays as arrays and checks the conversion is legal for empty arrays. - `CastFrom(object[])`, `CastFrom(Enum[])` and the obsolete `Variant(object)` constructors keep the array shape (previously a one-element array collapsed to a scalar and an empty one to Null). - `TypeInfo.Unknown` / `Variant.Null.TypeInfo` report `ValueRanks.Any`. - **Equality and hashing contracts fixed:** - Types: Matrix/MatrixOf, typed-null Variants, NodeId (the namespace index is now hashed), ExpandedNodeId, LocalizedText, DiagnosticInfo, StructureDefinition/EnumDefinition, CertificateIdentifier, UserIdentity and DateTimeUtc. - `MatrixOf.Dimensions` returns a copy. - Decimal equality and hashing now run in bounded time. - **Localized text:** a LocalizedText with locale `"mul"` and no usable translations, or null client locale ids, no longer throw. ### NodeSet, address space and utilities - **NodeSet export/import:** - Server index 0 is the local server on import and export, so local ids are no longer written as `svr=65535`. - `StructureDefinition.BaseDataType` is filled from the supertype on import. - **`NodeTable.Import`:** replaces duplicates, keeping the references neighbouring nodes hold. `TypeTable.Add` detaches a re-parented type from its old supertype, and `Node.Clone` gets its own ReferenceTable. - **`StringTable` / `NamespaceTable`:** capped at UInt16 indexes (`BadEncodingLimitsExceeded`) for Append and Update, instead of silently wrapping. - **Small fixes:** - `UnsecureRandom.Shared` is seeded randomly. - The peer's RetryAfterMs is clamped to one day. - The `CancelAsync` and `string.GetHashCode(StringComparison)` polyfills match BCL behaviour. - `CoreUtils.IsEqual` compares strings ordinally. ### Schema generation and ComplexTypeSystem - **JSON schema:** describes the 1.05 layout with the compact/verbose null rules, and now validates real `JsonEncoder` output. - **XSD:** - `EncodingMask` (`xs:unsignedLong`) only for StructureWithOptionalFields. - Guid and StatusCode are complex types. - Standard types are in the `Types.xsd` namespace. - **Structure-backed OptionSets** are described as `{Value, ValidBits}`. New `UaTypeDescription.IsStructureOptionSet` property and constructor overload. - **ComplexTypeSystem:** - A malformed type in a server dictionary is skipped per type instead of discarding the whole dictionary. - Cyclic supertype walks are bounded. - Declared definitions are no longer changed. - A dictionary validator is kept only if the dictionary parsed. ### Certificates and trust lists (Part 4 6.1.3, Part 12) - **Revocation:** - Checked for issuers supplied in the peer's chain and for inline-trusted issuers, against the CRLs in the trusted and issuer stores. - A store without CRL support no longer hides "revocation unknown". - `RejectUnknownRevocationStatus` is honoured for inline-trusted issuers unless the entry sets `SuppressRevocationStatusUnknown`. This changes the strict case of `ExplicitIssuerChainIsHonoredWithoutStorePathAsync`. - **Basic constraints / path length** violations fail with `Bad_CertificateInvalid`. - **`TreatAsInvalid`:** - Now yields `Bad_CertificateUntrusted`, which can't be suppressed or auto-accepted. - It is honoured on individual trust-list entries even when the certificate is also in a store. - **`EccEncryptedSecret`:** enforces the result of validating the sender certificate. - **`TrustListTransaction`:** - Routes each CRL to the store holding its issuer (TrustedCrls/IssuerCrls). - Validates staged CRLs, including the store's CRL support, before changing anything. - Applies certificates before CRLs. - **Other:** - One failing `CertificateChanges` observer no longer stops the others. - `InMemory:` stores share their contents per path, and `InMemoryStoreProvider` is `IDisposable`. ### WoT (`Opc.Ua.Types/Wot`) - **Projection security:** projection security schemes whose qualified names collide are refused. New diagnostic `WotDiagnosticCode.ProjectionSecurityConflict = 6063`. - **Input handling:** - Invalid Unicode (lone surrogates, bad UTF-8) is rejected as `JsonException`. - External-schema results are keyed per affordance. - A duplicate security-floor member is rejected. - `nsu=` ids are percent-encoded (Part 6 5.1.12). - The configured JSON depth is honoured on re-parse. - An oversized document is reported before it is parsed back. - **Performance and limits:** - Subtype-graph validation, `componentOf` linking and the namespace table are now linear. - Namespaces are capped at UInt16. ### New or changed public API - `ICertificateValidatorProvider` (server) - `SecurityPolicyInfo.GetUserTokenSignatureData(..., MessageSecurityMode)` overload - `XmlDecoder.TreatWhitespaceOnlyStringsAsEmpty` - `UaTypeDescription.IsStructureOptionSet` plus a constructor overload - `WotDiagnosticCode.ProjectionSecurityConflict` - `StatusCodes.BadIndexRangeDataMismatch` in the `Opc.Ua.Types` list - `InMemoryStoreProvider : IDisposable` ## Not changed / open - **Deferred:** - S1-14 (BSD OptionSet size needs the base type in `UaTypeDescription`, and BSD is deprecated). - C2-7 (restoring the own-certificate trust exemption would accept certificates the spec says to reject; removing the dead code is a separate decision). - A1-4 (config values keep surrounding whitespace, as the pre-April-2026 loader did). - H-2 (JsonDecoder has no real nesting counter). - A2-3 (the StringTable cap also applies to diagnostics tables; default `MaxArrayLength` already bounds them). - **Needs a product decision:** - The general `ConvertToString` writes a QualifiedName as `ns:Name`, while Table 121 drops the namespace. - Short (< 32 byte) client nonces are accepted on None channels. ## Verification - Every commit added a regression test that fails without its fix, where practical. - Before the rebases, with all audit and review fixes (before the `Spec: #…` commits): 0 build warnings and 16 test suites green on net10.0. The suites: Types, Core, Core.Encoders, Core.Security, Core.Schema, Client, Client.ComplexTypes, Server, Sessions, Gds, History, Features, Redundancy.Server, PubSub, WotCon and WotCon.Bindings. Core hit its known intermittent end-of-run hang once, and reruns were clean. - The `Spec: #…` fixes were verified per group on their own suites. Those include PubSub.Udp, Redundancy.PubSub, PubSub.Server, Configuration and SourceGeneration.Core. - **Not yet built or tested on this base.** It was rebased onto master (resolving overlaps with #4522, #4540 and #4564) and then onto #4569. This PR stays a draft until the full run on this base is green. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: marcschier <11168470+marcschier@users.noreply.github.com> Co-authored-by: Marc Schier <marcschier@users.noreply.github.com> |
||
|
Marc Schier
|
f9892020a3 |
Add configurable server resource isolation (#4541)
# Description Add configurable server resource isolation on top of the merged #4522 foundation. - Coordinate connection, handshake, retained-message, session-establishment, queue, execution, request-cost and parked-request limits through one server-scoped provider. - Keep configured startup, continuity/reconnect, control and trusted-owner reservations inside existing totals. Shared capacity remains usable up to explicit owner ceilings. - Use authoritative Session membership for reassembly continuity capacity without granting unrelated resource or rate privileges. Preserve explicit owner mappings and bounded tracking. - Cache immutable classification evidence while retaining current Session and classifier checks. - Support weighted request ordering or FIFO independently of resource accounting. Installed providers enforce decoded-request limits in both modes. - Preserve fail-fast reassembly, valid Abort handling, fixed deadlines, active-work protection, reverse-connect handoff and bounded OpenAPI processing. - Support custom-handler parking through `IRequestParkingPolicy`, `WithRequestParking(...)`, and `RequestLifetime.ParkSink`. Parking releases execution capacity only, keeping retained-data and parked charges until completion. Built-in Publish and the global opt-out remain unchanged. - Provide direct, fluent and DI configuration, public incoming-request parking support, effective startup limit logs and user-facing documentation. Balanced remains the default. SharedOnly without an explicit provider retains shared behavior for intentionally incompatible/unlimited configurations. Reservations are finite and are not one full-message allowance per Session. ## Related Issues Builds on merged #4522. Fixes #4554. ## Review and CI corrections All three previously missed findings in [review 5326734718](https://github.com/OPCFoundation/UA-.NETStandard/pull/4541#pullrequestreview-5326734718) are addressed in `9e25e4ecc`: 1. A custom FIFO provider receives RequestQueue, RequestQueueBytes, RequestExecution and ParkedRequest accounting. A blocked head cannot be overtaken. 2. `IResourceIsolationReassemblyProvider` replaces the sessionless threshold independently of `UseFairScheduling`. The aggregate byte ceiling remains enforced. 3. Runtime `ReservedReassemblyBytes` includes all configured protected stage floors, including ControlReserved. The review's two already-resolved changes remain intact. The CI collection race was fixed at its source in `e1a994a4b`: `ArrayOf.ToArrayOf` treats a separately read collection count as an initial capacity hint and grows/trims for the actual enumeration. Deterministic tests cover growth, shrinkage and stale-zero counts. The subsequent recovery-test race was fixed in `05753ce77`: fake time advances only after a gated keepalive response has been processed and reset its timer, not merely after request arrival. Recovery-order and timeout assertions remain intact, with additional response-state assertions. Instrumentation reproduced the original failure twice; both fixed variants passed 400 executions per framework. CodeQL alert 388 was verified and dismissed as a false positive. Startup logs contain aggregate numeric capacities, not credentials or caller identities. ## Validation | Scope | Result | | --- | --- | | Latest automated checks at `05753ce7775a58801b4ba9d8f6d7781b709d0b89` | 167 successful, one package-publication check skipped, no failures | | Remaining external prerequisite | `license/cla` pending: Contributor License Agreement is not signed yet | | FIFO/weighted queues, custom parking and server-channel selection | 175 passed on each net10.0/net48 | | Isolation, configuration and parking-hosting selection | 259 passed net10.0; 257 passed net48 | | ArrayOf conversion tests | 73 passed on each net10.0/net48 | | Recovery-audit and keepalive fixtures | 40 passed on each net10.0/net48 | | NativeAOT smoke from preceding updates | Published and executed successfully | No tests, assertions, coverage thresholds, or required checks were weakened. The remaining CLA prerequisite must be completed through [CLA Assistant](https://cla-assistant.io/OPCFoundation/UA-.NETStandard?pullRequest=4541). This is not CTT certification or an unconditional network-DDoS, whole-process-memory or fleet-wide guarantee. ## Checklist - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [ ] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [x] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [x] I have addressed **all** PR feedback received. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: marcschier <11168470+marcschier@users.noreply.github.com> |
||
|
Marc Schier
|
46fa7d364a |
Harden recovery, concurrency, and resource lifetimes across the OPC UA stack (#4516)
# Description Bound managed-session recovery, preserve Publish workers and recovery ownership, and address the client, server, transport, and certificate audit findings and subsequent concurrency review. Also reduce the memory retained by exhaustive session-test diagnostics. ## Recovery and Publish ownership - Add `ManagedSessionOptions.ChannelReconnectTimeout` through direct construction, the fluent builder, and DI. The automatic duration is `max(3 * KeepAliveInterval, revised SessionTimeout, OperationTimeout)`, sampled at cycle start. Positive overrides are supported; `Timeout.InfiniteTimeSpan` explicitly opts out. Raw sessions retain their unlimited default. - Add `CreateReconnectBudget(TimeProvider)` as a required `IReconnectParticipant` member, as agreed in review. Custom participants implement it and may return null for no participant-specific bound. - Enforce the earliest participant/caller deadline through transport reconnect, activation/recreation, and subscription restoration. Late callers can shorten an active cycle but cannot extend it. - Cancel and unwind recovery, fence late responses, and release channel recovery ownership before the existing `Faulted` event hands control to the outer reconnect/failover policy. Preserve session recreate admission through final Publish cleanup and keepalive suppression until shared recovery is complete. - Treat participant budget creation/evaluation failures, including exceptions at the exact deadline, as terminal shared-recovery failures. Notify participants before exposing `Faulted`, preserve the original diagnostic, and never silently replace a finite budget with unlimited recovery. - Keep caller-supplied replacement channels, managed bindings, wrapper event subscriptions, and recovery-owner identities consistent across success, failure, and cancellation. Reject replacement during active recovery and fence retired callbacks so they cannot clear newer ownership or suppress later keepalive recovery. - Retain eligible Publish workers through temporary subscription-identifier clearing without retaining workers for never-created or removed subscriptions. Preserve cancellation, acknowledgement rollback, and drain-before-replacement ordering. - Coordinate automatic subscription updates with session recovery: cancel admitted updates, keep them paused across a failed-restoration handoff, and retry only after explicit restoration completes. This prevents competing CreateSubscription requests and updates against a retiring session, while preserving manual recreation, transfer fallback, and pending option changes. - Exercise the real ManagedSession, V2 subscription engine, and channel ready gate with scripted transports and an injected clock. Cover transfer fallback, deadline handoff, late completions, and resumed notification delivery. ## Audit and review corrections - Correct certificate/trust-list transactions, change-pump lifetimes, validation, peer revalidation, and reset/resource ownership. - Harden node admission, attribute/range/dimension validation, monitored-item sampling and lifecycle transitions, event fanout, and server shutdown. Run lifecycle readiness and compensation outside shared monitored-item admission, with stale-completion ownership checks. - Close synchronous node-manager admission before teardown and retain resources until admitted service, lifecycle, and asynchronous method work returns. Dispose the monitored-item manager outside both locks, reject late calls before they can reach it, and forward asynchronous disposal through the adapter so the master awaits the same completion. Preserve reentrant shutdown initiation, cancellation/failure unwinding, and shared cleanup-failure reporting. - Bound WSS pending-request retention using connection-local wire handles while preserving caller-visible request handles and translating Cancel targets. Safely ignore retired replies and drain admitted send/close users before resource disposal. - Clarify that the shared `ClientBase` caller-handle sequence is distinct from the non-wrapping WSS wire allocator. Existing exhaustion/reopen, late-reply, and Cancel-translation regressions verify the connection-local contract; no allocator behavior was changed by this clarification. - Preserve virtual-node validation and resolver service errors instead of converting them to cached misses. A mismatched NodeId reports `BadNodeIdInvalid`; callback type mismatches and resolver service errors keep their exact status. Per-item boundaries preserve unaffected reads, writes, history operations, method calls, browse paths, and monitored-item creation/restoration, including permission-validation paths. Caller cancellation still propagates; stored-item validation failures are logged without aborting later restores. - Correct client file/role/state-machine/streaming APIs, node-cache behavior, redundancy/failover ownership, and session configuration/identity handling. - Cancel each owned peer-discovery scope before disposing it, including when the bounded wait wins a cancellation-delivery race. Keep late results out of the endpoint cache and preserve single-flight lookup ownership. - Keep materialized alias Browse results synchronized with the live store by default, preserving explicit startup-only opt-out. Apply HistoryUpdate service results per value while retaining explicit all-or-nothing provider APIs. - Update the directly related feature documentation and add focused positive, negative, concurrency, and ownership regressions. ## Latest review-feedback corrections - Fence stale failed lease observations with a successful-confirmation generation distinct from the revocation epoch. Snapshot it before each store read, including failed-CAS revalidation, so an old failure cannot revoke or reschedule a newer lease while fresh ownership loss still revokes leadership. Preserve independent overlapping acquisitions, expiry, disposal, and late-success protections. - Isolate attributable WSS response-decoding errors to their request, including retired handles. Keep admitted socket sends connection-owned so caller timeout/cancellation cannot abort sibling requests; shutdown still cancels and drains those sends. - Restart keepalive and classic Publish replenishment after successful session reuse. Retire V2 worker demand after intentional deletion/disable without confusing that state with temporary recovery resets. - Browse virtual directories lazily across continuation pages without the 1,024-entry snapshot cap. Preserve active-page cancellation, bounded metadata retention, pushback, and exactly-once asynchronous cursor cleanup. - Apply certificate/CRL staging in last-operation order, preflight required issuer storage before writes, select actual policy-compatible RSA keys for None endpoints, frame certificate chains independently of signature OIDs, and preserve admitted validation results during rejected-writer shutdown. - Preserve base asynchronous disposal when a legacy override omits its base call; avoid double derived disposal in the adapter. Let genuinely admitted nested work finish while rejecting stale captured contexts and returning null from closed lookup APIs. - Re-park recovered monitored items if AddNodes rolls back. Give event compensation a fresh finite deadline, and preserve healthy Browse/Translate siblings at every reference/path boundary while retaining starting-node errors and caller cancellation. - Give final reconnect notifications a live, bounded shutdown-linked lifetime. Synchronize child state before parent lifecycle handlers outside the parent transition lock. - Document the accepted existing policies: direct reverse connections are single-use without a ReverseConnectManager, and multi-node stream capacity is shared rather than reserved per node. The expanded readiness controls also exposed a real generation-handoff race: an old source could finish draining after activation was deferred but before readiness tasks were collected, causing the wanted replacement to be omitted from the wait set. A deterministic test gates that exact interval; the fix keeps the replacement in the readiness wait. The original race passed 200 repetitions after the fix. A separate control now awaits both producer readiness and iterator entry under the same original deadline, rather than treating those independent signals as equivalent. Temporary logging and repetition are removed. ## CI integration contracts The History and Server CI failures exposed integration tests that still asserted the behavior intentionally changed by the audit fixes. A Windows WoT negative test raced its caller deadline against upstream retries. A subsequent Ubuntu Client failure exposed a real automatic-update versus explicit-restoration race: - History insertion collisions now verify exact per-value statuses and all persisted timestamps/values, including preservation of the existing conflicting entry. Explicit atomic-provider rollback controls remain. - Alias materialization tests wait for actual publication and verify the added node plus forward/inverse `AliasFor` references, instead of racing a startup-only snapshot expectation. Explicit refresh opt-out controls remain. - The unavailable-upstream WoT test uses an explicit finite retry policy through existing DI configuration, preserving its original four-minute guard and requiring a mapped-value service error before caller cancellation. Production retry defaults remain unchanged. - The Client fix retains the exact one-create and late-response assertions. Three deterministic real-subscription/real-manager cases cover paused creation, disposal while paused, and cancellation of an admitted create followed by explicit restoration and a pending option update. Both composed restoration-deadline cases also prove automatic updates stay blocked at the outer handoff and resume after restoration. - The merged-head net48 correction prevents lost peer-resolver cancellation and distinguishes the harness's delayed initial `Connected` event from actual outer recovery completion. A deterministic timer-order test covers both cooperative and cancellation-ignoring resolvers, cancellation before return, rejection of late cached results, and a fresh subsequent lookup. - The remaining CodeQL alert was a false content flow through `ArrayOf.Count` into an integer-only trust-list summary. Projecting the same count through `Span.Length` keeps cardinality distinct from serialized contents without allocating, dropping diagnostics, suppressing the rule, or dismissing the alert. Five log-contract cases cover null/empty/nonempty results and mixed chain-capable/fallback listeners, asserting the exact count, message, and structured properties. - Classic transfer acknowledges messages selected for asynchronous Republish only after their responses arrive. The existing locked replay-selection decision returns the immediate-acknowledgement subset to Session, preserving the public transfer API, advertised availability, and immediate acknowledgement of older gapped messages not selected for replay. A callback changing `RepublishAfterTransfer` cannot change the already-selected acknowledgement ownership. - Transfer integration tests explicitly control acknowledged versus retained origin messages. Exact assertions distinguish retained replay from requested fresh initial values by sequence number, client handles, values, status, and source timestamp. Atomic counters, observed delivery/keepalive progress, and failure-safe session/file cleanup replace timing assumptions and racy resets. - Server lifecycle tests verify the documented independent five-second compensation deadline rather than requiring a non-cancellable token. They cancel the actual request, prove compensation and unrelated monitored-item creation still proceed one tick before the deadline, preserve the original failure and ownership assertions, and verify the completed deadline is retired. - Synchronous and asynchronous directory-browser tests require the exact provider exception instead of an empty-directory success, then verify the failed cursor is not enumerated again. These correct expectations left behind by the intentional, documented provider-error propagation change; production behavior is unchanged. No tests were disabled, assertions loosened, coverage thresholds lowered, or CI checks made optional. Existing platform exclusions and Explicit debug variants remain. ## Related Issues - Fixes #4514 - Fixes #4515 - Builds on merged #4503 and retains the #4508 publishing-drain behavior. ## Validation **Lease-election review correction (`a13afa641182c70983e16bbcfebf507e666a45a5`):** The correction is based on the existing upstream merge `43079e2b851fe338a4082066f8d14abfdafb6e32`. Three deterministic stale-verdict cases fail on the original source and pass with separate confirmation fencing. Six new cases cover the initial foreign-owner verdict, failed-CAS revalidation in both invocation orders, and fresh ownership loss through both read paths, including revalidation begun after another successful confirmation. | Target and configuration | Passed | Failed | Skipped | | --- | ---: | ---: | ---: | | net10.0 Debug | 120 | 0 | 0 | | net48 Debug | 120 | 0 | 0 | The selection is the full `Opc.Ua.Core.Tests.Redundancy` namespace, including all 38 `SharedStoreLeaseElectionTests` cases and the existing expiry, clock rollback, disposal, and overlapping-success regressions. New witnesses are `StaleForeignLeaseReadDoesNotRevokeNewerAcquisitionAsync`, `StaleFailedRevalidationDoesNotRevokeNewerAcquisitionAsync`, and `FreshForeignLeaseObservationRevokesLeadershipAsync`. They assert resulting leadership, exact transitions, independent progress, and the unchanged expiry boundary. Retaining the initial generation for the fallback read independently makes the fresh-loss-after-confirmation control fail; that mutation was removed before the final rebuilt runs. The successful scoped command is `dotnet test tests\Opc.Ua.Core.Tests\Opc.Ua.Core.Tests.csproj -c Debug --no-restore --filter "FullyQualifiedName~Opc.Ua.Core.Tests.Redundancy"`, with `CustomTestTarget` and `-f` matching each row and that framework's assets restored. Both changed C# files pass formatting and analyzers through informational severity on both frameworks. These are scoped results, not a full-solution test or measured coverage-delta claim. CI monitoring is paused for this feedback-only operation; no current-head CI success is claimed. **Latest CI contract correction (`1cd1f2047bb5b50c9bf300b1ff710b0b2e935a29`):** All four failing Server shard 23 jobs in [run 36120946740](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/36120946740) reported the same six cases. Two local replays reproduced all six failures. The source changes and documentation from `cd43b831a` establish both intentional contracts: independent bounded event compensation and surfaced provider errors. Only the two test files are corrected; production code and CI policy are unchanged. | Target and configuration | Passed | Failed | Skipped | | --- | ---: | ---: | ---: | | net48 Debug | 113 | 0 | 0 | | net10.0 Release, CI Coverlet settings | 113 | 0 | 0 | The selection contains `MonitoredItemManagerLifecycleTests`, `DirectoryObjectStateTests`, `DirectoryBrowserBoundedEnumerationTests`, and `EventCompensationReadinessHasFiniteDeadlineWithInfiniteSourceTimeoutAsync`. Both changed files pass formatting and analyzers through informational severity on both frameworks. Independent mutations restoring a non-cancellable compensation token and swallowing provider errors fail all four lifecycle cases and both browser cases, respectively. Both mutations were removed before the final passing runs and commit. The scoped command is `dotnet test tests\Opc.Ua.Server.Tests\Opc.Ua.Server.Tests.csproj` with `CustomTestTarget` and `-f` set to the selected framework, `--no-restore`, and `--filter "FullyQualifiedName~MonitoredItemManagerLifecycleTests|FullyQualifiedName~DirectoryObjectStateTests|FullyQualifiedName~DirectoryBrowserBoundedEnumerationTests|FullyQualifiedName~EventCompensationReadinessHasFiniteDeadlineWithInfiniteSourceTimeoutAsync"`. The modern run additionally uses `-c Release /p:CollectCoverage=true --collect "XPlat Code Coverage" --settings tests\coverlet.runsettings.xml`. These are scoped results, not a full-solution test or measured coverage-delta claim. Hosted checks for that commit were pending at publication. The fail-fast CI watch was stopped when the requested operation switched to review feedback; no current-head CI success is claimed. **Earlier master merge (`8b15ea6ef9fc2214ac8ec886f373bb3846dd6d6f`, base `b4136d8e9bab0df680f41d9ef985f519160534b1`):** The `TcpListenerChannel` conflict preserves both lifetime responsibilities: disposal clears the retained peer-certificate chain and unregisters the quota-scoped message-assembly scheduler. Both fields and all upstream deadline handling remain intact. The only additional manual source adjustment adds the parentheses required by the existing analyzer around the session-count fallback. Master advanced during validation. The initial transport merge `80303c434ce813cf75aa7fe54a6a32fe2aeafc30` was retained, and the newer base was merged normally in `8b15ea6ef9fc2214ac8ec886f373bb3846dd6d6f`, without rewriting history. The additional recovery, fluent-pipeline, and test-framework overlaps were inspected; the branch's cancellation, validation-error, and acknowledgement ownership fixes remain alongside the upstream changes. Full `UA.slnx` builds passed on net10.0 and net48 with **0 warnings and 0 errors**. The final scoped tests rebuilt against the committed tree `2074f8efcb61d8d9c6106e7d6f76ee941940e358`: | Target | Passed | Failed | Skipped | | --- | ---: | ---: | ---: | | net10.0 | 2,653 | 0 | 0 | | net48 | 2,529 | 0 | 7 | The net48 skips are existing platform exclusions: six RSA_DH_AesGcm nonce cases and one ClientWebSocket certificate-validation case requiring .NET 7 or later. The selection covers transport cleanup, reassembly budgets/deadlines/admission, peer-chain ownership, server session binding, fluent lifecycles and source generation, reconnect and keepalive, transfer acknowledgements, Publish queues, and JSON encoding. Modern-.NET WSS transport and real-server integration cases also pass. The resolved file passed solution-relative formatting and analyzer verification through informational severity on both frameworks. These are scoped test results, not unfiltered full-suite or measured-coverage claims. Hosted CI was not monitored or repaired as part of this merge-only operation. Selected local results below are overlapping validation runs, not distinct-test totals. **Earlier review-feedback batch (`cd43b831a8f11de3f7d65c3681712a61e3c669a5`):** The final net10.0 scoped selection passed **471 tests, with no failures or skips**. The final net48 solution-root selection passed **439 tests, with no failures and one unchanged platform skip** (`ValidateServerCertificateFallsBackToTlsPolicyErrors`, which requires newer ClientWebSocket certificate-validation support). Both full nonincremental solution builds completed with **0 warnings and 0 errors**, and all **40 changed C# files** passed informational-level formatting/analyzer verification. Both frameworks validated the identical final tracked diff. The focused matrix covers every one of the 20 current review threads, including real mid-frame WSS cancellation, stale admission contexts, directory continuation ownership, multi-element Browse/Translate failures, and the readiness drain race. The net10.0 solution-root test command has a separate pre-existing runner conflict: three unrelated AOT projects reject the VSTest target under Microsoft.Testing.Platform 2.4.0 and SDK 10. Per-project net10.0 results are recorded separately; no runner migration, test exclusion, assertion relaxation, or CI-policy change is included. These are bounded selections, not unfiltered full-suite or measured coverage claims. At publication of `cd43b831a`, hosted CI was not green: the `OPCFoundation.UA-.NETStandard` check reported `ACTION_REQUIRED`, AppVeyor reported `FAILURE`, and the two signed-build jobs were in progress. No CI jobs were rerun, cancelled, approved, or repaired as part of that feedback-only operation. **Classic transfer correction (`f425376fca045794122e19aaf9cafebc4a717d91`):** The preceding head's [macOS net10.0 shard 25/30](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/35980326299/job/107570512458) failed `TransferSubscriptionTest(DisconnectedRepublish).TransferSubscriptionAsync(True,False)` at the static-node count assertion: expected 10, received 20. The trace contains a successful, explicitly requested republish of retained sequence 1 and a fresh initial-value batch at sequence 3, not duplicate dispatch of one response. Controlled lost origin acknowledgements reproduced that incorrect test expectation on both the preceding PR head and exact master base `a2539cb53dd1666cef856f33812d7a8830d68909`. The maintainer explicitly authorized correcting this existing issue in the PR. Retained replay and requested initial values are independent, as specified by [OPC UA Part 4, TransferSubscriptions](https://reference.opcfoundation.org/specs/OPC-10000-4/5.14.7). The reason that the original hosted run retained its particular origin message remains unproved. The expanded full-suite run then exposed a **real production race**, not just a count assertion: Session queued acknowledgements for all transferred available messages before the asynchronous Republish responses arrived. Publish could delete those messages first, causing `BadMessageNotAvailable`. A gated real-Session/classic-engine regression failed both replay-enabled cases on the original implementation while both no-replay controls passed. The ownership fix passes all 24 combinations of empty/consecutive/unsorted-gapped queues, replay preference, sequential delivery, and a reentrant preference change. All 40 normal real-server transfer cases pass on both frameworks. Independent mutations suppressing actual replay and duplicating observed sequence records each fail the intended assertion; both mutations were removed. The full classic suite passes on both frameworks. Hosted CI was pending when that correction was published; the earlier failed hosted job was not rerun for a lucky pass. **Local net48 caveat:** the full Client run timed out in the pre-existing fake-clock test `ReconnectAsyncSwapsFaultedLeaseEntryAsync`, which does not exercise subscription transfer. Its unchanged isolated replay passed, but that does not resolve the full-suite timeout. No unrelated code or assertions were changed, and a clean full Client net48 run is not claimed. | Scope | net10.0 | net48 | | --- | --- | --- | | Full Client.Tests after transfer acknowledgement correction | 2,549 passed | 2,552 passed, 1 timeout failure noted above, 1 existing platform skip | | Full classic-subscription project after transfer correction | 57 passed, including all 40 normal transfer cases | 57 passed, including all 40 normal transfer cases | | Gated transferred-message acknowledgement ownership | 24 passed, included above | 24 passed, included above | | Saved-session consumer of the shared subscription helper | 40 passed | 16 passed, 24 existing unsupported-security-policy skips | | Audit regression matrix | 2,339 passed | 1,934 passed, 34 existing skips | | Full Client.Tests after review follow-ups | 2,520 passed | Focused selections only | | Full Client.Tests after CI recovery correction | 2,523 passed, CI coverage collector | 2,527 passed, 1 existing platform skip | | Latest merged-head recovery/redundancy selection | 139 passed | 139 passed | | Full Client.Tests after merged-head net48 correction | Focused selection above | 2,529 passed, 1 existing platform skip | | Lifecycle and actual-dimension regressions | 146 passed | 146 passed | | Focused recovery follow-ups | 7 passed | 7 passed | | WSS-related fixtures | 93 passed | Not applicable | | CI History integration corrections | 28 passed | 28 passed | | CI alias, dispatcher, and explicit-atomic controls | 63 passed | 63 passed | | Full WoT sample project with CI coverage collector | 29 passed | Not applicable | | Trust-list effects and count-only summary contract | 22 passed | 22 passed | | Virtual-node feedback, node-manager, dispatcher, and lifecycle selections | 661 passed, 63 existing not-executed cases; all 48 virtual-node cases passed | 661 passed, 63 existing not-executed cases; all 48 virtual-node cases passed | | WSS wire-handle exhaustion, late replies, and Cancel translation | 9 passed | Not applicable | | Synchronous node-manager disposal and related service/lifecycle suites | 1,037 passed, 64 not-executed cases from existing fixture assumptions; all 93 sampling/disposal cases passed | 1,037 passed, 64 not-executed cases from existing fixture assumptions; all 93 sampling/disposal cases passed | - Full solution builds passed on net10.0 and net48 during the audit work. The subsequent full non-incremental net10.0 build also completed with zero warnings/errors. - Modified source/test formatting and analyzer checks passed through informational severity. - Original composed recovery regressions passed repeated runs on both frameworks. Independent mutations removing Publish cancellation, bypassing the recreate drain call, or ignoring ready-gate cancellation each failed the intended assertion; all mutations were restored before passing runs. - The latest recovery correction passed 100 executions of each original restoration-deadline scenario with unchanged assertions. Restoring premature update admission independently made both strengthened handoff cases fail; the correct retention behavior was restored afterward. - Startup-event and cancellation-delivery timing probes reproduced the subsequent net48 failures, then passed 100 executions of each of the three scenarios with the fixes. All timing probes and repetition were removed. The permanent timeout-order regression also went RED on the original cancellation cleanup without sleeps or scheduling amplification. - Final post-format recovery/subscription/engine checks passed all 151 selected cases on net10.0. The first local full net48 attempt aborted in `clr.dll` with native access violation `0xc0000005`; an unchanged-assembly full diagnostic rerun passed as recorded above. That native host crash has not been attributed or claimed fixed. - A minimal replay with the same CodeQL CLI and query-pack versions reproduced the custom-count alert and confirmed that the standard span-length projection is clean while direct sensitive-value and raw-identifier logging controls remain flagged. The source was already logging only a count; no identifier or certificate disclosure was established at the reported location. - The virtual-node feedback regressions first reproduced five error-preservation/cache failures; expanded service-dispatch cases exposed two additional whole-batch failures before the operation-boundary fixes. The final selections pass on both frameworks. Genuine misses remain cached, successful results remain reusable, failed validation allows same-cache retry, and request cancellation still stops the batch. - Three deterministic synchronous-manager disposal regressions reproduced early resource release and lifecycle calls reaching a manager during disposal. The fix passes direct and adapted calls, a sampling-worker join after the last operation exits, reentrant shutdown, cancellation and failure paths, repeated disposal, and exact rejection across 39 service/lifecycle entry points both during drain and after cleanup. The original outside-node-lock disposal regression remains and now requires the worker to observe closed admission. - Prior-head CI on `c985f8bded77a60449d9cd381c8a004d8590ccf6` completed with **168 passing checks, one expected release-package-publication skip, and no failures or pending checks**. All test shards, signed builds, container builds, the coverage check, and the CodeQL PR check passed. Hosted CodeQL analysis `1829262912` completed successfully and marks count-flow alert 385 fixed. This is evidence for that head, not a claim that subsequent review follow-ups have completed hosted CI. ### Session-test diagnostic memory Complete browse listings are streamed to unique NUnit attachments rather than retained in inline NUnit/VSTest output. Detached regression-test results are cleaned up without deleting runner-collected attachments. - Two covered session-suite runs each completed with 782 passed and 379 existing skips. Peak working set fell from 4.86 GiB to 2.44-2.78 GiB. - TRX output shrank from 303.1 MiB to approximately 40.7 MiB, retaining all 46 complete browse attachments. - Seven output regressions cover exact content, empty/single/large listings, disabled/enabled output, bounded inline summaries, and unique filenames. The separate Azure build-18452 VM deprovisioning cause remains unproved without provider logs. The memory measurements do not establish its cause. Unfiltered local `UA.slnx` tests, including the long stress/soak suites, have not been run on both frameworks. A local coverage delta was not collected; builds and focused tests are not substitutes for those claims. ## Checklist - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [ ] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] All checks on the latest review-follow-up head have completed successfully; prior-head CI passed as recorded above. - [x] I have addressed **all** PR feedback received. Tests have been added and pass as recorded above; the hosted coverage check passed on the prior head identified above. The combined tests/coverage-increase checkbox remains open because no local coverage delta was measured. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 68f26810-22b8-476f-abcd-0a8c82d6e208 |
||
|
Marc Schier
|
57c0a32ed4 |
Fix incomplete-message buffering and channel cleanup (#4522)
# Description Fix incomplete-message buffering, channel cleanup, session resource accounting, and enforcement of existing transport admission settings. - Share a targeted `ChunkReassemblyBudget` across server listeners. The reference default is 64 MiB, with a 32 MiB global occupancy threshold for sessionless intermediate chunks. This is priority headroom for activated sessions, not a guaranteed bootstrap partition. Single/final chunks and unrelated response/client buffers remain outside this budget. - Retain precise incoming/final payload quotas, valid Abort handling without channel teardown, terminal late-final rejection, and exactly-once buffer/reservation cleanup. - Use one assembly timer per quota/clock scope, asynchronous cleanup, a fixed deadline, a 30-second fallback for non-positive lifetimes, and best-effort ERR `BadTimeout` before closure. - Apply negotiated TCP/WebSocket/Pipe receive limits and size pipe rentals to actual chunks. - Add the optional `ISessionBindingProvider` and immutable `SessionBindingContext` seam. Managed membership counts distinct committed sessions, not responses, and handles reactivation, transfer, closure, expiry, restoration and shutdown. Read-only snapshots require token/channel/security validation and never replace service authorization. - Wire default and explicitly injected binding providers through raw TCP, Kestrel and UACP WebSockets; preserve existing callback/manager interfaces and direct construction. - Enforce existing `ConnectionRateLimiter` and channel capacity settings on Kestrel TCP and UACP WebSocket admission, with physical-transport leases surviving reverse handoff and releasing on cancellation, close, failed attachment and stop. HTTP/JSON requests are not treated as UASC channels. - Add regression coverage and documentation. Stabilize the CI cancellation test by waiting for the resolver to enter before canceling. This is the foundation only. New Balanced/FairShare/TrustedReservations policies, tenant isolation and guaranteed bootstrap reserves belong in the separate stacked follow-up, not this PR. ## Related Issues Fixes #4521. ## Validation | Scope | Result | | --- | --- | | Core transport, budget, binding-context and DI tests, net10.0 | 717 passed | | Core transport, budget, binding-context and DI tests, net48 | 664 passed, 6 platform-specific skips | | Session bindings, transfer/security/expiry/restore and hosted DI, net10.0 | 69 passed | | Same focused Server suite, net48 | 69 passed | | ServerRedundancyHandlerTests, Release/net48 | 39 passed | | ConsoleReferenceServer net10.0 | Build succeeded, zero warnings/errors | | Server and HTTPS binding netstandard2.1 | Both builds succeeded, zero warnings/errors | The new admission fixtures also passed five repeated runs after correcting asynchronous connection-context disposal. Full-solution validation, coverage comparison and CI for this update remain pending. ## Checklist - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [ ] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [ ] I have addressed **all** PR feedback received. |
||
|
Marc Schier
|
138086863b |
Fix audit findings and review regressions across the OPC UA stack (#4503)
## Scope Audit fixes and review follow-ups across the OPC UA Core, Client, Server, transport, and redundancy components. This includes implemented Tier 1, Tier 2, and Tier 3 work; it does not claim that every item in the 244-finding audit backlog is complete. The review batch fixes mandatory full-key record protection and owned plaintext buffers; transactional pending-certificate claims; shared-channel nonce and certificate ownership; discovery and token-policy validation; reconnect/callback ordering without changing unbounded buffering defaults; server permissions, filters, monitored-item lifetime, and alarm topology; historian paging, retention, and annotations; atomic user/password metadata; and canonical file identities with bounded alias and role refresh workers. ## Review-batch verification at `ab686ccae` The following results come from fresh test builds using `dotnet test`, `--no-restore`, `-m:1`, and an explicit `CustomTestTarget`. They replace the earlier description's library-build/`--no-build` totals, which did not establish the state of the merged test sources. | Scope | Target | Passed | Skipped | |---|---|---:|---:| | Core review regressions | net10.0 | 827 | 0 | | Client and lifecycle regressions | net10.0 | 630 | 0 | | Server review regressions | net10.0 | 1970 | 0 | | Redundancy Server / Client / PubSub | net10.0 | 276 / 10 / 11 | 0 | | Sessions / file-and-alias / streaming / HTTPS-WSS integration | net10.0 | 8 / 21 / 30 / 20 | 0 | | Core review regressions | net48 | 786 | 39 | | Client recovery and policy regressions | net48 | 54 | 0 | | Server ownership, aliases, queues, and user-state regressions | net48 | 137 | 0 | Every listed run exited successfully. The 39 net48 skips are existing platform/capability cases. New key-agreement tests assert unavailable-policy behavior on unsupported targets rather than adding skips or pretending those targets performed an unsupported handshake. Additional exact node/permission/filter/alarm checks passed 13 Core and 46 Server cases. The PR-scoped formatter verification covers 26 projects and 389 non-generated C# files. A Roslyn syntax audit covers 53 changed interfaces and 263 members, with no missing parameter/type-parameter/return contracts and a 120-column limit for their declarations and documentation. No full-solution, all-target-framework, or NativeAOT matrix is claimed from these local runs; the new CI run must establish its own result. ## CI follow-up at `e1bc3ac3b` The follow-up fixes the failures reported by Azure build 18400. TLS certificates retain configured DNS names, including mixed-case URI schemes, without relaxing hostname validation. Reverse-WSS fixtures use the DNS identity in the client certificate. Reconnect completion is published only after cycle cleanup, so a subsequent reconnect cannot join an old completed cycle and retain old certificates. Certificate-alarm roots now support event subscriptions. The pump sample has one continuous notifier path, with no duplicate shortcut. Historian and aggregate fixtures share the provider's retention clock. Alarm collection lets the server expire idle Publish requests rather than canceling only the client wait. The corrected server, WoT, and managed-session fixtures exercise valid requests and initialized lifecycle contracts. Downlevel crypto tests preserve real credential and key-ownership assertions without enabling unsupported raw-ECDH policies or adding skips. Fresh local evidence after integration includes: | Scope | Target | Passed | Skipped | |---|---|---:|---:| | Client lifecycle / channel ownership / builder fixtures | net10.0 and net48, each | 96 / 103 / 123 | 0 | | Combined server lifecycle, events, aggregates, crypto, and contracts | net48 | 358 | 0 | | Full lifecycle plus certificate-alarm and aggregate fixtures | net10.0 | 116 | 0 | | Final server-contract and history-routing delta | net10.0 and net48, each | 84 | 0 | | Subscription / XRegistry / WoT event regressions | net10.0 | 7 / 2 / 3 | 0 | | Full Sessions project | net10.0 | 780 | 379 | | Full ComplexTypes project / downlevel resolver fixtures | net10.0 / net48 | 2854 / 15 | 0 | | Full WoT project / downlevel structured-write fixtures | net10.0 / net48 | 1496 / 27 | 0 | | TLS domain-name regressions | net10.0 and net48, each | 21 | 0 | All listed runs exited 0. The Sessions skips come from existing endpoint and policy capability guards; they are not counted as passes. No skip was added. Worker validation also covered full DI (426 passes), full History (540 passes, 21 existing skips), 38 exact downlevel event/history failures, and 60 repeated certificate-rotation race cases. The final crypto fixtures passed 73 cases on each target with the unchanged leak detector; a temporary premature-disposal mutation made all three borrower tests fail and was then removed. All 29 C# files in this follow-up pass included-file formatting at info severity across 12 projects. The complete PR interface audit now covers 54 interfaces and 271 members with zero missing contracts or line-width findings. No full-solution or NativeAOT matrix is claimed. New-head CI has started and remains authoritative for the remote result. ## Test-result gate follow-up at `4b6e1203a` Azure build 18401 reported the net48 Client job as successful despite a test-host crash after only 264 passing tests. Its old evaluator counted per-test failures but ignored the failed run verdict. This was a partial run, not a benign exit after the suite had completed. The shared result evaluator now rejects aborted, errored, incomplete, malformed, or inconsistent TRX reports. Each file must describe a completed run, so a passing file cannot mask another file's failure. It retains support for completed runs followed by a benign nonzero host exit, existing skipped-result conventions, and no-TRX MTP jobs. All 78 regression cases pass after integration using Windows PowerShell 5.1. Real completed Sessions and Client reports also pass. The process-test fixture follows the existing NET10_0-only tools-test pattern; its net48 build is not counted as a test run. Formatting passes for the added test file. The Client host crash itself has not reproduced locally. The isolated fixture passed 59 tests, and two fresh full fast-CI selections each completed with 2,392 passes and three existing skips, including a run with crash collection enabled. The CI merge tree is identical to the tested source tree. No speculative runtime change was made, and the original partial CI run is not claimed as a full-suite pass. Azure build 18402 has now completed the full net48 Client run: 2,392 passed, three existing console skips, and a successful test-host exit. The corrected evaluator independently validated outcome=Completed. Across the matrix, 119 test-project jobs succeeded; Sessions net10.0 was canceled when the remote provider deprovisioned its runner, so the overall Azure pipeline was canceled. That cancellation is not claimed as a code failure or a successful run. CodeQL and three Docker image jobs were still running at the latest check. The migration discovery failure remains separate. ## Documentation follow-up at `57ed02bea` The environment blocker has been removed by the maintainer. The three bundled migration references now exactly match the canonical documents after deterministic link rewriting; all 15 bundled documents pass `validate-migration-plugin.ps1`. The canonical migration prose was not rewritten. `FileSystemClient.md` now documents that extended Windows roots preserve literal trailing-dot names, completing the filesystem review follow-up. The filesystem thread has been replied to and resolved after publication. The review pass now has 108 of 110 threads resolved. The two explanation-only threads remain open as requested; no unfinished change request is being represented as resolved. ## Cross-platform follow-up at `0d9d936a8` With migration discovery restored, the Linux and macOS test matrix exposed two stale durable-subscription expectations. The expected value 83,442 encoded the old 32-bit overflow when converting 3,600 hours to milliseconds. The existing overflow-safe server correctly returns 14,400,000 lifetime intervals at a 900 ms publishing interval. Only those two expected constants changed; production behavior, negotiated-hour checks, and all other assertions remain unchanged. The exact local lifetime selection reproduced both failures before the correction. The full durable suite now passes 36 cases with 11 existing skips on each of net10.0 and net48. All four lifetime cases executed and passed on both targets, and included-file info-severity formatting passes. ## Session shutdown follow-up at `8daf7f091` A late net48 Sessions failure exposed a shutdown race. Canceling an in-flight keepalive read can produce a transport `ServiceResultException`. The retiring worker treated its own cancellation as proof that the peer was unavailable, so `CloseAsync` or `DisposeAsync` could skip the server-side `CloseSession` request. The worker now ignores that service error only when its own token is canceled; genuine peer errors still update liveness. The full integration fixture reproduced the original error-code mismatch. Four deterministic mock-transport cases then produced three failures and one passing genuine-error control before the fix. They verify successful close, returned server error, asynchronous disposal, and unchanged behavior for a real keepalive failure. Fresh related runs pass 75 client and 101 integration cases on each of net48 and net10.0, with no skips. Ten repetitions of the formerly flaky full net48 fixture pass all 1,010 cases. All three changed files pass info-severity formatting, and temporary diagnostic instrumentation was removed. ## Atomic listener snapshot follow-up at `82dc22b5d` Ubuntu Core CI exposed an unsafe formatter rewrite in TCP listener disposal: `ConcurrentDictionary.ToArray()` had become a collection spread. A concurrent channel removal between the spread's count and copy operations could leave a default entry, causing a null dereference during disposal. The listener now uses the dictionary's atomic `Values` snapshot and disposes those channels outside the listener lock. Deterministic regressions close zero, one, or all channels at the copy boundary. Before the fix, the one/all-removal cases reproduced the exact null dereference; the unchanged-count control passed. All 23 admission/listener cases now pass on net10.0 and net48, and the full fast Core net10.0 suite passes 5,267 cases with 88 existing platform skips. Both changed files pass info-severity formatting. A bounded audit of the formatter's other snapshot rewrites found no further verified unsafe concurrent-registry copies. ## Shutdown-test observation follow-up at `58403ee8b` The net48 shutdown-order test raced its own observation: it read `m_publishWorkerTask` after starting shutdown, which can legitimately drain the worker and clear that field immediately. The test now captures the worker before shutdown starts and confirms it is initially running. It still requires the worker to finish while shutdown remains blocked by the held manager semaphore. Production behavior is unchanged. A temporary scheduling perturbation reproduced the exact null-field failure. The corrected capture passed 100 perturbed iterations; both the temporary repeat and yield were then removed. The final 126 subscription/lifecycle cases pass on each of net48 and net10.0, and formatting passes. A complete local Sessions net10.0 run also passed 780 cases with 379 existing capability skips in 7m20s after the earlier keepalive correction. ## Retirement and alias-startup follow-up at `1a048ff39` The shadow-generation test now observes its documented asynchronous cleanup through the existing bounded helper. Removing the replacement does not join a background drain that already claimed the retired generation. The test still checks live-registration removal immediately and requires the old node to disappear; no timeout or production lifecycle behavior changed. All 81 lifecycle cases pass on each of net10.0 and net48. Initial alias-node materialization now retries one timed-out, read-only snapshot query before creating nodes for that root. The retry is logged through source-generated logging. Each query retains the original shared 100 ms deadline; repeated timeouts, other errors, and cancellation still abort startup. Client searches and live refresh do not gain this immediate retry. Fault-injection tests cover success after one timeout, a persistent timeout, a different store failure, and cancellation. All 187 related alias/hosting cases and the original certificate-rotation integration case pass on each target; the positive test records exactly one retry warning. The alias documentation now accurately describes the shared request budget. All changed C# files pass formatter checks, and the complete 54-interface/271-member documentation audit passes. Fresh CI is running on `1a048ff39`; the previous completed matrix exposed only the two failures corrected in this batch and their dependent summaries. ## Sample endpoint allocation follow-up at `d08a94925` The strong-cluster test harness allocated OPC UA and Raft endpoints in separate released-port batches. Uniqueness was enforced only inside each batch, so the same OS-assigned port could appear in both protocol groups. The harness now requests one distinct batch for both groups and splits it before launching any replicas. Deterministic one- and three-replica allocation cases reproduced overlapping groups with the old plan and pass with the corrected plan. The complete 17-case short-haul sample suite passes without skips; all failover, historian-continuation, and service assertions remain unchanged. Both changed files pass info-severity formatting. This prevents intra-cluster self-collisions but does not claim to reserve released ports against unrelated external processes. ## Code-review follow-up at `085fd9636` A review of the complete branch diff found five defects this PR introduced. All are now fixed, each in its own commit. `RelatedTo`'s recursive-to-iterative rewrite defaulted the optional `includeTypeDefinitionSubtypes` and `includeReferenceTypeSubtypes` operands to `false`. Part 4 §7.7.4 specifies both default to `true`, so a filter that omitted them silently stopped matching subtypes. Dynamically created key-credential nodes derived their namespace from the containing folder, which is the standard `i=18155` node in namespace 0 — the namespace reserved for the OPC UA standard address space; they now resolve the server-owned namespace and fail with `BadInternalError` rather than guessing index 1. The retained audience-less `CreateTokenData` overload produced tokens that verification always rejects, so it is now `[Obsolete]` and throws instead. A failed reference to a remote parent left the new node registered with a one-sided inverse reference, and is now rolled back. An existing `.pfx` is restricted before it is rewritten, rather than after its new key bytes are on disk. One further reported finding was rejected after investigation. `JwksIssuerKeyResolver` deliberately does not dispose retired key snapshots: `RetiredKeysAreCollectibleWhileResolverRemainsAliveAsync` proves the resolver retains no retired snapshot, and `ReturnedKeysSurviveRepeatedRefreshAndResolverDisposalAsync` requires already-returned keys to keep verifying across refreshes and resolver disposal. Adding disposal would have broken that tested contract, so the exploratory change was reverted. Validated on net10.0 and net48: 256 Core and 633 Server cases pass, with one pre-existing net48 skip. The build reports zero compiler and analyzer diagnostics, all eight included-file info-format checks pass, the contract audit covers 58 interfaces and 310 members with no findings, and the 15 bundled migration documents validate. All 122 review threads are resolved; the key-credential namespace thread was reopened to retract an earlier incorrect answer and then resolved against the real fix. ## Review-body follow-up at `da0d59cfb` The re-review at `8daf7f091 + 82dc22b5d` listed four items in its summary that had no inline thread of their own, so they had not been picked up. All four are now fixed. Both event-dispatch permission gates had been tightened from not-Bad to `IsGood`, which silently dropped events whose permission verdict is Uncertain; the data-change path in the same file still treated Uncertain as deliverable, so the two had diverged. `EventSourceRegistry` only cleared `ConsecutiveFailures` when a source was deactivated, so a source that had backed off to the 30-second cap kept that delay for the rest of the process even after starting successfully. `UserNameIdentityTokenHandler` built a temporary issuer chain whose `Add` calls take independent certificate references, and never released them. The fourth was the receiver-side sequence rollover, which the reviewer had correctly re-flagged as still open despite its thread being resolved. `m_sequenceRollover` was set permanently, so the second legal wrap on a long-lived channel was rejected and faulted the channel along with the transport-reconnect OPN. OPC 10000-6 does not cap a SecureChannel at one wrap. The flag is now cleared once an accepted number has traversed the normal range, which keeps rejecting a replayed low number that never left the rollover window. Validated on net10.0 and net48: 636 / 593 Core cases and 135 / 134 Server cases pass, including the new `UaSCBinaryChannelSequenceRolloverTests` and the Uncertain-permission delivery cases. The build reports zero compiler and analyzer diagnostics and the changed files pass included-file info-format checks on both targets. The previous head `085fd9636` was fully green: 279 checks passed and Azure build 18414 succeeded with 137 of 137 jobs, after the platform automatically retried a job whose NuGet tool download had hit a transient TLS error. ## Reported issues #4508 and #4509 at `6ad1ecf16` **#4508 (V2 subscription recreate deadlocks against a faulted transport) was not yet fixed and is now closed by `bac260a06`.** Both drain sites the report identifies were still unbounded. A publish worker that started its request before the quiesce parks inside the service call on the channel ready gate while holding an active publish request; the recreate that would make the channel ready is the very operation awaiting the drain, so the wait never ends. The drain now cancels in-flight attempts through a per-attempt token linked to the worker token. An aborted worker rolls its acknowledgements back, releases the request and returns to the paused park instead of terminating, so it resumes when the caller leaves the quiesce. Publish ingress stays quiesced for the caller's operation, so `DropPendingForSubscription` keeps the guarantee the report asks to preserve and no blind timeout is added. The new regression parks a worker on a cancellable gate that never completes on its own: it reproduces the hang for the full 30-second budget before the change and passes after it. **#4509 (identity provider fails the initial connect) was already fixed for its primary symptom.** `HandleConnectAsync` materializes the provider's identity through `CreateIdentitySelectionContext` before the first `CreateAsync`, so Anonymous is no longer substituted against a UserName-only endpoint. `6ad1ecf16` adds the two remaining details from the report: the enabled-policy set now includes the negotiated channel policy, so a client cannot reject the policy it already accepted as `NotEnabledByClient`; and when discovery has not populated the endpoint's token policies the connect keeps the configured identity so the attempt fails at the transport layer for the reconnect policy to retry, instead of failing at the identity layer. Validated on net10.0 and net48: 477 / 473 client session and subscription cases pass, the build is free of compiler and analyzer diagnostics, and the changed files pass included-file info-format checks on both targets. The full `Opc.Ua.Sessions.Tests` suite also passes locally on net10.0 with 782 passed and 379 pre-existing capability skips. GitHub checks are green on this head: Build and Test .NET 10.0, Docker Sample Images CI, and NuGet Publish all succeeded. Azure build 18417 completed 134 of its 135 jobs successfully; its net10.0 Sessions job was cancelled by the hosted-agent provider ("Received request to deprovision"), the same infrastructure flake that hit build 18409 earlier on a head predating these changes. That job is not claimed as passing, and the local full-suite run above covers the same project. The previous head `da0d59cfb` completed Azure build 18415 with every job succeeded. ## Review follow-up at `a08f0c12b` Eleven new review threads arrived, nearly all documentation. Before acting on each "this is new in 2.0" remark I checked the API against `master378` rather than assuming, because a genuine 1.5 API still needs its migration entry. `ManagedSession`'s server-redundancy refresh had two hard-coded two-second bounds. The new `ServerRedundancyOptions` exposes `RefreshTimeout` for the metadata read and `PeerDiscoveryTimeout` for each peer lookup, defaults both to the previous two seconds, validates that a bound is positive or `Timeout.InfiniteTimeSpan`, and flows through `ManagedSessionOptions`, `ManagedSession.CreateAsync`, the fluent builder, and `DefaultServerRedundancyHandler`. `AsyncCustomNodeManager` and `KeyCredentialBridgeAuthenticator` are both absent from `master378`, so their obsolete members had no compatibility constraint and are removed rather than carried: `AsyncCustomNodeManager.ReadInitialValue`, the audience-less `CreateTokenData` that only threw, and the audience-less `CreateProof` that produced a proof no server accepts. `CustomNodeManager2.ReadInitialValue` and the `MonitoredNode2` migration section are genuine 1.5 APIs and are untouched. The documentation now states master's behaviour instead of the delta from 1.5.378, with each moved item already covered under `docs/migrate/2.0.x/`. `IHistorianAnnotationProvider` is described by its annotation-time key rather than as "legacy" — it is new in 2.0, so the label implied a compatibility constraint that does not exist. The KeyCredential bridge payload's `"version"` field is explained as a vendor-extension format version with no OPC UA counterpart, and its version-1 upgrade narrative is gone. `CertificateManager.md`'s migration walkthrough is replaced by the resolver API it targets. `NodeManagers.md`, `Sessions.md`, `AsyncServerSupport.md`, `Certificates.md`, and `Profiles.md` no longer announce features relative to 1.5. `Benchmarks.md`, `WhatsNewIn2.0.md`, `DeveloperGuide.md`, and `docs/migrate/` keep their 1.5.378 references, which are their subject. CI then exposed a defect this batch introduced: five fixtures resolve `ManagedSession`'s private constructor by its exact positional parameter types, so the added parameter compiled cleanly but made `GetConstructor` return `null` at run time. `7df1d5c40` updates every site; `Redundancy.Client` passes 126 of 126 and the affected client fixtures 112 of 112. The BouncyCastle question is answered and tracked in #4510: `Opc.Ua.Core` already project-references `Opc.Ua.Security.Certificates`, which already carries the conditional BouncyCastle reference, so the raw-ECDH polyfill belongs there rather than as a new Core dependency. The `IUserDatabase` metadata question is addressed to another maintainer and is deliberately left open. ## Reported issue #4509 follow-up at `5d4d13d63` The reporter verified `6ad1ecf16` and found one case still failing, correctly. The empty-token-policies branch kept the configured identity so a server that is down surfaces a transport failure rather than an identity error. A provider-only configuration has no configured identity, so it kept `null`: `OpenCoreAsync` turned that into Anonymous and `OpenValidateIdentity` rejected it with `BadIdentityTokenInvalid`, silently ignoring the configured provider. The branch now requires a configured identity to defer to. Without one the provider is consulted, so its own `BadIdentityTokenRejected` names the offered and rejected policy set instead. Two regression tests cover both configurations; the first fails against the previous behaviour by reaching the session factory with a `null` identity, which is the exact step that produced the reported error. That branch had shipped without a test, which is why the case was missed. ## Review and CI status at `5d4d13d63` **All actionable review threads are resolved: 132 of 133.** The single open thread is a question the reviewer addressed to another maintainer about moving user metadata into `IUserDatabase`; it is left open for them rather than resolved silently. Every resolved thread carries a reply naming its fix commit. **Latest-head CI is green on `5d4d13d63`: 279 checks passed, zero failed, zero pending**, with one conditional signed-package publishing job skipped. Local validation for this batch: the full `Opc.Ua.Client.Tests` suite passes 2,430 of 2,430 on net10.0, `Opc.Ua.Redundancy.Client.Tests` 126 of 126, and `Opc.Ua.Server.Tests` 590 of 590 for the affected selections. Both net10.0 and net48 build clean for `Opc.Ua.Client`, `Opc.Ua.Server`, and the changed test projects, and all changed files pass included-file info-severity formatting on both targets. The PR remains open; no merge was performed. As stated at the top, this does not claim that every item in the 244-finding audit backlog is complete. ## Latest CI follow-up at `b6f2dfa43` Azure build 18442 exposed a net48 compile regression in the Kafka adapter: `RuntimeHelpers.GetUninitializedObject` is unavailable on .NET Framework. Commit `2cf1e0eed` now uses normal `ProducerMessage` construction on `NETFRAMEWORK` while retaining reflection-based construction for modern targets that can load a different Dekaf runtime asset. The exact net48 CI selection passes 92 of 92 tests, and the netstandard2.1 adapter guard selection passes 11 of 11. After merging the latest `master`, GitHub Actions run 35643592139 is green. The initial Ubuntu Core.Schema job built successfully and passed 160 of 160 tests, then hit an external 403 while finalizing its test-result artifact; its failed-job rerun completed successfully. Azure build 18451 was canceled when its remote provider deprovisioned the net10.0 Sessions runner (`Received request to deprovision: The request was cancelled by the remote provider`). Azure attached no test task log or source error, and the canceled parent left that child check stale as in-progress. No source workaround, test skip, or check relaxation was made for either infrastructure failure. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 68f26810-22b8-476f-abcd-0a8c82d6e208 |
||
|
romanett
|
aee30e9bf6 |
Echo the RequestHandle in ServiceFaults for requests that cannot be decoded (#4488)
# Description OPC 10000-4 §7.33 (ServiceFault): *"The requestHandle in the ResponseHeader should be set to what was provided in the RequestHeader even if these values were not valid."* When the decoder rejects a request, for example a string argument above `MaxStringLength` (`BadEncodingLimitsExceeded`), the server answered with a ServiceFault carrying RequestHandle 0 and no Timestamp. The OPC Foundation CTT reports this: *"CallResponseHeader.RequestHandle does not match expected RequestHandle. Expected: 18 Received: 0"*. It was seen during the CTT GDS triage of #4481, with a FindApplications ApplicationUri of MaxStringLength + 10 % characters. ### Changes - **`RequestHandleReader`** (new, internal, `Opc.Ua.Core`) reads the RequestHandle best effort from a message that could not be decoded: - binary: the encoding NodeId, then the leading RequestHeader fields AuthenticationToken (NodeId), Timestamp (UtcTime) and RequestHandle (UInt32); - JSON: `UaBody.RequestHeader.RequestHandle`, ignoring same-named members anywhere else. It returns 0 when the handle cannot be read. - **`TcpServerChannel` / `TcpListenerChannel`:** new `SendServiceFault` overloads take the RequestHandle, and faults now carry a response Timestamp. `ProcessRequestMessage` (decode failure, null request, discovery-only rejection) and the OpenSecureChannel error path pass the handle. This covers `opc.tcp`, the Kestrel TCP listener and UA-SC over WebSocket, which all use `TcpServerChannel`. - **`EndpointBase.CreateFault`:** - new internal overload with a fallback RequestHandle for a request that was not decoded; - the Timestamp is always set; - no longer dereferences a null RequestHeader. - **`HttpsTransportListener`:** - The JSON path and both JSON-over-WebSocket loops (`opcua+uajson`, OpenAPI) now echo the handle read from the payload. - Discovery-only faults use the handle of the decoded request. ### ⚠️ Behaviour change for reviewers: HTTPS binary A binary HTTPS request the decoder rejects used to get **HTTP 500 with a text body** and no ServiceFault. It now gets **HTTP 200 with a ServiceFault** carrying the decoder status and the echoed RequestHandle, the same as the JSON path in the same listener. Other exceptions in that path still return HTTP 500. OPC 10000-6 §7.4 does not specify how an HTTPS server reports an undecodable request. The existing test `SendAsyncReturnsInternalServerErrorForInvalidBodyAsync` was replaced by `SendAsyncRespondsWithServiceFaultForInvalidBodyAsync`. If HTTP 500 should be kept, that hunk can be reverted independently. WebApi REST (`WebApiServer`) already echoed the handle and is unchanged. ### Tests - `RequestHandleReaderTests`: - binary messages above MaxStringLength, with numeric and string authentication tokens, from a byte array and from a stream; - truncated and null input; - JSON field order, decoy `RequestHandle` members, wrong types and malformed JSON; - `CreateFault` fallback and precedence. - `TcpServerChannelBufferTests.UndecodableRequestServiceFaultEchoesRequestHandleAsync`: feeds a ReadRequest with a 640-character NodeId (MaxStringLength 64) through `TcpServerChannel`. It asserts that the sent ServiceFault has `BadEncodingLimitsExceeded`, RequestHandle 4711, the request id and a Timestamp. - `HttpsTransportListenerTests`: the binary ServiceFault for an invalid body, the echoed handle for an oversized binary request, and the echoed handle for an undecodable JSON request. - No unit test covers the two JSON-over-WebSocket loops; the test infrastructure has no WebSocket driver, and the change there is mechanical. Local results (net10.0): - `Opc.Ua.Core.Tests`: 4,669 passed, 0 failed, 88 skipped. - `Opc.Ua.Bindings.Https.WebApi.Tests`: 346 passed. - `Opc.Ua.Client.Tests`: 2,266 passed. `Opc.Ua.Core.Tests` also builds for net48. `dotnet format whitespace --verify-no-changes` passes on the changed files. ## Related Issues - Found while triaging the CTT GDS group in #4481 (`plans/ctt-issues.md` there, "Open server findings"). ## Checklist - [x] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [x] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [x] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [ ] I have addressed **all** PR feedback received. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
romanett
|
747f635518 |
Fix the Opc.Ua.Core defects from the server/core audit (#4475)
# Description
Fixes every `src/Opc.Ua.Core` item found by a twelve-reviewer audit of
`src/Opc.Ua.Server` + `src/Opc.Ua.Core` taken at
|
||
|
Marc Schier
|
7cec2d614d | Move repository files to the consolidated layout |