3 Commits

Author	SHA1	Message	Date
romanett	849840be5d	perf: publish/write/event load scenarios for the server load harness (#4817) ... Builds on #4815 (merged); rebased onto master. Two commits, touching only `perf/`. Extends `perf/ServerLoadHarness` and `perf/TraceAnalyzer` to profile publish-, write- and event-heavy server load that mirrors the explicit Sessions `LoadTest`. Built for the publish/write load investigation: [findings report](https://claude.ai/artifact/BWuEEPcvvbTAbjY44mJFsa). ## New options | Option | Scenario | Meaning | |---|---|---| | `--scenario events` | | Event subscriptions on the Server object; a server-side `EventGenerator` (in `EventWorkload.cs`) reports events, driven by the server child's new `events <rate>` / `events stop` stdin commands. | | `--event-rate` | `events` | Events/s the server reports; 0 = as fast as one thread can. | | `--writers N` | `sub` | Dedicated writer sessions in addition to `--sessions` (0: the first subscriber session writes, as before). | | `--write-interval 0` | `sub` | Closed-loop writers instead of a timer. | | `--write-batch` | `sub` | Values per Write; writers rotate through the nodes in slices. | | `--node-type all` | `sub` | The LoadTest's Int16…UInt64 mass node set (up to 600 nodes). | | `--queue-size` | `sub`, `events` | Monitored item queue size (default 1 / 10000). | | `--publish-requests` | all classic | Minimum outstanding Publish requests per session. | Also: - `Workload.Secondary` / `OnServerReport` hooks and a `per-op2:` result line: server cost per written value (`sub`) or per generated event (`events`). - TraceAnalyzer `threads` mode: busy share of each thread with its main roles (finds serialized stages). - TraceAnalyzer `cpu <trace> <top> <focus>` additionally lists the focus frame's direct callees. - README rows and paragraphs for all of the above. Conflicts with #4815's rewrite (native workers, `suspend`/`samples` modes) were resolved so #4815's code and text stay and these additions are layered on top. ## Verification Builds: `CustomTestTarget=net10.0 dotnet build perf/ServerLoadHarness -c Release` and `dotnet build perf/TraceAnalyzer -c Release`: 0 warnings, 0 errors. Smoke tests (`--duration 5 --warmup 3`, `DOTNET_gcServer=1 SERVER_DOTNET_gcServer=1`), each run only once foreign CPU was below 1.5 cores, with foreign CPU measured during the run (avg 0.5–0.6 cores): | Run | Result | |---|---| | `events --sessions 5 --subs 3 --event-rate 2000` | generated/s=2009, events/s=30413, received/expected=100.9 %; `per-op2` 588.9 µs CPU / 31.5 KB per generated event | | `sub --sessions 5 --subs 3 --nodes 600 --node-type all --writers 4 --write-interval 0 --write-batch 100` | 131k values/s written, 77k notif/s, writeErrors=0; received/expected=3.9 % (queue size 1 with closed-loop writers: values are overwritten before publishing, as intended); `per-op2` 50.5 µs / 1680 B per written value | | `sub --sessions 2 --subs 1 --nodes 100 --queue-size 2 --publish-requests 4` | received/expected=98.0 %, writeErrors=0 | | same as the second run with `--trace` | 23 MB trace; `TraceAnalyzer threads` lists 58 threads with busy shares and roles; `cpu … 10 MonitoredItem.Publish` prints callers, first Opc.Ua frame and the new "direct callees of" section | #4815's own scenarios still parse: `counters` passes; `native-read` (with `--server-affinity`, `--thread-states`) and `native-counters` start the contract server and fail only at launching the (absent) native worker executable, as expected without the workers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Marc Schier <marcschier@users.noreply.github.com>	2026-10-17 07:56:43 +02:00
romanett	8b56937fdf	perf: drive the server with external native benchmark workers; TraceAnalyzer GC modes (#4815) ... ## What this adds Harness-only changes under `perf/` (not part of `UA.slnx`, `net10.0`). No product code changes. ### External native worker scenarios (`ServerLoadHarness`) Two new scenarios run the native open62541 workers of the [o6-automation/opcua-benchmarks](https://github.com/o6-automation/opcua-benchmarks) project against a server, so the .NET server and other OPC UA stacks are measured with the same client as that project's published comparison: | Scenario | What it measures | |---|---| | `native-read` | Fixed-load scalar Read throughput (`throughput/open62541/client.c --worker async`): `--clients` worker processes, each with `--inflight` single-value Reads of ns=1;i=1001..1100 in flight, starting together. Reports aggregate req/s, and server CPU and allocated bytes per request. | | `native-counters` | Subscription capacity (`subscription/native/client.c`): one subscription with `--nodes` UInt64 counters at `--sampling` 10 ms / `--pub` 1000 ms, queue size 1. Passes when every item gets 99 % of the samples its sampling interval calls for. | The result block has the same lines as the other scenarios, and the CPU time of the native workers goes in `client-per-op:`. The server is either the harness child in **contract mode** (`server --contract 1`) or any external executable (`--server-exe`, `--server-args` with `{port}`, `{items}`, `{sampling}` and `{pki}` placeholders), for example the benchmark's open62541 server or one of its .NET workers. By default the harness child is configured like the benchmark's .NET worker for UA-.NETStandard 2.0.1: | Setting | Default | Override | |---|---|---| | Node managers | `AsyncCustomNodeManager` | `--async-node-manager 0` (`CustomNodeManager2`) | | Counter increment | under a lock owned by the counters | `--counter-lock 0` (interlocked) | | Garbage collector | Server GC | `SERVER_DOTNET_gcServer=0` | | `MaxQueuedRequestCount` | clients × in flight + 3 (minimum 200) | `--queue` | Other options: `--native-client`, `--clients`, `--inflight`, `--iterations`, `--warmup-calls`, `--server-affinity` (processor mask; combine it with `SERVER_DOTNET_PROCESSOR_COUNT`, because the mask is applied after the runtime has sized itself), `--client-affinity`, `--server-priority`, `--thread-states 1` (samples Running/Ready server threads and appends a `sched:` line), `--security encrypt` (Basic256Sha256 SignAndEncrypt with generated DER identities) and `--verbose`. Also, the `MinRequestThreadCount` of the server child is now the smaller of 50 and `--threads`, so `--threads` below 50 takes effect. **Worker protocol.** The workers define a few names that are kept as string literals in named constants: the environment variables `O6_BENCHMARK_SECURITY_POLICY`, `O6_BENCHMARK_CERTIFICATE` and `O6_BENCHMARK_PRIVATE_KEY`, the `O6_BENCHMARK_READY` stdout token, the identity application URI / subject prefixes, and the server URI `urn:o6:benchmark:server`. The workers need that server URI because the contract puts its nodes in the namespace of the server URI and the workers address them as ns=1. Identifiers, file names, scenario and option names are neutral. ### TraceAnalyzer - `cpu` with a focus frame also reports the first `Opc.Ua` frame above it, past framework wrappers such as `Monitor.Enter` or `SemaphoreSlim`. - **GC-poll attribution:** the thread-time sampler suspends running threads at their next GC poll, so many on-CPU samples have `Thread.PollGCWorker` as their leaf. `cpu` used to drop them as waits (in one trace only 3.5k of 20k on-CPU samples were kept); it now attributes them to the caller. - `gc` mode (for `--trace-profile gc`): collections by generation, type and reason; the suspension time and the whole pause, each with count, total and percentiles. - `samples` mode: all samples by type (managed/external) and by leaf and first `Opc.Ua` frame, including the ones `cpu` leaves out. - `suspend <trace> [thresholdMs]` mode: every SuspendEE longer than the threshold, with the events recorded inside it by thread (an `AllocationTick` there means a thread that had not reached a safe point yet); event counts including the heap count DATAS chose and GC handle churn by kind. ## Building the workers The benchmark repository is AGPL-3.0, so nothing of it is vendored here. Clone it and build the workers separately. On Windows they build with `zig cc` and CMake against the open62541 commit the benchmark pins (OpenSSL backend, `UA_ENABLE_DEBUG_SANITIZER=OFF`, Release). The subscription workers also need `poll()` on stdin replaced by `PeekNamedPipe`, plus unbuffered control output. `perf/README.md` describes all of this. ## Commits 1. `perf: drive the server with external native benchmark workers` 2. `perf: configure the contract server like the benchmark's 2.0.1 .NET worker` 3. `perf: TraceAnalyzer gc and samples modes, samples taken at GC polls` 4. `perf: scheduling and GC-suspension diagnostics for the native worker runs` Each commit builds with 0 warnings (`CustomTestTarget=net10.0 dotnet build perf/ServerLoadHarness -c Release`, `dotnet build perf/TraceAnalyzer -c Release`), and `dotnet format whitespace` makes no changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 14:35:16 +00:00
romanett	28f1401324	Server: cut request-path lock contention and allocation; add load harness (#4707) ... Profiling the reference server under load showed most request-path CPU spent spinning on contended locks, plus several avoidable allocations per request or per node. This PR removes them, switches the reference server to Server GC, and adds the load harness used to measure it. ## Results Paired A/B against `master` (3 alternating rounds, medians, 20 sessions, workstation GC on both sides, 24-core machine): | Scenario | Throughput | p50 | p90 | Server CPU / op | Server alloc / op | Lock contention | |---|---|---|---|---|---|---| | Read, 1 value | **+45%** (94k → 137k req/s) | −20% | −46% | −37% | −39% | −94% | | Read, 100 values | **+40%** | −26% | −17% | −24% | −25% | +130% ¹ | | Read, 100 values, SignAndEncrypt | **+48%** | −36% | −20% | −30% | −26% | +106% ¹ | | Browse (Server object) | **+33%** | −31% | −12% | −13% | −14% | −40% | | Write, 100 values | +13% | −12% | −4% | −10% | −19% | −23% | | Data-change subscriptions | ±0 ² | ≈ | ≈ | −14% | −17% | −29% | ¹ The faster server spends more time in the remaining per-node locks. ² Throughput is bounded by the writer rate in this scenario. Separately, Server GC instead of workstation GC doubles Read-100 throughput (21k → 44k req/s) and cuts p90 from 2.1 ms to 0.6 ms, because workstation GC paused the server for about half the wall time under load. The working set grows from ~210 MB to ~430 MB. ## Changes - **`ObjectPool<T>`**: lock-free slot array. `ConcurrentBag.Count` took every per-thread lock on each `Return` of the pooled response value-task source, once per request. - **`FairRequestQueue`**: - drops redundant gate entries: the fail-fast pre-checks and the stopped check read volatile fields and are re-checked under the gate; the owner count is folded into selection; - coalesces wake pulses with a pending flag, so the bounded signal channel's lock is only taken when no token is outstanding. Wake-count semantics in the tests are unchanged. - **`DefaultServerResourceIsolationProvider`**: - compares peer addresses without `GetAddressBytes()` allocations; - builds the unsecured verification context only when a classifier is configured. - **TCP listener / channel**: caches the certificate DER copies per channel, keyed on the certificate instance. `X509Certificate2.RawData` returns a fresh copy on every access, and the listener read it for every request. - **`EndpointBase`**: resolves the request `ActivitySource` once instead of `Assembly.GetCallingAssembly()` per request. - **Node managers**: - one `ServerSystemContext` copy per request (cached on the `OperationContext`) for per-node callbacks: permission metadata, node metadata, view membership, method lookup; - removes a discarded copy in `GetReferenceDescriptionAsync`; - per-request operation caches are plain `Dictionary` instead of a `ConcurrentDictionary` with 24 lock objects; - the validation cache is built in one pass; - fixes zero-capacity error lists. - **`ResourceManager`**: translation lookups read an immutable snapshot that is rebuilt after writes. Browse serialised on the global lock for every display name. - **`MonitoredItem`**: shares the default `DataChangeFilter` instead of allocating one per sampled value. - **Reference server**: `ServerGarbageCollection=true`; new "Garbage collector mode" section in `docs/ServerScalability.md`. - **`perf/`**: `ServerLoadHarness` (out-of-process server plus load driver, server-side CPU/GC/alloc/contention deltas, optional `dotnet-trace` capture), `TraceAnalyzer` (TraceEvent-based CPU/allocation/contention aggregation), and paired A/B scripts. Not part of `UA.slnx`. ### Behaviour notes for custom node managers - The `IDictionary<NodeId, NodeState>` operation cache handed to overrides is now a `Dictionary`. It was a `ConcurrentDictionary`, but only one request flow uses it. - Within one request, per-node callbacks of a node manager share one `ServerSystemContext` copy. ### Later commits on this PR - **Permission checks without `NodeMetadata`** (`9068684c3`): the role-permission and access-restriction validators run on an internal `PermissionMetadata` struct; `AsyncCustomNodeManager` fills it on the stack from the same attributes and namespace defaults. Browse's reference filter reads 3 attributes per target instead of 13. Read 100 values: 120.6 → 95.7 KB/request, ~21k → 26k req/s; Browse 67.6 → 51.8 KB, 26.7k → 29.8k req/s. - **Read-only channel evidence** (`936b28d76`, `dc638ab9f`): `SecureChannelContext.ClientChannelCertificate`, `ServerChannelCertificate` and `ChannelThumbprint` are `ByteString`, so the cached per-channel certificate buffer is shared without a copy and cannot be altered by handlers or classifiers. **Public API change**, see the new section in `docs/MigrationGuide.md`. The isolation provider's revalidation snapshot still copies. - **Session lookup** (`936b28d76`): `SessionManager.TryGetSessionContext` takes the candidate snapshot without the bindings lock; the validating check under the lock is unchanged (2 instead of 4 global lock acquisitions per request). - Unit tests for `ObjectPool`, the per-request system context cache, the translation snapshot, `SecureChannelContext` null semantics and a stopped `FairRequestQueue`; `perf/` follows the repository conventions. ## Testing - `Opc.Ua.Server.Tests` (net10.0): 8386 passed before the `ByteString` change. After it, 13 secure-session and isolation tests failed; both causes are fixed in `dc638ab9f` and those tests (25 with their variants) pass. A full rerun of the Server, Sessions, Core, Redundancy, LDS and GDS suites on the final commit is in progress. - Full solution builds for net10.0 and net48; `dotnet format whitespace` clean. ### o6-automation opcua-benchmarks (sampling, deep pipelines) Replicated both tests of the [o6-automation benchmark](https://github.com/o6-automation/opcua-benchmarks) in the harness (`--scenario counters`, and `read` with 10 clients × N outstanding), paired against `origin/master`: | Test | master | this PR | |---|---|---| | Subscription sampling, 100 Hz, every item ≥ 99 % of samples, 1 item | FAIL (75–94 %) | PASS (100 %) | | same, 1024 items | FAIL (64–73 %) | PASS (99.9 %) | | same, capacity | — | between 4096 and 6144 items (~410–520k samples/s) | | Read, 10 clients × 100 outstanding, None | stalls (0–1.7k req/s) | 121–131k req/s | | Read, 10 clients × 100 outstanding, SignAndEncrypt | stalls (0 req/s) | 89–91k req/s | | Read, 10 clients × 1000 outstanding | channels faulted (`BadTcpNotEnoughResources`) | 0 errors, ~45–54k req/s | - **Fixed-rate sampling** (`30b769d13`): the sampling loop waited one interval after each sample, adding timer overshoot and sampling time to every cycle; samples now have absolute deadlines. - **Async response send** (`cdac5c5ca`): `SendResponse` blocked thread-pool workers on the channel gate; with many requests in flight the receive loop that was handed the gate waited for a pool thread and the channel stalled. The TCP, Kestrel TCP and WebSocket listeners now use `SendResponseAsync`. - **Backpressure instead of channel fault** (`191cd2ba7`): server channels stop reading while more than 100 responses are queued and resume at 50, instead of faulting the channel. Client channels keep reading. At 10,000 outstanding requests throughput is ~50k req/s with or without the limit engaged, so the remaining gap to the 100-per-client case is not the backpressure. ## Not changed - `SentMessageQueue` deep-clones every `NotificationMessage`. Retained messages return their pooled payloads on acknowledgement while the response copy may still be queued for send, and a client can acknowledge a sequence number before receiving it; the clone keeps a recycled payload from reaching another session's response. Removing it needs a send-completion hook from the transport. - Generated responses allocate a default `ResponseHeader` that is replaced (~2-3 % of per-request allocation). Fixing it means making structure fields of every generated type lazily initialized, a generator-wide change to default values, equality and encoding. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 13:00:49 +02:00