Closes #4842. Measures the items that `docs/Benchmarks.md` listed under *Open measurements* and makes every published table reproducible from checked-in files. All runs: `master`f6b27025a, AMD Ryzen AI 9 HX 370 (4 Zen 5 + 8 Zen 5c cores, 24 logical), 64 GB, Windows 11 26300, Balanced power plan, .NET SDK 10.0.402 / runtime 10.0.13. Each table has 5 rounds per variant, alternated and reversed every other round, and reports the median with its min–max range. ## Docs - **Reference Read table** re-run with ranges. The 2026-10-18 single runs move to a *Historical results* subsection. - .NET pinned to 4 logical processors: 440k req/s at 8.9 µs CPU per Read. - open62541: 137k req/s at 7.3 µs. - .NET on 24 logical processors: 600k req/s at 22.9 µs. - **Garbage collector modes**: workstation, Server GC with DATAS, and Server GC with fixed heaps, on `native-read`, `read` and `events`. - Server GC gave 45–89 % more throughput than workstation GC. Workstation GC paused the server for 30–46 % of the window. - Turning DATAS off cut GC pauses 4.6–8× at 1.9–2.8× the working set, with no throughput difference beyond the ranges. - **Processor placement** on the hybrid CPU, with and without SignAndEncrypt. - Server on the Zen 5 logical processors (`0xFF`): median throughput within 7 % of the default placement at about half the CPU per Read (11.7 vs 22.9 µs). - **Publish and event delivery** on the merged 2.0.2 master: delivered share, latency, CPU and allocation per notification, and GC pause. - **Recommended baseline** now cites these measurements. `ServerScalability.md` GC and affinity paragraphs are updated to match. - **Open measurements** now lists only what is still unmeasured: Linux, Intel hybrid CPUs, a dedicated host, and session establishment and scaling. ## Regression found (documented, not fixed here) The 50 sessions × 15 subscriptions × 600 items publish shape from #4829 now delivers only about 26 % of the expected notifications. | Commit | Delivered | Server CPU / notification | |---|--:|--:| | `37a2a831f` (#4829 merge) | 93 % | 3.9 µs | | `f6b27025a` (master) | 23 % | 46 µs | Both rows use the same harness arguments. A `git bisect` finds65a67dd2f(#4757) as the first bad commit: its parent0edc3b55adelivers 98 % at 2.8 µs, while65a67dd2fdelivers 6 % at 94 µs. The fix is being worked on separately. ## Tooling - `perf/variants.sh` runs one harness build under several configurations, given as `name|env|args` lines. It discards and repeats any run that overlaps another test host or load harness. - `perf/variants-summary.ps1` prints the median and range of each metric per variant. - `perf/benchmarks/*.txt` holds the variant and scenario files for each table. - `perf/README.md` documents all of the above. ## Test plan - [x] All four batches ran with the checked-in files. 2 of 140 runs were discarded because of foreign load; none failed. - [x] The summaries reproduce the tabulated numbers. - [x] CRLF line endings preserved in the edited docs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
46 KiB
Performance Benchmarks
This document explains how to run the 2.0 stack's BenchmarkDotNet harnesses and summarizes its performance against 1.5.378. It covers improvements, remaining slowdowns, and planned future work. It also reports the pooled encodeable micro-benchmarks, server request throughput measured with the out-of-process server load harness, and server session scalability.
Current and historical results. Server request throughput records dated results from the 2.0.2 development line (
2.0.2-preview,master). The other sections are historical records: they were measured on earlier 2.0 builds and other hardware, before the 2.0.2 request-path, publish and event-delivery changes, and are kept as measured, not re-run. Do not combine numbers from different sections, and do not extrapolate a deployment's capacity from one machine.
Contents
How to run
The benchmarks are BenchmarkDotNet harnesses hosted inside the test projects (a
BenchmarkSwitcher in tests/Common/Main.cs). Run a single class from its test
project directory:
cd tests/Opc.Ua.Core.Encoders.Tests
dotnet run -c Release -f net10.0 -- --filter "*BinaryEncoderBenchmarks*" --runtimes net10.0
# add --job short for a faster (higher-variance) directional run
Encoder/decoder benchmark classes live in Opc.Ua.Core.Encoders.Tests
(BinaryEncoderBenchmarks, BinaryDecoderBenchmarks, JsonEncoderBenchmarks,
JsonEncoderTests). The end-to-end session benchmarks (SecurityPolicyBenchmarks)
live in tests/Opc.Ua.Sessions.Tests and spin up a real in-process client + server
across every security policy.
Tip: the combined
--filter *build is flaky on this repo — run per class. A per-class build may need one or two retries (dotnet build-server shutdownbetween attempts).
2.0 vs 1.5.378
Historical record: BenchmarkDotNet measurements of earlier 2.0 builds on the Xeon host described under Environment and caveats.
This section compares the 2.0 stack with 1.5.378 on .NET 10. It explains what improved, what remains slower, and what future work could close the gaps.
All ratios below are 2.0 ÷ 1.5.378. < 1.0 means 2.0 is faster / allocates less; > 1.0 means 2.0 is slower / allocates more.
Keep this section current as performance-sensitive changes land. Update the relevant results and rerun the affected benchmark class (see How to run).
Why 2.0 differs from 1.5.378
The 2.0 redesign replaced several reference types with readonly struct
value types: Variant, DataValue, NodeId, ExpandedNodeId,
ByteString, ArrayOf<T>, and DateTimeUtc. It also moved client and server
operations to a fully async (TAP) pipeline.
The redesign reduces allocations and GC pressure by creating fewer heap
objects and less Gen2 traffic. It can also increase per-access CPU cost.
Reading a property from a large readonly struct may copy the struct or rerun
a discriminated-union switch.
The benchmarks below quantify this trade-off. Version 2.0 allocates less and runs faster on most client paths, but some value-type encoding and decoding paths use more CPU.
What was measured (2.0 vs 1.5.378, 409 matched benchmarks)
- Allocation geomean: 0.93× — 2.0 allocates less than 1.5.378 overall.
- Time geomean: ~1.07× — 2.0 is slightly slower on aggregate, concentrated in two areas only: the binary encoder and session establishment. Everything else is at parity or faster.
The two aggregate geomeans above come from the baseline full-job 2.0-versus-1.5.378 sweep of 409 matched benchmarks. Several areas have since improved, including binary decode allocation and in-memory and external-stream JSON (see the per-area rows below). The team did not rerun the full sweep on the final tree. The per-area table shows current results, while the aggregate remains the documented baseline. These improvements favor 2.0 further.
| Area | time | alloc |
|---|---|---|
Client read / browse (ClientTest, RequestHeaderTest) |
0.67–0.79× | 0.5–0.65× |
JSON encode, in-memory (JsonEncoderTests) |
0.97× | 0.60× |
Binary decode (BinaryDecoderBenchmarks) |
~1.14× | 0.69× |
JSON encode, external stream (JsonEncoderBenchmarks) |
1.33× | 1.29ײ |
Binary encode (BinaryEncoderBenchmarks) |
still slower¹ | ~parity¹ |
Session establishment (SecurityPolicyBenchmarks) |
still slower³ | — |
¹ The encoder remains slower than 1.5.378 on time, consistent with the
intrinsic cost of value-type structs (see below). The codec work did not change
allocation. A GC.GetAllocatedBytesForCurrentThread A/B test on the
benchmark payload measured the same allocation before and after the
BinaryPrimitives change: 76,808 B/op for ArraySegmentStream.
BenchmarkDotNet's Allocated values for ArrayPool/BufferManager-backed
streams are an accounting artifact. Pool rentals and returns can be attributed
differently across independent runs, so these values are not reliable per-op
figures.
² External-stream JSON allocation improved from ~1.44× to ~1.29× once the
Utf8JsonWriter is pooled/reused (below); the residual is inherent UTF-8 transcoding.
³ Session establishment remains the largest single regression vs 1.5.378; this is
detailed below.
What improved in 2.0 vs 1.5.378 (and why)
Lower allocation almost everywhere
Aggregate allocation across the 409 matched benchmarks is 0.93×. The value-type redesign means most messages, node ids, values, and byte strings no longer allocate individual heap objects across client, server, and encode paths.
Client read / browse: ~20–33% faster, ~half the allocation
ClientTest.BrowseFullAddressSpaceBenchmarkAsync: ~30% faster across every security policy (e.g. ~315 ms → ~213 ms).RequestHeaderTest.ReadValues…: ~20–25% faster, ~0.5× allocation.
The async pipeline plus value-type request/response handling cut both CPU and garbage on the most common client operation (reading/browsing the address space).
In-memory JSON encoding: faster and ~40% leaner
The in-memory JSON path (JsonEncoderTests) is 0.97× time / 0.60× allocation
vs 1.5.378 — i.e. 2.0 both runs faster and allocates ~40% less. Two changes:
- Flush at structural boundaries.
Utf8JsonWriternever flushes on its own, so for a large/streamed payload its internal buffer grows by doubling onto the Large Object Heap until it holds the entire message. 2.0 flushes buffered JSON at object/array boundaries once ≥ 16 KB has accumulated. Flushing only writes already-complete tokens and never changes the output. Large streamed payloads allocate ~7.3 MB instead of ~15.7 MB (−53%) with Gen2/op → 0. - Pooled in-memory buffer. The default (no external stream) encoder writes
through a private
ArrayPool<byte>-backedIBufferWriter<byte>instead of a per-encoderMemoryStream, and reads the result text from the pooled span. Buffers are returned withclearArray: trueso encoded payloads (which may contain user tokens or secrets) are not exposed to the next pool consumer.
JSON is the PubSub and REST/gateway encoding path; the LOH-growth fix in particular removes large, bursty Gen2 allocations for big messages.
Binary decoding: ~31% less garbage, CPU near parity
Binary decode allocates ~0.69× vs 1.5.378 (the value-type DataValue removes the
per-field heap object) while staying close on CPU at ~1.14×. The decoder reads
primitives directly from the buffer span via System.Buffers.Binary.BinaryPrimitives
and bulk-copies fixed-width numeric arrays (Int16/UInt16/Int32/UInt32/Int64/UInt64/ Float/Double) in a single blit (little-endian) rather than element-by-element. Decoding
is on the hot receive path of every client and server, so the allocation win applies
broadly.
Target-framework note: the quoted decode allocation (0.69×) is a .NET 10 number. On legacy target frameworks the figure is higher — see the legacy target-frameworks bullet under Environment and caveats.
Faster low-level primitives
UtilsIsEqual* and HiResClock* are faster on 2.0 (e.g. byte-array compares
~25% faster), reflecting span-based and modern-API implementations.
What is still slower in 2.0 vs 1.5.378 (and why)
Binary encoder — still slower (the main remaining encode regression)
Encoding a Variant/DataValue walks the 2.0 readonly struct accessor chain in
WriteVariantValue / WriteDataValue. Each property read can copy the large struct
and/or re-run the built-in-type switch. In the micro-benchmark,
WriteDataValueArray (a 10-element DataValue[]) dominates (~65% of the payload
iteration) because every element pays that struct cost. 1.5.378 used a
reference-type DataValue/Variant whose accessors were plain field reads, so this
is the direct CPU cost of the allocation-reducing value-type design. Closing this gap
to parity is tracked under Future work.
The encoder writes scalar primitives via BinaryPrimitives into the destination span
(IBufferWriter<byte>) and bulk-blits primitive numeric arrays. The default
(no-stream) encoder uses a pooled ArrayPool<byte> buffer. This change is allocation-
neutral: a ground-truth GC.GetAllocatedBytesForCurrentThread A/B over the benchmark
payload shows identical per-op allocation before and after it (the Allocated column
BenchmarkDotNet reports for the ArrayPool/BufferManager-backed stream variants is a
pooled-buffer accounting artifact, not a real per-op delta). The remaining encoder gap is
time, not allocation.
Impact: encode CPU on the send path, absolute cost a few microseconds per message. The bulk-array fast-path materially helps realistic large-array payloads (PubSub datasets, historical values); the residual mostly shows up in synthetic array-heavy micro-benchmarks.
Binary decoder — ~1.14× CPU, ~0.69× allocation
Near time parity with a clear allocation win, via the BinaryPrimitives span reads and
bulk numeric-array copies described under what improved.
JSON encoder, external-stream — improved by Utf8JsonWriter pooling
The external-stream JSON variants in JsonEncoderBenchmarks write to a
caller-provided stream. They trailed 1.5.378 because each encoder allocated a
new Utf8JsonWriter and paid for UTF-8 transcoding (1.5.378 used
StreamWriter).
Version 2.0 now pools and reuses Utf8JsonWriter instances for external
stream and IBufferWriter constructors. The pool is small, keyed by
Indented, and capped. Each encoder rents a writer, retargets it with
Utf8JsonWriter.Reset(...), and returns it on disposal. This removes the
per-encoder writer allocation; UTF-8 transcoding remains. The in-memory path
continues to use the ArrayPool<byte> buffer described above.
Session establishment — still slower; dominated by discovery + crypto
SecurityPolicyBenchmarks ('Create and close session' / 'Session lifecycle with read') remains the largest single regression vs 1.5.378. These are end-to-end
client+server benchmarks (channel + crypto handshake + server session setup), not
an encoder path.
2.0 also supports additional security policies (e.g. the ECC AES-GCM /
ChaCha20-Poly1305 suites) that 1.5.378 does not. Each configured policy adds per-session
cost on the server side, so a benchmark run that includes the new policies is not a
like-for-like comparison and inflates the all-policy geomean. A benchmark restricted to
the security policies common to both 1.5.378 and 2.0 — for the true apples-to-apples
change — now exists (SecurityPolicySessionCommonWithV15378Benchmarks, covering
Basic128Rsa15, Basic256, Basic256Sha256, Aes128_Sha256_RsaOaep, Aes256_Sha256_RsaPss).
An allocation profile of one full connect, including discovery
(Basic256Sha256) shows about 3.2 MB/op. Per-connect endpoint
discovery through GetEndpointsAsync accounts for most of the allocation.
The profile attributes it to:
| Allocator (Basic256Sha256 connect) | ~bytes/op |
|---|---|
System.String |
605 KB |
System.Char[] |
581 KB |
System.Byte[] |
222 KB |
UserTokenPolicy |
176 KB |
XmlWellFormedWriter |
133 KB |
(isolated NodeCache ctor) |
39 KB |
The eager per-session object graph (NodeCache + subscription engine) is only
~1.2% (~39 KB) of the connect, so lazy-initialising it would not move the
regression and was deliberately not pursued. Session establishment is a
one-time-per-connection cost, so absolute throughput impact is bounded to connect
churn.
Benchmark fidelity: real applications cache discovered endpoints.
SecurityPolicyBenchmarks.CreateCloseSessionAsyncandSessionLifecycleWithReadAsyncpass cachedEndpointstoConnectAsync. These benchmarks create, activate, and close sessions without rerunningGetEndpointsAsyncon every iteration.DiscoverEndpointsAsyncmeasures endpoint discovery separately.DiscoveryClient.PatchEndpointUrlsskips redundant endpoint-URL rewrites and reduces allocation by about 22 KB/op. Most remaining discovery cost comes from materializing endpoints, user-token policies, and application descriptions; that work remains future work.
JsonEncoderTests.ServiceMessageContext — ~2× on a tiny absolute
~122 ns vs ~63 ns (~0.94× alloc). Namespace/node-id-heavy message-context encoding;
the absolute cost is tens of nanoseconds and is partially absorbed by the NodeId
work above. Low priority.
Future work
-
Binary encoder — drive time to 1.5.378 parity. Scalar writes use
BinaryPrimitives, and numeric arrays bulk-blit; allocation is at parity. The remaining gap is time. A micro-profile points to theVariantvalue-type access path. InsideWriteVariantValue, thebuiltInTypeswitch already knows the type, butVariant.GetXxx()callsTryGetValue/TryGetScalar, which rereadTypeInfoand check the scalar and type before returning the union value. Scalar writes take about 60–160 ns/op, compared with 5–7 ns/op for a raw primitive write.StatusCode,LocalizedText,Guid, andNodeIdare slowest.Internal, already-validated union accessors might let the encoder skip these checks. An initial attempt showed no clear measured gain. Revisit this only if profiling demonstrates an improvement over the intrinsic value-type cost.
-
Session establishment — discovery allocations. The
SecurityPolicySessionCommonWithV15378Benchmarksbenchmark compares only security policies supported by both 1.5.378 and 2.0. Further work could reduce allocations asGetEndpointsAsyncbuilds endpoint, user-token, and application-description data and processes XML. A low-risk reduction has already landed, but most remaining cost comes from building the endpoint description. CachingConfiguredEndpointcould also let repeat connections skip discovery.
Environment and caveats
- Runtime: .NET 10.0 (
--runtimes net10.0), Release config. - Host: BenchmarkDotNet v0.15.x, Windows 11 on a shared Hyper-V VM, Intel Xeon Platinum 8473C, 8 physical / 16 logical cores, .NET SDK 10.0.30x.
- Virtualization caveat: BenchmarkDotNet warns that a shared/virtualized host affects measurements. Treat absolute numbers as indicative and sub-10% deltas as noise; focus on the direction and magnitude of large deltas.
- The 1.5.378 baseline is a full-job run; the per-class encoder refresh on 2.0 is a faster, higher-variance ShortRun, so treat those ratios as directional. Non-encoder classes carry run-to-run variance.
- Pooled-stream allocation columns are unreliable. For
ArrayPool- andBufferManager-backed streams such asArraySegmentStream, BenchmarkDotNet can attribute pool rentals and returns differently across independent runs. Do not compare these columns as per-op figures. Use aGC.GetAllocatedBytesForCurrentThreadA/B test instead, as in the binary-encoder comparison above. - Legacy target frameworks: 2.0 perf optimization targets the modern runtime (.NET 10).
The .NET Framework (net48) and netstandard2.0 target frameworks are not a
performance optimization target for 2.0: where a fast path is gated on
NET6_0_OR_GREATER(for example thestackalloc/ArrayPool<byte>BinaryDecoder.ReadStringpath), the legacy frameworks keep the simpler allocating fallback. Allocation/throughput figures in this document are .NET 10 numbers and legacy-TFM behaviour may be higher; that gap will not be closed for 2.0.
Pooled encodeable
Historical record: measured on an earlier 2.0 build (environment below).
Microbenchmark results for the IPooledEncodeable + PooledEncodeableType<T>
activator pooling feature added in support of ManagedSessionBuilder.WithPoolNotifications().
Source
tests/Opc.Ua.Subscriptions.Tests/PooledNotificationBenchmarks.cs
Methodology
- BenchmarkDotNet v0.15.8,
[MemoryDiagnoser],InProcessEmitToolchain(avoids the cold-rebuild timeout that the default toolchain hits on the test assembly's transitive reference graph). ShortRunjob: 3 warmup iterations, 3 measured iterations, single launch.- Inner loop is
Iterations = 1000allocate/use cycles per benchmark invocation — measured values are aggregate over the inner loop. - Pre-warm in
[GlobalSetup]populates all four type pools to steady-state before the first measured iteration so the pool-hit fast path is exercised rather than the cold-poolnew T()fallback.
Environment
| Item | Value |
|---|---|
| OS | Windows 11 (10.0.26200.8390 / 25H2) |
| CPU | Intel Xeon W-2235 @ 3.80 GHz, 6 physical / 12 logical cores |
| SDK | .NET SDK 10.0.300 |
| Host runtime | .NET 10.0.8 (RyuJIT, x86-64-v4) |
| Config | Release, -c Release -f net10.0 |
Results
DataValue is a readonly struct that lives inline in
MonitoredItemNotification.Value, so a notification item does not allocate a
separate DataValue heap object.
| Method | Mean | Allocated/op | Allocation ratio | Gen0/1000 ops |
|---|---|---|---|---|
new MonitoredItemNotification() (baseline) |
34.874 µs | 104,408 B | 1.000 | 24.17 |
MonitoredItemNotificationActivator + Reuse |
23.323 µs | 408 B | 0.004 | 0.092 |
new DataChangeNotification() |
66.759 µs | 216,344 B | 2.072 | 50.05 |
DataChangeNotificationActivator + Reuse |
54.812 µs | 32,344 B | 0.310 | 7.45 |
new EventFieldList() |
8.967 µs | 48,408 B | 0.464 | 11.22 |
EventFieldListActivator + Reuse |
22.280 µs | 408 B | 0.004 | 0.092 |
Interpretation
For the dominant publish-payload allocator (MonitoredItemNotification,
which arrives in arrays of arbitrary length on every data-change publish):
- Mean time per 1,000 allocate/use iterations drops from 34.874 µs to 23.323 µs — a roughly 33% time reduction in this synthetic benchmark.
- Allocations per 1000 ops drop from 104 KB to 408 B — ~256× reduction.
- Gen-0 collections per 1000 ops drop from 24.17 to 0.09 — ~263× reduction.
For DataChangeNotification (container + inner items + backing array):
- Pooling both the container and the inner
MonitoredItemNotificationitems brings allocation down to 31% of baseline. The residual 32 KB is theMonitoredItemNotification[]backing array itself (array pooling is out of scope).
For EventFieldList:
- In the historical empty-
EventFieldsbenchmark, pooling lowers allocations but increases mean time from 8.967 µs to 22.280 µs per 1,000 iterations. The emptyEventFieldsdefault allocates no backing storage, leaving little allocation work to offset theInterlocked.CompareExchangeand pool round-trip overhead. Populated events were not measured.
Reproducing
# Build release
dotnet build tests/Opc.Ua.Subscriptions.Tests -c Release -f net10.0
# Run BDN harness
cd tests/Opc.Ua.Subscriptions.Tests/bin/Release/net10.0
dotnet Opc.Ua.Subscriptions.Tests.dll --filter "*PooledNotificationBenchmarks*"
Artifacts (markdown, csv, html) are written to
tests/Opc.Ua.Subscriptions.Tests/bin/Release/net10.0/BenchmarkDotNet.Artifacts/results/.
Out of scope
The following payloads are not pooled and remain
attributable to the residual allocation in the pooled
DataChangeNotification numbers above:
Variantvalue payload (arbitrary user data insideDataValue.Value)- Dispatch backing arrays (
DataValueChange[],EventNotification[],MonitoredItemNotification[])
DataValue is now a readonly struct and no longer allocates on the
heap. It is not an IEncodeable and does not participate in the
activator pool system.
Server request throughput
The server load harness runs the server in a child
process, so its CPU time, allocations and GC counters exclude the load
generator. Its native-read scenario drives the server with the native
open62541 Read workers of
o6-automation/opcua-benchmarks,
so the .NET server and other OPC UA stacks are measured with the same client.
The managed scenarios (read, sub, events) use the harness itself as the
client.
Run record
All tables in this section except the historical results share this run record.
| Run record | Value |
|---|---|
| Date, commit | 2026-10-21, master f6b27025a (2.0.2-preview), Release build of perf/ServerLoadHarness |
| Host | AMD Ryzen AI 9 HX 370: 4 Zen 5 cores (logical processors 0–7, one core complex) and 8 Zen 5c cores (8–23, the other complex), SMT, 24 logical processors; 64 GB RAM |
| OS, power policy | Windows 11 Home 10.0.26300, Balanced power plan, mains power |
| SDK, runtime | .NET SDK 10.0.402, .NET runtime 10.0.13 |
| Server | Harness child; MaxRequestThreadCount 200. native-read: benchmark-contract mode (AsyncCustomNodeManager, MaxQueuedRequestCount clients × in flight + 3). Managed scenarios: reference server node set, MaxQueuedRequestCount 2000 |
| GC | Per table; where not stated, Server GC with DATAS (the .NET 10 default for Server GC). The harness client keeps workstation GC |
| Placement | Client and server on the same machine, not pinned unless the table says so |
| Repeats | 5 rounds per row, variants alternated and reversed every other round (perf/variants.sh); median with the range (min–max) of the 5 rounds |
| Artifacts | RESULT blocks per round, produced by the commands under each table |
The machine was a developer workstation, not a dedicated host. Rounds that overlapped another test host or load harness were discarded and repeated, but other background work (builds, editors) was not excluded; it shows as the low end of some throughput ranges, while CPU and allocation per operation stay narrow. Read the ranges before comparing rows, and treat throughput as the capacity of this machine only.
Reference numbers
native-read: 10 native Read worker processes with 100 single-value Reads of
ns=1;i=1001..1100 in flight each, no security unless noted, steady state
(100,000 warm-up and 300,000 measured Reads per worker).
| Server | Processors | Throughput (req/s) | Server CPU / Read | Allocated / Read |
|---|---|---|---|---|
| UA-.NETStandard | 4 logical (0xF, two Zen 5 cores) |
440k [395k–467k] | 8.9 µs [8.4–9.9] | 2.3 KB |
| open62541 (single-threaded) | — | 137k [105k–141k] | 7.3 µs [7.1–9.4] | — |
| UA-.NETStandard | 24 logical | 600k [573k–666k] | 22.9 µs [21.7–24.3] | 2.4 KB |
| UA-.NETStandard, 10 × 1000 in flight | 24 logical | 664k [505k–672k] | 20.8 µs [19.7–24.8] | 2.4 KB |
| UA-.NETStandard, Basic256Sha256 SignAndEncrypt | 24 logical | 507k [415k–524k] | 24.9 µs [24.0–25.7] | 2.4 KB |
Restricted to 4 logical processors, the .NET server used about 1.2 times the CPU per Read of the single-threaded open62541 server and ran the Reads on four processors in parallel. On all 24 logical processors throughput rose by about a third while CPU per Read rose 2.6 times: the server spans both core complexes and both core types; see Processor placement.
Short runs include JIT warm-up: with the benchmark's default parameters (10,000 warm-up and 100,000 measured calls per client) the same server showed about 60 µs per Read on 24 logical processors in earlier measurements. See JIT warm-up.
Reproducing the results
Build the harness and the native workers as described in
perf/README.md. Every table in
this section has a variant file and a scenario file in
perf/benchmarks; perf/variants.sh runs them (5 rounds
by default) and perf/variants-summary.ps1 prints the medians and ranges.
WORKERS is the build directory of the benchmark workers:
cd perf
WORKERS=<workers> VARIANTS=benchmarks/read-variants.txt SCENARIOS=benchmarks/read-scenarios.txt OUT=read.txt ./variants.sh
pwsh -File variants-summary.ps1 read.txt
| Table | Variants | Scenarios |
|---|---|---|
| Reference numbers | read-variants.txt |
read-scenarios.txt |
| Garbage collector modes | gc-variants.txt |
gc-scenarios.txt |
| Processor placement | affinity-variants.txt |
affinity-scenarios.txt |
| Publish and event delivery | delivery-variants.txt |
delivery-scenarios.txt |
Throughput is client: req/s (notif/s, events/s), CPU and allocated bytes
per operation are per-op: cpuUs and allocB (per-op2 per written value or
generated event), and the GC counts, pause time and working set of the window
are on the server: line. See
Reading the result block.
Garbage collector modes
The server child under three runtime configurations of the same build; the
harness client is unchanged. SERVER_DOTNET_gcServer=0 selects workstation
(concurrent) GC, SERVER_DOTNET_gcServer=1 Server GC with DATAS, and adding
SERVER_DOTNET_GCDynamicAdaptationMode=0 Server GC with a fixed heap count
(one heap per logical processor, 24). Pause is the total GC pause time of the
server in the measured window; working set is at the end of the window.
| Workload, window | GC | Throughput | Latency p50 / p99 | Server CPU / op | GC pause | Working set |
|---|---|---|---|---|---|---|
native-read 10 × 100 (300,000 Reads per worker) |
Workstation | 385k req/s [378k–405k] | — | 24.2 µs | 2320 ms | 164 MB |
| Server, DATAS | 661k req/s [606k–671k] | — | 22.5 µs | 214 ms | 186 MB | |
| Server, fixed heaps | 640k req/s [613k–671k] | — | 24.3 µs | 30 ms | 511 MB | |
read, 20 sessions × 100 values (15 s) |
Workstation | 45.6k req/s [45.3k–45.7k] | 0.27 / 1.81 ms | 231 µs | 4885 ms | 209 MB |
| Server, DATAS | 66.1k req/s [64.6k–66.7k] | 0.26 / 1.15 ms | 222 µs | 602 ms | 325 MB | |
| Server, fixed heaps | 68.3k req/s [66.8k–68.7k] | 0.26 / 1.15 ms | 214 µs | 132 ms | 616 MB | |
events, 50 sessions × 15 subscriptions, encrypted (15 s) |
Workstation | 44 events/s [43–45] | 206 / 323 ms | 53 ms / event | 6923 ms | 315 MB |
| Server, DATAS | 83 events/s [81–86] | 153 / 207 ms | 37 ms / event | 519 ms | 803 MB | |
| Server, fixed heaps | 84 events/s [83–86] | 152 / 199 ms | 35 ms / event | 65 ms | 2041 MB |
The events rows report events generated per second (each is delivered to
750 event monitored items; 99.8–100 % were received in every round), the
event-to-notification latency and the server CPU per generated event. The
sub scenario of gc-scenarios.txt (50 × 15 × 600) is not tabulated: it is
limited by the regression described under
Publish and event delivery and delivered
about 26 % in all three modes.
In these runs:
- Workstation GC lowered throughput by 31–47 % against Server GC and paused the server for 30–46 % of the window.
- Server GC with a fixed heap count cut the pause time by a factor of 4.6–8 against DATAS, at 1.9–2.8 times the working set. Throughput and CPU per operation stayed within the ranges of DATAS.
- Allocated bytes per operation did not depend on the GC mode.
Processor placement
native-read, Server GC with DATAS. Logical processors 0–7 are the Zen 5
(performance) cores of one core complex, 8–23 the Zen 5c cores of the other.
The server variants set --server-affinity and a matching
SERVER_DOTNET_PROCESSOR_COUNT; the native client workers run unpinned unless
--client-affinity is set.
| Placement | No security: throughput | CPU / Read | Busy cores | SignAndEncrypt: throughput | CPU / Read | Busy cores |
|---|---|---|---|---|---|---|
| Default (no affinity) | 637k [578k–657k] | 22.9 µs | 14.3 | 499k [284k–510k] | 24.5 µs | 12.2 |
Server on Zen 5 (0xFF, 8) |
593k [241k–668k] | 11.7 µs | 7.0 | 500k [495k–519k] | 14.6 µs | 7.3 |
| Server on Zen 5, clients on Zen 5c | 630k [602k–676k] | 12.0 µs | 7.6 | 471k [254k–525k] | 14.0 µs | 6.6 |
Server on Zen 5c (0xFFFF00, 16), clients on Zen 5 |
730k [344k–752k] | 20.0 µs | 14.4 | 479k [236k–493k] | 22.2 µs | 10.7 |
CPU per Read stayed within 8 % of the median in every round except one Zen 5c round (16.9 µs). In these runs, the server on the eight Zen 5 logical processors had a median throughput within 7 % of the default placement (and the same with SignAndEncrypt) at about half the CPU per Read and half the busy cores. The Zen 5c complex, with clients kept off it, gave the highest median throughput without security, at a CPU per Read between the two. The low ends of several ranges (241k–344k) are single rounds disturbed by other work on the machine.
These results describe one processor, one workload and Windows; Linux and Intel performance/efficiency-core processors were not measured. Commands for setting the affinity are in Hybrid CPUs and processor affinity.
Publish and event delivery
The 2.0.2 line changed the server paths that deliver notifications: the
publish cadence (#4829), event delivery (#4837, #4845, #4859) and data change
notification allocation (#4830). delivery-scenarios.txt, Server GC with
DATAS, Basic256Sha256 SignAndEncrypt, publishing interval 100 ms, queue size 1
(sub) and 10000 (events), 15-second window after 10 s of warm-up:
| Workload | Rate | Delivered | Latency p50 / p99 | Server CPU / notification | Allocated / notification | GC pause |
|---|---|---|---|---|---|---|
sub 10 sessions × 15 subscriptions × 600 items, all written every 150 ms |
600k notif/s [594k–605k] | 100 % [98.1–101] | 104 / 147 ms | 10.8 µs [10.4–11.3] | 803 B [468–1018] | 205 ms |
sub 2 × 1 × 100, written every 150 ms |
1344 notif/s [1332–1345] | 101 % [100–101] | 61 / 97 ms | 43 µs [34–69] | 5.7 KB | 5 ms |
events 10 sessions × 15 subscriptions, 1000 events/s requested |
598 events/s generated [593–610], 89.8k notif/s | 100 % [99.7–100] | 61 / 134 ms | 32 µs [30–33] | 6.6 KB | 605 ms |
Latency is from the write (or the event report) to the notification at the
client. A sub delivery above 100 % means some items reported more than one
value per write interval. In the events row the server generated 598 of the
requested 1000 events per second, at about 4.8 ms of server CPU per event
delivered to 150 monitored items; the reporting thread did not reach the
requested rate.
Regression in many-subscription data change delivery. The 50 sessions ×
15 subscriptions × 600 items shape of #4829 (sub in gc-scenarios.txt) is not
tabulated: on this master it delivered about 26 % of the expected
notifications in every GC mode, with writes taking 0.4–1.2 s and the server
reporting about a million lock contentions in the window. With the same
harness arguments (--client classic, Server GC for client and server), the
#4829 merge commit 37a2a831f delivered 93 % at 3.9 µs of server CPU per
notification. A git bisect over this shape found #4757 to be the first
commit with the lower delivery (0edc3b55a: 98 %, 2.8 µs per notification;
65a67dd2f: 6 %, 94 µs). The 10-session row above may be affected as well:
its CPU per notification is about twice the value reported in #4829.
Results of the individual pull requests, measured as paired A/B rounds
against the previous master when each was merged, are recorded in
#4829 (publish cadence: 50 × 15 × 600 delivered 67.7 % → 97.8 %),
#4837 (event delivery: 768 → 1001 generated events/s at 50 × 15) and #4859
(event queueing split). They show the size of each change on its commit,
not the current behaviour.
Recommended baseline
Measured on the machine and workloads above; check it against your own deployment:
- Server GC on (
ServerGarbageCollection). It raised throughput by 45–89 % over workstation GC in every workload above and cut the pause time by a factor of 8–13. - DATAS at its default. Disabling it (
GCDynamicAdaptationMode=0) cut GC pauses 4.6–8 times at 1.9–2.8 times the working set, without a throughput difference beyond the ranges. Choose it for latency-sensitive servers that can afford the memory;GCHeapCountcaps the heaps. MaxQueuedRequestCountat least the pipelining of all clients, so that established sessions do not receiveBadServerTooBusywhile CPU is idle.- Default scheduling (no affinity) unless CPU time matters more than peak throughput. On this hybrid CPU, the server restricted to the performance core complex reached a similar median throughput at about half the CPU; on a host shared with other processes this frees the remaining cores. Measure the placement on the deployment host, since the gain comes from this processor's core types and core complexes. See Hybrid CPUs and processor affinity.
- Compare priority boost on and off on small Windows hosts. In the one-logical-processor measurement, disabling it cut the 99.9th-percentile Read latency from 32–56 ms to about 3 ms; see Small Windows machines and thread priority boost.
- Measure at steady state, after the JIT warm-up.
The baseline changes when the workload changes its bottleneck: node managers
that block (raise MaxRequestThreadCount), many subscriptions or event
monitored items (watch delivery, latency and GC pauses, not only
throughput), memory limits (DATAS on, GCHeapCount), or a host shared with
other processes (affinity). See
Runtime and host settings for
each setting and how to verify that it is in effect.
Measuring
Follow the measurement method of the
harness: allocated bytes per operation as the primary metric, CPU per
operation with the server pinned (--server-affinity and
SERVER_DOTNET_PROCESSOR_COUNT) to one core type, the native scenarios at
steady state, and throughput
only from paired, alternating rounds: perf/ab.sh for two builds,
perf/variants.sh for configurations of one build (5 rounds, medians
with ranges), discarding rounds during which other test hosts or load
harnesses ran on the machine. Record every published number with the
run record of the harness.
Open measurements
Tracked in #4842:
- Linux, and Intel performance/efficiency-core processors (GC modes and processor placement).
- A dedicated benchmark host without other background work, to narrow the throughput ranges.
- Session establishment and steady-state session scaling on the current stack (the session scalability results are historical).
Historical results (2026-10-18)
Single runs on the same machine, master 014f99a53, .NET SDK 10.0.401 and
runtime 10.0.12, before the worker wake-up (#4833) and event delivery changes;
superseded by the reference numbers above:
| Server | Processors | Throughput | Server CPU / Read | Allocated / Read |
|---|---|---|---|---|
| UA-.NETStandard | 4 logical, pinned (0xF) |
— | 8.8 µs | 2.6 KB |
| open62541 (single-threaded) | — | 117k req/s | 8.1 µs | — |
| UA-.NETStandard | 24 logical | 574k req/s | 27.5 µs | — |
| UA-.NETStandard, 10 × 1000 in flight | 24 logical | 518k req/s | — | — |
| UA-.NETStandard, Basic256Sha256 SignAndEncrypt | 24 logical | 447k req/s | — | — |
Server session scalability
Historical record: measured on an earlier 2.0 build, before the 2.0.2 publish cadence change (#4829); the host is described under Observed scaling.
The [Explicit] macro test
ServerManySessionsLoadTestAsync(int sessionCount) in
tests/Opc.Ua.Sessions.Tests/LoadTest.cs exercises the reference server with
many concurrent sessions. Each session opens a secure channel and creates a
slow-publishing subscription (1000 ms) with one monitored item on a shared
value node. A separate writer session changes that value periodically. The
test checks that every session receives value-change notifications during a
steady-state window. It uses Basic256Sha256 (sign and encrypt) and requires
all sessions to connect and receive notifications.
The test offers these session counts:
- 500 (baseline)
- 1000, 1500, 2000, 2500
- 4000, 5000, 8000, 10000 (stress case)
Select a case by name, for example ServerManySessionsLoadTestAsync(2000).
For a deep, code-referenced analysis of why a single node tops out here — the establishment vs steady-state boundaries and the built-in controls for degrading gracefully under load — see Server Session Scalability.
| Tested configuration | Value |
|---|---|
| Concurrent sessions | 500 baseline, up to 10000 (selectable case) |
| Secure channels | one per session (MaxChannelCount) |
| Subscriptions per session | 1 (1000 ms publishing interval) |
| Monitored items per subscription | 1 (shared value node) |
| Security policy | Basic256Sha256 (sign & encrypt) |
| Steady-state duration | 60 s |
Observed scaling
On comparable hardware, 1.5.378 capped a single node at about 2000
concurrent sessions. Each held long-poll Publish pinned a request
processing worker, so the worker pool—not the CPU—became the bottleneck as
sessions increased.
Version 2.0 decouples held Publishes from the worker budget by default
(ServerConfiguration.DecoupleHeldPublishRequests). A small worker pool can
therefore serve many parked Publishes. The limit shifts to session
establishment, where CPU-bound RSA handshakes support about 4000 sessions
on the six-core machine below.
These measurements used an Intel Xeon W-2235 (6 physical cores / 12 logical
threads), 64 GB RAM, 64-bit Windows. The client and in-process reference
server shared a developer machine under light background load. Both ends
competed for the same cores, so Basic256Sha256 RSA handshakes on the client
and server dominated establishment time. Throughput declined as concurrency
increased.
A per-cycle sweep parallelized across cores delivered steady-state Publishes. For every count that established successfully, all sessions received 100% of their notifications within the steady-state window (0 drops).
| Concurrent sessions | Sessions establish | Average sessions/sec created | All sessions receive notifications |
|---|---|---|---|
| 500 | Yes | 26 | Yes |
| 1000 | Yes | 15 | Yes |
| 1500 | Yes | 11 | Yes |
| 2000 | Yes | 14 | Yes |
| 2500 | Yes | 13 | Yes |
| 4000 | Yes | 11 | Yes |
| 10000 | No (*) | — | — |
(*) These figures depend on hardware and load. They are directional and may improve on dedicated hardware with separate client and server machines.
Enable ServerConfiguration.DecoupleHeldPublishRequests (the default) and
set MaxRequestThreadCount to active session-establishment concurrency
(about 200). On this six-core machine, that setup established every session
and delivered 100% of notifications up to ~4000 concurrent sessions.
At 10000 sessions, repeated
handshake aborts and retries caused a secure-channel connect storm, with tens
of thousands of channel attempts. The server did not establish all sessions
within the connect budget. On this machine, session establishment—not
notification delivery—limits capacity; more cores can raise the limit.
If sessions connect but do not receive notifications, the last table column
shows No (N drops). N is the number of sessions that missed notifications
during the window. See Server Session Scalability for
the code-referenced analysis and controls for graceful degradation under load.
Subscription transport buffer pooling
BufferManager previously stored a one-byte ownership cookie after the usable transport data in all builds. Configuring TransportQuotas.MaxBufferSize to an exact power of two such as 65,536 could therefore call ArrayPool<byte>.Rent(65,537), selecting the 131,072-byte bucket. The cookie is now enabled in Debug builds (and explicit TRACK_MEMORY diagnostic builds) but removed from normal Release builds; the secure-channel normalization still keeps data plus any active diagnostic cookie in the same bucket.
Channels now normalize their advertised send and receive limits to the largest protocol-safe value that keeps the data plus cookie in the same ArrayPool bucket (while preserving the OPC UA 8,192-byte minimum), and both TCP transport creation paths receive that normalized limit. A configured value of either 65,535 or 65,536 consequently uses 65,536-byte arrays instead of 131,072-byte arrays.
The explicit net10.0 harness
SubscriptionArrayPoolDiagnosticsLoadTests
(tests/Opc.Ua.Sessions.Tests/SubscriptionArrayPoolDiagnosticsLoadTests.cs)
measures real subscription traffic in three cases: 255 and 257 session
Publish queues (just below and above the parallel Publish threshold), plus a
slow consumer.
Across both configured buffer sizes:
- Every scenario used only the 65,536-byte bucket.
- Rent and return counts balanced after quiescence.
- Outstanding buffers returned to zero; peak outstanding buffers ranged from 528 to 536.
- No Gen 2 collections occurred during measurement.
The 500-session regression baseline also completed for opc.tcp, https,
and opc.https. All 500 sessions received notifications.
The net10.0 in-process short BufferManagerBenchmarks run completed 256 BufferManager.TakeBuffer/ReturnBuffer pairs in 56.18 us versus 52.52 us for direct ArrayPool<byte>.Shared use, with no managed allocation reported for either pooled path. These short-run timings are directional; the subscription harness is the authoritative ownership and bucket-size check.
Sizing and configuration
-
MaxSessionCount(default 100) caps the concurrent open sessions; sizeMaxChannelCount(one channel per session) andMaxSubscriptionCountabove the target session count. -
MaxRequestThreadCountcaps concurrent request processing. WithServerConfiguration.DecoupleHeldPublishRequestsenabled (the default), a held long-pollPublishreleases its worker while it waits. SetMaxRequestThreadCountfor the number of sessions being established at once, not the total number of open sessions. In these measurements, 100–200 workers served thousands of sessions. A very large pool oversubscribes cores during connection bursts and slows establishment.MinRequestThreadCountprewarms the pool to avoid thread-pool cold start. See Request workers and the request queue for the current behaviour of these settings and ofMaxQueuedRequestCount.Set
DecoupleHeldPublishRequests = falseto restore the legacy behavior. Held Publishes then occupy workers. A server handling N sessions needs a pool well above N to prevent other services from returningBadRequestTimeout. -
MaxFailedAuthenticationAttempts(default 5;0disables) is the brute-force lockout threshold, keyed per client certificate. A single-certificate client that opens many sessions can trip it on transient handshake failures, after which further sessions are rejected withBadUserAccessDenied; disable or raise it for such clients. -
Session establishment is CPU-bound (RSA handshakes) but parallelizes across cores; throttle/stagger concurrent connects and scale cores for bulk connection throughput.
A full macro benchmark that produces a capability matrix (sessions × subscriptions × items against CPU/memory) across configurations is planned as follow-up work.