UA-.NETStandard/perf

Performance tooling

Tools for measuring server and client latency, throughput and GC cost under load. They are not part of the product build (not in UA.slnx) and target net10.0 only.

Path	Purpose
ServerLoadHarness/	Starts a reference server in a child process and drives load against it from the parent process, which is also the measured client.
TraceAnalyzer/	Aggregates the dotnet-trace files the harness records (CPU, allocations, lock contention, garbage collections).
ab.sh, ab-summary.ps1	Paired A/B runs of two harness builds, and the median summary.
variants.sh, variants-summary.ps1	Paired runs of one build under several configurations (GC mode, affinity), and the median and range summary.
benchmarks/	The variant and scenario files of the published tables in docs/Benchmarks.md.
server-scenarios.txt, client-scenarios.txt	The scenario lists used for the server and client comparisons.

The server runs in its own process, so the server-side numbers (CPU, allocated bytes, GC count and pause time, Monitor.LockContentionCount) exclude the load generator, and the same counters taken in the parent process measure the client stack without the server. The NUnit LoadTest fixture in tests/Opc.Ua.Sessions.Tests checks correctness under load; this harness measures it.

• Build
• Run: ServerLoadHarness run scenarios and options
• External benchmark workers: native-read, native-counters, steady-state parameters
• Analyse a trace: TraceAnalyzer modes for CPU, allocations, lock contention, GC, suspensions and thread scheduling
• A/B comparison: ab.sh, ab-summary.ps1
• Configuration comparison: variants.sh, variants-summary.ps1
• Measurement method: run record, reading the result block, artifacts

Current results and the recommended server baseline are in docs/Benchmarks.md; the runtime settings are described in docs/ServerScalability.md.

Build

CustomTestTarget=net10.0 dotnet build perf/ServerLoadHarness -c Release
dotnet build perf/TraceAnalyzer -c Release
dotnet tool install -g dotnet-trace

Run

perf/ServerLoadHarness/bin/Release/net10.0/ServerLoadHarness.exe run --scenario read --nodes 100 --sessions 20 --duration 15 --warmup 8

Option	Default	Meaning
--scenario	read	read, write, browse (Server object), sub (data-change subscriptions plus writers), events (event subscriptions on the Server object, events reported by the server), counters, or native-read/native-counters (see below).
--sessions	20	Concurrent sessions.
--nodes	100	Values per Read/Write request, or monitored items per subscription.
--inflight	1	Outstanding requests per session (closed loop).
--security	none	none, sign or encrypt (Basic256Sha256).
--client	managed	managed: ManagedSession with the V2 subscription engine (DefaultSubscriptionEngineFactory), used by sub, events and counters. classic: Session with the classic subscription engine. shared: classic sessions on secure channels shared through one ClientChannelManager, created one at a time (--connect-parallel).
--pool-notifications	0	1: the V2 subscription manager recycles notification objects after the handler returns (ManagedSessionBuilder.WithPoolNotifications).
--callback	fast	Classic engine only: data changes through Subscription.FastDataChangeCallback (fast) or the per-item MonitoredItem.Notification event (item).
--subs, --pub, --write-interval	5, 100, 100	Subscriptions per session, publishing interval and writer interval in ms (sub). events defaults to 15 subscriptions; counters to a 1000 ms publishing interval.
--write-nodes	all	Number of the monitored nodes the writer changes (sub), for large subscriptions in which few items change.
--writers	0	Dedicated writer sessions in addition to --sessions (sub). 0: the first subscriber session writes.
--write-interval 0		Writers write in a closed loop instead of on a timer (sub).
--write-batch	all	Values per Write request; writers rotate through the written nodes in slices (sub).
--node-type	int	int: Int32 mass nodes. all: the Int16…UInt64 mass node set of the Sessions LoadTest (up to 600 nodes) (sub).
--queue-size	1 (sub), 10000 (events)	Monitored item queue size.
--publish-requests	engine default	Minimum outstanding Publish requests per classic session.
--event-rate	0	Events per second the server reports (events); 0 reports as fast as one thread can.
--duration, --warmup	20, 10	Measurement window and warm-up in seconds.
--diag, --audit	0	Server diagnostics and auditing.
--threads, --queue	200, 2000	MaxRequestThreadCount and MaxQueuedRequestCount of the server; MinRequestThreadCount is the smaller of 50 and --threads.
--trace <file>		Record the measurement window with dotnet-trace.
--trace-profile	cpu	cpu (sampled thread time), alloc (allocation ticks), contention or gc.
--trace-side	server	The process to record: server (the child) or client (the harness itself).
--out <file>		Append the result block to a file.
--gc-events	1	0: turn the in-process GC event listener off (no client-gc: line).
--client-cores	all	Windows only. Relaunch the client with DOTNET_PROCESSOR_COUNT=N (thread pool, GC and ConcurrentDictionary sizing) and affinity to the last N allowed processors. If processors remain, pin the server to them. On Linux the process-affinity API changes only the main thread's affinity; use a cgroup (docker --cpuset-cpus) or launch under taskset instead.
--client-memory	unlimited	Emulated client memory budget in MB. Sets the GC heap hard limit to 75 % of that budget; native memory is not capped.

Environment variables prefixed SERVER_ are passed to the server process without the prefix, for example SERVER_DOTNET_gcServer=1 to compare garbage collector modes. GC settings set for the harness itself (DOTNET_GC*, COMPlus_GC*, for example DOTNET_gcServer=1 or DOTNET_GCgen0size=0x4000000) configure only the client: the server child does not inherit them.

Managed subscription setup has a one-minute deadline per subscription. Rejected monitored items and subscription error or deletion states fail the scenario instead of waiting indefinitely or starting a measurement with incomplete subscriptions.

The result block reports client throughput and latency percentiles, the deltas of both processes for the window (server:, client-process:), and the allocated bytes and CPU time per request (or per notification) of the server (per-op:) and of the client (client-per-op:). Classic sessions are created without diagnostics in responses, as an application would; the test fixture otherwise asks for them. Use bytes/op to compare allocation cost: it divides process-wide allocation deltas by completed operations. Keep the workload, runtime and configuration matched, including background activity and batching. On a shared machine, throughput and latency vary from run to run, so compare builds with ab.sh rather than with two separate runs; see Measurement method.

The client-gc: line uses runtime GC events captured by an in-process EventListener. It reports the GC mode and collections by condemned generation and trigger (gen2bg is a background gen2), plus pause and suspension percentiles. A background gen2 stops the threads more than once, so pauses can exceed n.

It also reports bytes surviving gen0 and gen1 per collection, the last gen2 and LOH sizes, the largest post-collection heap (peakAfterGcMB), committed bytes, the heap limit (limitMB, the hard limit under --client-memory), DATAS heap counts and the gen0 budget. When survivedKB/gen0 approaches the gen0 budget, inspect promotion into later generations alongside their collection counts. CollectionCount in client-process: counts a gen1 collection under both gen0 and gen1; client-gc: counts each collection once. The fixed log-linear latency histogram (about 1.6 % resolution) records each observation without allocating or retaining individual samples.

External benchmark workers

--scenario native-read and --scenario native-counters drive a server with the native open62541 workers of o6-automation/opcua-benchmarks, the client used for that project's published comparison of OPC UA stacks, so the .NET server and other stacks are measured with the same client. The harness starts the server, the worker processes and the measurement window; the result block has the same lines as the other scenarios, with the CPU time of the native workers in client-per-op:.

Scenario	Workload	Pass / metric
native-read	throughput/open62541/client.c --worker async: --clients processes, each with --inflight single-value Reads of ns=1;i=1001..1100 in flight, --iterations measured calls after --warmup-calls, common start.	Aggregate req/s, server CPU per request.
native-counters	subscription/native/client.c: one subscription, --nodes monitored UInt64 counters, --sampling 10 ms, --pub 1000 ms, queue size 1. The counters advance in their read callback.	Every item reaches 99 % of the samples in the measured window (10 s at 10/1000 ms); a FAIL exits with 1. samples/s counts the samples the server took (counter advancement); notif/s is shown when the worker reports delivered notifications.

Option	Default	Meaning
--native-client	client.exe (native-read), sub_client.exe (native-counters)	Path of the worker executable.
--clients	10	Read worker processes (native-read).
--inflight	100	Outstanding Reads per worker (native-read).
--iterations, --warmup-calls	100000, 10000	Measured and warm-up Reads per worker (native-read). The defaults are the benchmark's own and are a smoke test for a .NET server; see Steady-state parameters.
--nodes, --sampling, --pub	1, 10, 1000	Monitored counters, sampling and publishing interval in ms (native-counters).
--server-exe, --server-args	harness child, --port {port}	An external server executable and its arguments; {port}, {items}, {sampling} and {pki} are replaced. --server-args takes the next argument verbatim, even if it starts with --.
--server-affinity	all processors	Hexadecimal processor mask of the server process, for example 0x1 to compare the throughput per core with a single-threaded server.
--client-affinity	all processors	Hexadecimal processor mask of the worker processes, for example the mask of --server-affinity to co-locate clients and server.
--server-priority	Normal	ProcessPriorityClass of the server process, for example BelowNormal so that co-located workers preempt server threads.
--thread-states	0	1: sample the scheduling state of the server threads (see below).
--security	none	encrypt: Basic256Sha256 SignAndEncrypt with DER identities written to {pki} (native-read only; the subscription worker has no identity, so native-counters rejects it).
--async-node-manager, --counter-lock, --queue	see below	Configuration of the harness child.
--threads	200	MaxRequestThreadCount of the harness child.
--trace <file>, --trace-profile, --duration		Record the server for --duration seconds: native-read from the start of the measurement (default 20 s); native-counters from before the window is armed, by default long enough for the settle time, the warm-up and the window.
--verbose	0	1: echo the worker output.

Without --server-exe the server is the harness child in contract mode (server --contract 1): endpoint opc.tcp://127.0.0.1:<port>, server URI urn:o6:benchmark:server, which the workers require because the contract puts its nodes in the namespace of the server URI (ns=1). The external server can be, for example, the open62541 server of the benchmark or one of its .NET workers.

The harness child is configured like the benchmark's .NET worker for UA-.NETStandard 2.0.1:

Setting	Default	Override
Node managers (contract and counters)	AsyncCustomNodeManager	--async-node-manager 0: CustomNodeManager2
Counter increment	under a lock owned by the counters	--counter-lock 0: interlocked
Garbage collector	Server GC	SERVER_DOTNET_gcServer=0
MaxQueuedRequestCount	clients × in flight + 3 (minimum 200): the whole worker pipeline plus the three reserved bootstrap, reconnect and control slots, as the benchmark runner sizes it for 2.0.1 (current builds add the reserved slots on top of the configured value, so the + 3 is headroom)	--queue

The counters scenario keeps CustomNodeManager2 with interlocked increments.

--server-affinity (Windows and Linux) is applied after the server started, when the .NET runtime has already sized the thread pool and the GC heaps for all processors. Set SERVER_DOTNET_PROCESSOR_COUNT to the number of processors in the mask so that the runtime sizes them for the restricted machine:

SERVER_DOTNET_PROCESSOR_COUNT=4 ServerLoadHarness.exe run --scenario native-read --server-affinity 0xF --native-client <workers>/build/client.exe

With --thread-states 1 a thread in the harness samples the threads of the server process every ~3 ms during the measurement and appends sched: samples=<n> running=<avg> ready=<avg> readyMax=<max> to the server: line. ready counts threads in the Ready or Standby state. Sustained runnable-thread pressure suggests CPU contention; check available cores, affinity and scheduling events. The sampler enumerates the server threads every time, so leave it off for throughput comparisons.

ServerLoadHarness.exe run --scenario native-read --native-client <workers>/build/client.exe --clients 10 --inflight 100 --iterations 100000 --warmup-calls 10000
ServerLoadHarness.exe run --scenario native-read --native-client <workers>/build/client.exe --server-exe <workers>/build/server.exe --server-args "--port {port}"
ServerLoadHarness.exe run --scenario native-counters --native-client <workers>/build/sub_client.exe --nodes 4096

Steady-state parameters

The default native-read run (10,000 warm-up and 100,000 measured Reads per worker) took only a few seconds in the reported measurements. Those runs included JIT warm-up: about 28–31 µs of CPU per Read on 4 pinned logical processors against 8.8 µs at steady state, and about 60 µs against 28 µs on 24 logical processors. A native server such as open62541 has no JIT compilation, but still needs its connections and caches warmed up. Use the defaults as a smoke test, and for CPU per request and comparisons with other stacks measure at steady state:

SERVER_DOTNET_PROCESSOR_COUNT=4 ServerLoadHarness.exe run --scenario native-read --native-client <workers>/build/client.exe --server-affinity 0xF --iterations 300000 --warmup-calls 100000

The per-request cost should not change when the warm-up is made longer; if it still falls, extend the warm-up. SERVER_DOTNET_TieredCompilation=0 (optimizing JIT for methods without precompiled code) helps investigate tier-promotion costs in a short run, but it also removes dynamic PGO, so do not use it for the reported numbers.

Building the workers

The benchmark repository is licensed under the AGPL-3.0; its sources are not included here. Clone it and build its workers separately. On Windows they build with zig cc and CMake against the open62541 commit the benchmark pins (deps/open62541, OpenSSL backend, UA_ENABLE_DEBUG_SANITIZER=OFF, Release). The subscription workers additionally need poll() on stdin replaced by PeekNamedPipe and unbuffered control output (the C runtime treats _IOLBF as full buffering). The open62541 win32 event loop is select() based.

Analyse a trace

dotnet perf/TraceAnalyzer/bin/Release/net10.0/TraceAnalyzer.dll alloc trace.nettrace 40
dotnet perf/TraceAnalyzer/bin/Release/net10.0/TraceAnalyzer.dll cpu trace.nettrace 40 Monitor.Enter_Slowpath
dotnet perf/TraceAnalyzer/bin/Release/net10.0/TraceAnalyzer.dll contention trace.nettrace 15
dotnet perf/TraceAnalyzer/bin/Release/net10.0/TraceAnalyzer.dll gc trace.nettrace
dotnet perf/TraceAnalyzer/bin/Release/net10.0/TraceAnalyzer.dll samples trace.nettrace 30
dotnet perf/TraceAnalyzer/bin/Release/net10.0/TraceAnalyzer.dll suspend trace.nettrace 5
dotnet perf/TraceAnalyzer/bin/Release/net10.0/TraceAnalyzer.dll threads trace.nettrace 10

cpu with a focus frame also lists the callers of that frame and the first Opc.Ua frame above it, past framework wrappers such as Monitor.Enter or SemaphoreSlim, and the frame's direct callees. gc reads a trace recorded with --trace-profile gc. It lists collections by generation, trigger and condemn reason (condemn budget=gen1 indicates an exhausted gen1 budget). It reports average phase times for blocking collections outside background GCs: marking time and bytes by root type (stacks, handles and old-generation cards), summed across Server GC heaps, followed by elapsed plan/relocate/compact and restart time. Background gen2 collections and foreground collections overlapping them are excluded from the phase averages.

Each pause includes a separate suspension duration. For long suspensions, inspect safe-point and thread-scheduling delays. If little data is marked but pauses remain long, compare the reported phases before attributing the cost to stack scanning, card scanning or compaction. samples lists all thread-time samples by type and leaf frame (and by the first Opc.Ua frame), including the ones cpu leaves out.

suspend <trace> [thresholdMs] (default 40) reads a --trace-profile gc trace and prints every GC suspension that took at least the threshold, with the events recorded between SuspendEE start and stop by thread. An AllocationTick inside that window records allocation activity on the reporting thread during suspension; use event ordering and scheduling evidence to identify the cause of a delay. It also counts all events, including HEAPS <n> gen0budget~<bytes> (the heap count DATAS chose, from GlobalHeapHistory) and HANDLEKIND <kind> (GC handle churn by kind), and ends with SetGCHandle=<count> trace ms=<duration>.

threads lists the busy share of each thread (its on-CPU samples over all of its samples) with its main roles. This is a sample ratio, not a measurement of wall-clock CPU utilization. A thread near 100% busy while other cores are idle points to a possible serialized stage; confirm with its call stacks and the workload's available parallelism.

The sub and events scenarios add a per-op2 line with the server cost per written value or per generated event. Async methods with loops run on-stack-replaced code that the trace rundown does not name (samples show as !?); set SERVER_DOTNET_TC_QuickJitForLoops=0 and SERVER_DOTNET_TC_OnStackReplacement=0 when recording to attribute them.

The sampler suspends running threads at their next GC poll, so many on-CPU samples have Thread.PollGCWorker as their leaf; cpu attributes those to the calling frame instead of dropping them as a wait (threads that wait for a real collection land there too, which the gc mode shows). The EventPipe sampler attributes the on-CPU time since the previous sample to the poll site it stops at, and the poll is often the return from a native call (DateTime.UtcNow, TimeProvider.GetTimestamp), so such frames are heavily over-represented: in one pinned native-read trace the clock reads held about 70 % of the samples, yet removing them did not change the CPU per request. Read the shares as where the threads run, not as costs, and check a suspected hot spot with an A/B run of the change. For reliable CPU attribution use ETW CPU sampling, which samples at the instruction the processor executes: PerfView (PerfView collect) or WPR/WPA (wpr -start CPU), both from an elevated prompt on Windows.

Use ETW context-switch and ready-thread events (wpr -start CPU records them; open CPU Usage (Precise) in WPA) to identify which thread readied a waiting thread and how long it then waited for a processor. In the reported one-processor analysis, these events traced the latency tail to priority boost rather than timers; see Small Windows machines and thread priority boost. In that investigation, context-switch recording with stacks lost events at about 600k requests per second. Check the trace's lost-event counts and reduce the workload or recording volume if necessary; missing events leave gaps in the wait analysis.

The sampled thread-time profile also samples blocked threads. The cpu mode drops samples whose leaf frame is a known wait (thread-pool semaphore, I/O completion poll, sleep), so the remaining samples approximate on-CPU time. Spin-waiting in Monitor.Enter_Slowpath or Lock.TryEnterSlow is real CPU and shows up there; check it against a contention trace, though, because a thread blocked in the native part of Monitor.Enter has the same leaf frame. Socket sends and receives on loopback show up as on-CPU kernel time.

The alloc profile samples one allocation per ~100 KB and attributes the whole sample to the object that crossed the threshold. That over-weights small objects allocated right after large arrays, so read the type list as a pointer and the per-operation bytes in the result block as the measurement.

A/B comparison

Build the baseline harness in a second worktree, then alternate the two builds. SCENARIOS selects the scenario list (server-scenarios.txt by default, client-scenarios.txt for the client), ROUNDS the rounds per scenario and OUT the result file:

git worktree add --detach C:/uabase origin/master
cp -r perf C:/uabase/ && (cd C:/uabase && CustomTestTarget=net10.0 dotnet build perf/ServerLoadHarness -c Release)
cd perf && BASELINE=C:/uabase/perf/ServerLoadHarness/bin/Release/net10.0/ServerLoadHarness.exe SCENARIOS=client-scenarios.txt ROUNDS=5 ./ab.sh
pwsh -File ab-summary.ps1

The summary prints medians as baseline -> candidate (change): throughput, latency percentiles, client allocated bytes and CPU per operation (calloc, ccpu), GC pause and lock contention in the window (cpause, ccont), and the same for the server (alloc, cpu, pause, cont). It prints medians only; take the ranges from the result blocks in ab-results.txt.

Configuration comparison

variants.sh runs one harness build under several configurations of the server, for example the GC modes or processor masks of the tables in docs/Benchmarks.md. VARIANTS names a file with one variant per line, <name>|<environment>|<extra arguments>; SCENARIOS a file with the harness arguments of one scenario per line. {workers} in either file is replaced by WORKERS, the build directory of the native workers:

# benchmarks/gc-variants.txt
workstation|SERVER_DOTNET_gcServer=0|
server-datas|SERVER_DOTNET_gcServer=1|
server-no-datas|SERVER_DOTNET_gcServer=1 SERVER_DOTNET_GCDynamicAdaptationMode=0|

cd perf
WORKERS=<workers> VARIANTS=benchmarks/gc-variants.txt SCENARIOS=benchmarks/gc-scenarios.txt OUT=gc.txt ./variants.sh
pwsh -File variants-summary.ps1 gc.txt

Each round runs every variant once, in reverse order every other round, ROUNDS times (default 5). While a run is in progress the script checks every 2 seconds for test hosts and for more ServerLoadHarness processes than the run's own (OWN, default 2; one fewer for a variant with --server-exe, whose server is not a harness process); such a run is discarded, logged in <OUT>.discarded and repeated after the machine has been free of them for a minute. Other load, such as builds, is not detected. variants-summary.ps1 prints, per scenario and variant, the median and range (min–max) of the rate, delivery, latency, server CPU and allocation per operation (and per written value or generated event), GC pause, busy cores and working set, and the GC counts of every round.

Measurement method

Use the following procedure for comparisons:

• Allocated bytes per operation (per-op: allocB) measure process-wide allocation divided by completed operations. Compare matched workloads and repeated runs: background work, pool misses and batching can change this ratio.
• CPU per operation (per-op: cpuUs): pin the server when comparing a specific core type. In the native-read and native-counters scenarios use --server-affinity 0xF and SERVER_DOTNET_PROCESSOR_COUNT=4 for the child, so that the runtime sizes the thread pool and the GC heaps for the mask. The other scenarios ignore --server-affinity: start the harness itself with the affinity (start /affinity F on Windows, taskset -c 0-3 on Linux), which the server child inherits at its start, keeping in mind that the in-process client then shares those processors. On a hybrid CPU, choose a mask of one core type only (all performance or all efficiency cores) to separate that core type's cost from scheduling across types. Use the default mixed-core placement for a separate deployment comparison. Measure the native scenarios at steady state.
• Throughput and latency only from paired, alternating A/B rounds (ab.sh), never from two separate runs. Other test hosts, load harnesses or benchmarks running on the machine at the same time make a build look slower (fewer requests per second while cores sit idle); discard every round during which one ran, and repeat it.
• Run 5 rounds per scenario and report the median with the range (min–max) of each side. Retain the paired differences as well; overlapping ranges alone do not determine whether a difference is statistically significant.
• Publish every number with its run record and keep the result files.

Run record

Record with each published result:

Item	Where to get it
Date, commit or package version	git rev-parse --short HEAD; version.json for the version line
SDK and runtime	dotnet --version, dotnet --list-runtimes
OS and build	winver / ver, or uname -a and /etc/os-release
CPU model and topology	cores, logical processors and core types (Sysinternals Coreinfo, lscpu --extended)
Affinity and processor count	--server-affinity, --client-affinity, SERVER_DOTNET_PROCESSOR_COUNT, or how the harness was started
Power policy	powercfg /getactivescheme; on Linux the cpufreq governor
Memory	installed RAM, container limits
GC and JIT settings	SERVER_DOTNET_* variables set for the run (gcServer, GCDynamicAdaptationMode, TieredCompilation)
Security, placement	--security; whether client and server shared the machine
Workload	the full RESULT line: scenario, sessions or clients, in flight, subscriptions, monitored items, queue sizes, publishing, sampling and write intervals, --threads, --queue
Duration, repeats	--duration/--warmup or --iterations/--warmup-calls; rounds, and how many were discarded

Reading the result block

Each run prints one RESULT block (and appends it to --out <file>); the first line repeats the scenario and all arguments.

Metric	Line and field
Throughput	client: req/s (requests), notif/s (sub), events/s and generated/s (events), samples/s (native-counters)
Delivery, completeness	client: received/expected (sub, events); PASS/FAIL for native-counters
Latency	client: percentiles p50/p90/p99; write->notify latency (sub), event->notify latency (events)
CPU and allocation per operation	per-op: cpuUs, allocB per request or notification; per-op2: per written value or generated event
GC	server: gen0/gen1/gen2 collections and pauseMs in the window; client-process: the same for the client
CPU use	server: cpuMs and cpuCores (average busy cores)

Artifacts

Artifact	Produced by	Content
Result file	run --out <file>	RESULT blocks, one per run
ab-results.txt	ab.sh (OUT)	### side=<A/B> round=<n> <scenario> headers with the result blocks
Summary	ab-summary.ps1	medians per scenario, baseline -> candidate (change)
variants-results.txt	variants.sh (OUT)	### variant=<name> round=<n> <scenario> headers with the result blocks; <OUT>.discarded lists the repeated runs
Variant summary	variants-summary.ps1	median and range per scenario and variant
.nettrace	run --trace <file>	dotnet-trace recording of the measurement window, input of TraceAnalyzer