## Description
Six small changes to the per-request path from the o6 benchmark gap
report (S1–S6), chosen from the allocation profile of the o6 read
benchmark (`AsyncCustomNodeManager`, Server GC, 10 clients × 100
single-value Reads in flight). Each change is its own commit. The
receive loop, its threading and its dispatch are not touched; there are
no wire, public API or generator changes.
### Results
Paired A/B runs, builds alternated per round, 5 rounds unless noted,
medians (min..max). Baseline: master d40843c23 with #4787, #4788 and
#4805 (all three merged since; this branch is rebased on master). Each
item was measured on its own on the baseline, and all six together. The
machine is shared with other sessions: a run during which another test
host or load harness ran was discarded and repeated. Allocated bytes per
request are deterministic; CPU per request is best read from the runs
pinned to 4 logical processors (`--server-affinity 0xF`, the 4 Zen 5
cores).
**Allocated bytes per request** (o6read 10 × 100, pinned):
| Node manager, security | baseline | S1 | S2 | S3 | S4 | S5 | S6 | all
|
|---|---|---|---|---|---|---|---|---|
| async, plain | 4,219 B | 4,114 (−2.5 %) | 3,631 (−13.9 %) | 4,057
(−3.8 %) | 4,123 (−2.3 %) | 4,099 (−2.8 %) | 3,511 (−16.8 %) | **2,411
(−42.9 %)** |
| async, encrypted | 4,267 B | −2.6 % | −13.8 % | −3.7 % | −2.4 % | −3.1
% | −16.5 % | **2,465 (−42.2 %)** |
| `CustomNodeManager2`, plain | 5,407 B | −2.1 % | −5.1 % | −3.0 % |
−1.8 % | −2.5 % | −12.9 % | **3,910 (−27.6 %)** |
| `CustomNodeManager2`, encrypted | 5,457 B | −2.0 % | −5.1 % | −2.9 % |
−1.9 % | −2.4 % | −12.9 % | **3,963 (−27.4 %)** |
**CPU per request, pinned to 4 logical processors:**
| Node manager, security | baseline | all six | each item alone |
|---|---|---|---|
| async, plain | 28.3 µs (27.8..29.8) | **27.1 µs (25.6..27.7), −4.2 %**
| −2.7 .. +1.9 % |
| async, encrypted | 31.5 µs (30.8..32.9) | 30.6 µs (30.1..40.1), −2.8 %
| −2.8 .. +0.8 % |
| `CustomNodeManager2`, plain | 34.0 µs (33.1..36.6) | **31.2 µs
(30.5..34.1), −8.1 %** | −5.0 .. +2.5 % |
| `CustomNodeManager2`, encrypted | 35.8 µs (35.0..38.9) | **34.3 µs
(33.5..35.2), −4.1 %** | −1.0 .. +3.8 % |
Garbage collections in the pinned plain run (300,000 requests): gen0 19
→ 12, pause 85 → 50 ms.
**After review (S6 with immutable codecs):** the tables above were
measured with a decoder and an encoder reset per message. Keeping
`BinaryDecoder` and `BinaryEncoder` immutable, so every message gets new
ones, costs about 214 bytes per Read: all six together allocate 2,625 B
per Read instead of 2,411 B in the pinned plain run (2 runs), −37.8 %
against the baseline.
**24 cores and the managed client** (all six against the baseline; the
managed scenarios 3 rounds):
| Scenario | baseline | all six |
|---|---|---|
| o6read 10 × 100, 24 cores: req/s | 296.3k (283.4..305.4k) | 302.2k
(245.0..323.0k) |
| same: allocated / request | 4,169 B | 2,349 B (−43.7 %) |
| o6read 10 × 1000, 24 cores: req/s | 297.4k (284.2..309.8k) | 295.6k
(269.6..311.5k) |
| same: allocated / request | 4,196 B | 2,373 B (−43.4 %) |
| `read --nodes 1`, 20 sessions: req/s, CPU / request | 259.3k, 40.2 µs
| 271.4k, 38.9 µs (−3.2 %) |
| same: allocated / request | 4,172 B | 2,444 B (−41.4 %) |
| `read --nodes 1`, 200 sessions: req/s, CPU / request | 257.1k, 36.7 µs
| 283.6k, **33.1 µs (−9.8 %)** |
| same: allocated / request | 4,041 B | 2,354 B (−41.7 %) |
| `read --nodes 100`: CPU / request | 226.1 µs | 214.9 µs (−5.0 %) |
| same: allocated / request | 72,379 B | 52,046 B (−28.1 %; S2 alone
−26.3 %) |
| `browse`: CPU / request (6 rounds) | 147.9 µs | 148.8 µs (+0.6 %) |
| same: allocated / request | 38,234 B | 32,603 B (−14.7 %) |
| `sub`: allocated / notification | 126 B | 121 B (−4.0 %) |
**Reading the CPU numbers.** Every item lowers the allocation per
request by a deterministic amount, and the savings add up (the six items
alone sum to −42.1 % against −42.9 % together). CPU per request follows
only partly: each item alone changes it within the run-to-run noise (±3
%); all six together lower it by 3–10 %, with the min..max ranges
separated in the pinned plain async and the `CustomNodeManager2`
encrypted runs and in `read --nodes 1` with 200 sessions. With Server GC
on 4 cores, garbage collection is a small share of the CPU, so 40 %
fewer bytes buy a few percent of CPU. At 24 cores throughput and CPU per
request are dominated by where the scheduler places the threads (Zen 5
and Zen 5c cores) and do not resolve a change of this size.
Allocation trace (o6read 10 × 100, first `Opc.Ua` frame): gone are the
`ResponseHeader` from `ServerBase.CreateResponse`, `Dictionary<NodeId,
NodeState>` (`AsyncCustomNodeManager.ReadAsync`), one of two
`NodeHandle`s, `Variant[]`/`uint[]` (permission attributes),
`List<DataValue>`/`List<DiagnosticInfo>`/`DiagnosticInfo[]`,
`ArrayOf<ExtensionObject>` (`Variant.Copy`), `RolePermissionType[]`,
`SecureChannelContext`, `PendingAdmission`, `System.Action`,
`RequestExecutionScope`, the `DequeueAsync` state machine box,
`MemoryStream`/`BinaryReader` of the decoder, `ArraySegmentStream`,
`BinaryWriter` and one `BufferCollection`.
## Changes
**S1: one `ResponseHeader` per response.** The generated response
constructors already create a default `ResponseHeader`; the services
replaced it with the one `ServerBase.CreateResponse` allocated. The
services that construct their response now offer its default header, and
the base `CreateResponse(RequestHeader, StringTable)` fills it. The
offer goes through a field of the thread, so the protected virtual keeps
its signature: an override that does not call the base returns its own
header as before, and a re-entrant call never gets the outer offer. The
services that receive their response from the subscription manager keep
the previous path. The third `ResponseHeader` in the profile is the
generated `Reuse()` resetting the response after it was sent. It stays:
`LeasedPublishResponse` releases its notification lease there, and the
reset is generated code.
**S2: a single-node Read without collections.**
- The dispatcher fills arrays instead of lists that were copied into the
response (the node managers fill them by index), and allocates the
operation diagnostics only for the first one.
- `AsyncCustomNodeManager.ReadAsync` creates its per-request node cache
only for a request with several nodes, or when a node must be validated.
- The permission check already resolves every node through
`GetManagerHandleAsync`, whose handle "does not require the NodeId to be
validated again". The Read service passes that handle on in
`ReadValueId.Handle` when it is an in-memory node of an
`AsyncCustomNodeManager` that was validated when it was returned, and
the node manager reads through it instead of looking the node up and
allocating a second handle. A handle passed on for an earlier read of
the same `ReadValueId` is cleared.
- The permission check without an attribute cache reads the three
permission attributes into arrays of the thread.
**S3: no copy of empty arrays.** `Variant.Copy` deep-copies arrays of
ExtensionObjects, DataValues and Variants because the caller can change
their elements. An empty array has no element to change but was copied
into a new boxed empty array. The Read service copies the namespace's
DefaultRolePermissions and DefaultUserRolePermissions on every
permission check (the variables' `CopyOnRead` policy); they are empty
unless roles are configured. `TryGetStructure` of an empty array returns
the empty array instead of a new one. Non-empty arrays, scalars and
matrices are copied as before.
**S4: one `SecureChannelContext` per channel token.**
`TcpTransportListener.OnRequestReceived` created a context per request,
although it only holds values that stay the same for the life of a
token. The channel keeps the last context with the values it was created
from and returns it while they are the same references; a renewed token
(new thumbprint, rotated certificate) or a transport reconnected to the
channel gets a new one. The isolation provider (#4787) and the session
binding check compare contexts by value.
**S5: fewer bookkeeping objects per request.**
- A request inside its service dispatch scope, which is every request
the server dispatches, needs no `RequestExecutionScope`: its
`OperationContext` completes the request itself, exactly once, when it
is disposed.
- `FairRequestQueue.TryEnqueue` owns the unpublished entry in a local
instead of a `PendingAdmission` wrapper, and registers the entry's
cancellation with a static callback instead of a delegate per request.
- `DequeueAsync` pools its state machine on .NET.
**S6: fewer codec objects per message.** Encoding a message is
synchronous, so a thread encodes one message at a time. The decoder and
the encoder stay immutable: every message gets new ones (see the
review).
- `EncodeSymmetricMessage` writes the body to the `ArraySegmentStream`
of the thread, reset per message, through a new `BinaryEncoder` that
borrows the `BinaryWriter` the thread keeps over that stream (closing or
disposing such an encoder flushes the writer but leaves it and the
stream open). The stream is given back once it handed its buffers over;
a body that fails to encode releases the stream's buffers, and the
stream is not reused. The stream is per thread, not per channel: since
#4782 the producers encode the responses of a channel concurrently and
only sealing runs in the writer.
- A response (client) in one chunk is decoded from that chunk's buffer,
as the server decodes requests; the client used a stream over the chunk
list for every response.
- A `BinaryDecoder` over a buffer creates its `MemoryStream` and
`BinaryReader` only when `BaseStream` exposes the stream, and reports
`Position` without exposing it. `DecodeMessage` reads `Position`, which
exposed the stream for every message and made every later read
synchronize its position with it.
- `ArraySegmentStream.GetBuffers` hands over the stream's own collection
instead of copying it; a message in one chunk is reassembled into a
collection sized for one.
### Not done
- **Skipping the reset of a sent response** (the third
`ResponseHeader`): `LeasedPublishResponse` releases its notification
lease in `Reuse()`.
- **Pooling `RequestLifetime`, `OperationContext` and
`EndpointIncomingRequest`:** they leave the stack. `RequestLifetime` is
public and handed to every service handler, and its token to node
managers. `EndpointIncomingRequest` is handed to the overridable
`ScheduleIncomingRequest`.
- **Pooling the fair queue `Entry`:** its cancellation registration and
the stop path can still reach it after it was released.
- **A struct token for the isolation lease:** leases are returned as
`IDisposable` through `IServerResourceIsolationProvider.TryAcquire`.
- **Reading the permission defaults without a copy when roles are
configured:** not measurable with the o6 server, which configures none.
## Behaviour notes
- **S1:** an override of `ServerBase.CreateResponse(RequestHeader,
StringTable)` that calls the base gets the response's default header
instead of a new one, the header the response carried anyway.
- **S2:**
- The node managers receive the Read result lists as arrays (`IList<T>`,
fixed size). The lists were pre-filled per item; adding to them was
never part of the contract.
- `ReadValueId.Handle` of a Read request holds the passed-on
`NodeHandle` during the call.
- **S4:** the requests of one token share one `SecureChannelContext`.
`UpstreamIdentity` has a public setter; the TCP transport never sets it,
but application code that set it on the context of a TCP request would
now change it for the token's later requests as well.
- **S6:**
- Each thread that encodes channel messages keeps one stream and writer
(a few hundred bytes), with a reference to the buffer manager of the
last channel it served.
- `BinaryDecoder.Position` of a buffer decoder no longer exposes
`BaseStream`.
## Testing
- **New tests:**
- S1, `StandardServerResponseStringTableTests` and
`ReferenceServerTests`:
- the offered header is filled with every field (timestamp, request
handle, service result, no diagnostics, string table);
- an override without the base keeps its own header;
- an override with the base fills the offered one;
- a re-entrant response never shares it;
- Read, Write and Browse responses keep every header field;
- a failed Read throws its status.
- S2, `AsyncCustomNodeManagerTests.ReadRouting` (all three node manager
fixtures):
- a single-node read uses the handle of the validation and returns what
a multi-node read returns;
- a handle passed on for an earlier read is not reused (another node,
removed node, no session);
- operation diagnostics keep one entry per node and are absent when not
requested.
- S3, `VariantCoverageTests`: empty arrays of each kind are shared; a
null array stays null; `TryGetStructure` of an empty, a null and a
one-element array.
- S4, `TcpListenerChannelTests`: the context is shared until the
thumbprint or the transport changes.
- S5, `RequestManagerTests`:
- an attached request completes once, however often its context is
disposed;
- the ambient request id is restored outside a dispatch scope;
- a stress test of attach, double dispose from two threads, cancellation
and drains.
- S6:
- `BinaryEncoderTests`: encoders borrowing one writer encode like new
encoders, and leave the writer and the stream open; null arguments;
- `BinaryDecoderTests`: `BaseStream` and `Position` of a buffer decoder;
range checks;
- `ArraySegmentStreamTests`: `GetBuffers` hands over and the stream
continues; a read stream still returns a copy; `Reset` and
`ReleaseBuffers`.
- **Run locally (net10.0, all pass):** Opc.Ua.Types.Tests,
Opc.Ua.Core.Tests, Opc.Ua.Core.Encoders.Tests, Opc.Ua.Server.Tests,
Opc.Ua.Client.Tests, Opc.Ua.Sessions.Tests, Opc.Ua.Subscriptions.Tests,
Opc.Ua.Redundancy.Server.Tests, Opc.Ua.Redundancy.Client.Tests,
Opc.Ua.Stress.Tests Contract + Integration. net48 builds.
- **Measured with:** `perf/ServerLoadHarness` scenarios `o6read` (o6
harness, not part of this PR), `read`, `browse`, `sub`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Description
Two changes to the per-request server path from the o6 benchmark roadmap
(R2, R5), found in the CPU and allocation profiles of the o6 read
benchmark (`AsyncCustomNodeManager`, Server GC, 10 clients × 100 Reads
in flight) on master 5b03e9afa (after #4767 and #4782). Each change is
its own commit. The request queue and the dispatch hop are not touched.
### Results
Paired A/B runs against master (5b03e9afa), alternating builds per
round, 5 rounds, medians (min..max). The machine is shared with other
sessions: allocated bytes per request are deterministic; CPU per request
is best read from the runs pinned to 4 cores. Each item was measured on
its own on master.
| Scenario | master | R2 | R5 |
|---|---|---|---|
| o6read 10 × 100, pinned to 4 cores: allocated / request | 4,702 B |
4,698 B | **4,358 B (−7.3 %)** |
| same: CPU / request | 31.8 µs (30.6..34.4) | 31.4 µs (29.3..33.1) |
32.1 µs (28.7..39.4) |
| same, `CustomNodeManager2`: allocated / request | 5,986 B | 5,980 B |
**5,532 B (−7.6 %)** |
| same: CPU / request | 37.7 µs | 36.2 µs | 37.1 µs |
| o6read encrypted, pinned, async NM: allocated / request | 4,745 B |
4,748 B | 4,397 B (−7.3 %) |
| o6read encrypted, pinned, `CustomNodeManager2`: allocated / request |
6,036 B | 6,037 B | 5,581 B (−7.5 %) |
| o6read 10 × 100, 24 cores: req/s | 270.9k (264.0..271.9k) | 272.4k
(269.5..275.4k) | 272.2k (262.8..273.7k) |
| same: allocated / request | 4,713 B | 4,710 B | 4,368 B (−7.3 %) |
| o6read 10 × 1000, 24 cores: req/s | 262.4k | 264.6k | 263.1k |
| `read --nodes 1`, 20 sessions: req/s | 206.2k | 204.9k | 208.9k |
| same: allocated / request | 4,631 B | 4,623 B | 4,397 B (−5.1 %) |
| `read --nodes 100`: allocated / request | 73,534 B | 73,531 B | 73,306
B |
| `browse`: allocated / request | 39,019 B | 39,035 B | 38,707 B (−0.8
%) |
| `sub`: allocated / notification | 139 B | 138 B | 136 B |
CPU per request and throughput change within run-to-run noise for both
items; a further 7-round pinned comparison of master against R2 + R5
(with the clock-read change that was dropped from this PR) gave 29.7 µs
(29.2..31.3) against 29.4 µs (29.1..30.7).
Sampled thread time at 24 cores (cpu trace, first `Opc.Ua` frame above
the frame), master against the combined build:
| Frame | master | after |
|---|---|---|
| `Buffer.BulkMoveWithWriteBarrier` (exclusive) | 11.4 % (`ProcessAsync`
5.1, `TrySendResponseWithoutGate` 3.7, `WriteCoalescedAsync` 1.7) | 0.3
% |
| `CastHelpers.Box` | 2.6 % (`ProcessAsync` 2.0, `NodeId` 0.6) | 0.3 %
(`NodeId`) |
Allocation trace: `ServerSystemContext` 6.7 % → 2.2 % (two copies per
Read, each now 48 bytes); `StringTable`, its `List<string>` and its
`Lock` (about 2.5 %) are gone.
**Why the CPU does not follow the profile.** The sampler suspends
threads at their next GC poll, so frames before a poll site, such as
runtime helpers and returns from native calls, are over-represented.
Once the copies were gone, the samples moved to the next poll sites in
the same code: `List<DataValue>.ToArray` went from 1.3 % to 13 % for a
one-element copy. CPU per request is unchanged in the pinned runs. The
struct copies cost less than their sample share suggested.
## Changes
**R2: no large struct copies per request.**
`EndpointBase.EndpointIncomingRequest` is a sealed class instead of a
mutable struct. It was boxed when scheduled as an
`IEndpointIncomingRequest` and copied again into `CallAsync`'s state
machine, both with write barriers. The class allocation replaces the
box. `UaSCUaBinaryChannel.PendingWrite` shares one payload reference and
two integers between its three kinds (chunk, buffers, unsealed message):
40 bytes instead of about 80 with a nullable `EncodedSymmetricMessage`
inside. Every response is copied into the write queue, the writer's
batch and back after sealing. `BeginWriteSymmetricMessage` takes the
message by `in`.
**R5: slim per-request system context, no string table per request.**
- `SessionSystemContext` keeps the values a copy takes from its template
in one immutable object: namespace and server tables, type table,
factories, handle, session values. A copy shares that object; a setter
replaces it rather than changing it, so copies stay snapshots. Server
contexts build it in one allocation. The per-request
`ServerSystemContext` of a node manager therefore holds four references:
48 bytes instead of about 160.
- `OperationContext` creates its `StringTable` on first use.
`StandardServer` builds the response header of a request that wrote no
diagnostic strings with an empty table of the thread.
## Behaviour notes
- **`EndpointIncomingRequest` (R2):** a protected nested type of
`EndpointBase` changes from a struct to a sealed class. Equality is by
request header as before, and is now null-safe.
- **System context copies (R5):**
- Copies still behave as snapshots: a value set on the template after a
copy was made does not reach the copy, and a value set on the copy does
not reach the template. This holds for derived types that copy
themselves with a plain `MemberwiseClone` too.
- Setting a value on a context allocates a new value object (a context
is set up once, not per request).
- **String table (R5):** a response still carries an empty (not a null)
string table when the request wrote no diagnostic strings. An override
of `ServerBase.CreateResponse(RequestHeader, StringTable)` still runs
for every response. The thread lends its empty table to the call and
takes it back only if the table is still empty when the call returns, so
strings an override adds (also when it then throws) never reach another
response, and a response created re-entrantly gets a table of its own.
## Testing
- **New tests:**
- `ServerSystemContextCopyTests`:
- every shared value is set and copied;
- a copy for a request allocates at most 64 bytes;
- changes stay where they are made: on a copy, on the template, on a
copy of a copy, and on a derived type;
- a copy for a session;
- a plain `MemberwiseClone` of a derived type stays a snapshot.
- `StandardServerResponseStringTableTests`: an empty table without
strings; the request's strings; an override that adds strings and
throws, adds strings, or creates a response re-entrantly leaks nothing
into later responses.
- `OperationContextTests`: the string table is created on first use.
- `ReferenceServerTests`: an empty response string table without
diagnostics, the strings with diagnostics, and an empty table again
after them.
- **Run locally (net10.0):** Opc.Ua.Core.Tests (Stack/Tcp, Transport,
Server), Opc.Ua.Server.Tests, Opc.Ua.Sessions.Tests,
Opc.Ua.Subscriptions.Tests, Opc.Ua.Redundancy.Server.Tests,
Opc.Ua.Stress.Tests Contract + Integration. net48 builds.
- **Measured with:** `perf/ServerLoadHarness` scenarios `o6read` (o6
harness, not part of this PR), `read`, `browse`, `sub`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
romanett