mirror of
https://github.com/OPCFoundation/UA-.NETStandard.git
synced 2026-10-17 05:11:13 +08:00
romanett
a14b23e7dd
# Server: cheaper symmetric crypto path, FairRequestQueue without a queue-wide lock Roadmap items E1 and 5b of the o6 benchmark gap analysis, one commit each. - **E1:** Windows one-shot `Aes.EncryptCbc`/`DecryptCbc` import the key (`BCryptImportKey`) on every message, although #4720 caches one `Aes` per channel token. The AES-GCM and ChaCha20-Poly1305 policies also created a new keyed instance per message. - **5b:** with `AsyncCustomNodeManager`, which is what the o6 benchmark's 2.0.1 worker uses, the `FairRequestQueue` gate became the largest waiting point. `TryEnqueue`, `TrySelect` and `TryDequeue` spun on one `Lock` for every request. No wire, configuration or public API changes. ## Results These are paired A/B runs against `origin/master`, five rounds per scenario. The two harness builds alternate per round, starting side swapped every round, on the same shared machine (Ryzen AI 9 HX 370, 12 cores / 24 threads, Windows 11). The tables show medians, with ranges in brackets where they matter. The server is configured like the o6 2.0.1 worker: `AsyncCustomNodeManager`, Server GC, queue sized clients × outstanding + 3. o6read uses the benchmark's native open62541 client workers; `read` uses the harness's .NET client. On this machine throughput moves between rounds by up to 2×, so CPU and bytes per request are the reliable signals. ### E1: symmetric crypto (measured on0dff94bd3) | Scenario | req/s | Server CPU µs/req | Server alloc B/req | Client CPU µs/req | Client alloc B/req | |---|---|---|---|---|---| | read --nodes 1, Basic256Sha256 SignAndEncrypt, 24 cores | 115,300 → 103,600 (ranges overlap) | 64.6 → 60.9 (**−6 %**) | 5,698 → 5,486 | 44.9 → 41.7 (**−7 %**) | 2,672 → 2,456 | | o6read 10×100, encrypted, 4-CPU mask | 71,300 → 75,900 | 51.8 → 51.4 | 6,323 → 6,111 | – | – | | o6read 10×100, encrypted, 24 cores | 123,500 → 118,200 (ranges overlap) | 115.7 → 117.2 | 6,716 → 6,515 | – | – | | o6read 10×100, unencrypted, 4-CPU mask (control) | 79,100 → 70,600 | 48.0 → 48.4 | 6,088 → 6,090 | – | – | Isolated costs per call (one pinned core, 224-byte chunk, variants alternating in one process): | Operation | Before | After | |---|---|---| | AES-256-CBC encrypt | 778 ns, 128 B (one-shot) | 329 ns, 24 B (cached transform) | | AES-256-CBC decrypt | 682 ns, 128 B | 239 ns, 24 B | | AES-256-GCM seal | 1,022 ns, 72 B (new `AesGcm`) | 260 ns, 0 B (cached) | | HMAC-SHA256 | 306 ns (per-token instance, unchanged) | – | For comparison, `HMACSHA256.TryHashData` costs 458 ns and `IncrementalHash` 318 ns. **E1 does not close the 20–50 µs encrypted gap, and no crypto change could.** Pinned to four CPUs, encryption costs master only about 3–4 µs per request (51.8 vs 48.0 µs). The extra cost at 24 cores comes from threads spinning on locks while signing and encryption run inside the channel gate (roadmap A1). The ~7 µs of HMAC in the report's profile is a thread-time artefact. `HashProviderCng.AppendHashData` shows 40× more samples on the sign side, inside the gate, than on the verify side for the same chunk sizes. Threads are preempted there while spinners hold the CPUs. ### 5b: FairRequestQueue (unencrypted, E1 has no effect here) | Scenario | req/s | Server CPU µs/req | Server alloc B/req | Contention events/s | |---|---|---|---|---| | o6read 10×100, 24 cores | 149,000 → 150,200 | 110.8 [106–117] → 99.6 [96–109] (**−10 %**) | 6,404 → 6,367 | 4,533 → 2,125 | | o6read 10×1000, 24 cores | 152,200 → 161,200 | 106.7 → 96.3 (**−10 %**) | 6,421 → 6,358 | 4,131 → 1,921 | | read --nodes 1 --sessions 200, 24 cores | 126,100 → 141,200 (+12 %) | 53.6 → 51.1 (−5 %) | 5,353 → 5,294 | 4,078 → 455 (**−89 %**) | | o6read 10×100, 4-CPU mask | 85,600 → 83,900 | 45.7 [44.1–47.8] → 46.8 [44.4–48.9] (noise) | 6,089 → 6,060 | 281 → 176 | Traces of o6read 10×100 at 24 cores, one run per side: - **Lock spinning.** In the sampled CPU profile, frames above `Lock.TryEnterSlow` in the fair queue fell from **16.6 % of on-CPU samples** (`TryEnqueue` 5.8, `TrySelect` 5.8, `TryDequeue` 4.9) **to none**. - **Contention events in the window:** 15,331 → 5,879. - **What remains of the queue:** - the wake channel in `DequeueAsync`: idle workers parked on the coalesced signal, 1.3 % of on-CPU samples, kept unchanged; - `SessionManager.TryGetSessionContext`'s server-wide `m_bindingsLock`, which every request takes at admission and again at dispatch. It is now the largest spin point (~26 % of `Lock.TryEnterSlow` samples) and is a separate follow-up. The channel gate (A1) is another ~17 %. In the o6 configuration all ten sessions map to **one** owner, so per-owner locks would not have helped. An intermediate version with a lock per owner FIFO left 12.8 % of the samples on those locks. That is why the FIFO itself is lock-free. ## Changes ### Symmetric cipher cache (E1) - `SymmetricCipherCache` replaces `AesCache`; a `ChannelToken` keeps one per direction (`ServerCiphers`, `ClientCiphers`). Per slot it retains: - a CBC encryptor keyed with the token's key and IV; - a CBC decryptor keyed the same way; - the AES-GCM or ChaCha20-Poly1305 instance. - **How a CBC message is processed:** the cached `ICryptoTransform` transforms the whole message in place with `TransformBlock`. An empty `TransformFinalBlock` (padding `None`) then rewinds the chaining state to the IV without re-keying, on CNG, OpenSSL and .NET Framework CAPI alike. Cached and fresh ciphertext are tested equal. - **Concurrency and ownership:** - A cipher is taken out of its slot while in use, so concurrent senders never share one. - It is matched to the key and IV by reference, and replaced if the key material differs. - It is disposed instead of returned after a failed transform, or once the token has been disposed. - **Unchanged:** - The uncached public `CryptoUtils` overloads keep the one-shot path. - The order is still verify, then decrypt, then padding, so a tampered padding byte under a valid signature is still reported as padding, after the signature check. - HMAC is unchanged. ### FairRequestQueue (5b) - **No lock in admission or dispatch.** - Each owner has a lock-free FIFO: Michael–Scott, with the entries themselves as the nodes, so queueing allocates nothing. The `LinkedListNode` per request is gone. - A request is claimed exactly once, by a compare-and-swap from selection, cancellation or stop. Claimed entries are dropped lazily when they reach the head. - **Weighted round robin over a lock-free ready queue of turn tokens.** - A turn holds one token per queued request, up to the owner's weight. A weight of 1,000,000 therefore costs no more scheduling state than the backlog. - Taking a turn's last token queues the owner's next turn at once, behind the other owners. The turn state is one 64-bit word changed by CAS, so the service order is the token order even while workers race. Weights 1:3 under 8 concurrent workers stay within 2.5–3.5. - **Failed execution admission:** the head goes to a small sorted list under the owner's gate, served before the FIFO. Its owner is not retried before capacity changes, as before. - **Owner FIFO retirement:** an owner FIFO is retired by an atomic CAS of its count from 0 to a sentinel, and a retired FIFO can never be counted again. The next request of that owner starts a new FIFO, and the retired FIFO's leftover tokens are skipped. - **Unchanged:** - The count bound is reserved with `Interlocked` before linking. - Global FIFO mode still selects one at a time, so a blocked head cannot be overtaken. - Wake coalescing (#4707), parked-request decoupling and `ServerBase.RequestQueue`'s worker growth. ## Behaviour notes - **Cipher cache memory:** a token now keeps up to two CBC transforms (plus their `Aes`) or one AEAD instance per direction, instead of one `Aes`. They are released with the token. - **Weighted ordering under concurrency:** dispatch order across owners follows token order. Within an owner, admission order is kept, including requests that fail execution admission and go back to the front. - **Idle scheduling state:** cancelled requests leave stale turn tokens in the ready queue until the next dispatch pass. They are discarded at once when the last queued request is cancelled. - **net48:** the BouncyCastle-backed AEAD types are cached like the platform ones. ## Testing - **New and extended tests:** - `CryptoUtilsAesCacheTests`, for all CBC and AEAD policies (Basic128Rsa15, Basic256, Basic256Sha256, Aes128_Sha256_RsaOaep, Aes256_Sha256_RsaPss, ECC_nistP256, ECC_nistP256/P384/RSA_DH AesGcm, ChaChaPoly): - cached vs fresh ciphertext for 0–300-byte bodies; - IV or key change; - tampered signature, ciphertext, tag and header, each rejected with the cache recovering; - bad padding under a valid signature; - 256 concurrent senders; - token disposal during a transform, and a return racing disposal. - `FairRequestQueueTests`: - turn tokens bounded by the backlog, not the weight; - cancellation leaves no tokens behind; - 1:3 weights under 8 workers; - no lost wakeups (4 producers, 8 waiting workers, both modes); - cancellation racing dequeue completes or dispatches each request exactly once and returns every lease; - stop while workers wait and admissions race; - a 50-owner, 16-worker stress run with cancellations; - pipelined clients (1×200, 10×100, 64×1) with ample and with scarce execution capacity. - The last test stalls on an earlier version of this change that briefly marked an emptying owner as retired, when the race window is widened with an injected delay. That bug stalled o6read 10×100 every time in the harness; a dump showed 1,000 queued requests with no turn tokens. Without the delay the window is too narrow to hit at unit level. - All 90 FairRequestQueue tests pass 30/30 repeated runs on net10.0 and 5/5 on net48. - **Suites,** net10.0, after rebasing on8b360da7d(#4748): | Suite | Passed | Failed | Skipped | |---|---|---|---| | Opc.Ua.Core.Tests | 6,357 | 0 | 88 | | Opc.Ua.Server.Tests | 8,483 | 0 | 5 | | Opc.Ua.Sessions.Tests (all secure-channel policies) | 851 | 0 | 379 | | Opc.Ua.Stress.Tests Contract + Integration | 45 | 0 | 0 | | Opc.Ua.Interop.Tests (1.5.378 peer, SignAndEncrypt Basic256Sha256, Aes256_Sha256_RsaPss, ECC_nistP256) | 218 | 0 | 29 (AEAD policies the 1.5.378 peer does not offer) | - **net48:** Opc.Ua.Core.Tests builds, and CryptoUtilsAesCacheTests plus FairRequestQueueTests pass. `dotnet format whitespace` is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>