Files
UA-.NETStandard/tests/Opc.Ua.Server.Tests/AsyncCustomNodeManagerTests.ReadPermissions.cs
romanett 6cfb8cf196 Server: remove per-request context copies and lock hot spots from the request path (−16 % allocation per Read) (#4767)
## Description

Five small changes to the per-request server path, found in the CPU and
allocation profiles of the o6 read benchmark (`AsyncCustomNodeManager`,
Server GC, 10 clients × 100 Reads in flight) on master 3a31473cf. Each
change is its own commit and can be reviewed and reverted on its own.
The channel gate (about 40 % of waits) is not touched; that is A1.

### Results

Paired A/B runs against master (0e40e1229), alternating builds per
round, 5 rounds, medians (min..max). The machine is shared with other
sessions: allocated bytes per request are deterministic; CPU per request
is best read from the runs pinned to 4 cores.

| Scenario | master | this PR | Change |
|---|---|---|---|
| o6read 10 × 100, server pinned to 4 cores: CPU / request | 39.1 µs
(35.9..39.8) | 36.9 µs (35.5..37.6) | **−5.6 %** |
| same: allocated / request | 6,198 B | 5,224 B | **−15.7 %** |
| same: req/s | 98.7k (95.4..108.5k) | 104.6k (97.6..105.9k) | +5.9 % |
| o6read 10 × 100, 24 cores: req/s | 183.7k (173.3..198.0k) | 189.8k
(178.9..197.5k) | +3.4 % |
| same: allocated / request | 6,500 B | 5,578 B | −14.2 % |
| o6read 10 × 100 Basic256Sha256, 24 cores: req/s | 159.8k
(149.4..178.7k) | 164.2k (156.7..165.3k) | +2.7 % |
| same: allocated / request | 6,590 B | 5,694 B | −13.6 % |
| `read --nodes 1`, 20 sessions: req/s | 155.3k (141.6..157.6k) | 163.8k
(163.4..166.5k) | +5.5 % |
| same: allocated / request | 5,548 B | 4,689 B | −15.5 % |
| `read --nodes 100`: allocated / request | 74.4 KB | 70.1 KB | −5.8 % |
| `browse`: allocated / request | 39.8 KB | 35.0 KB | −12.2 % |
| `browse`: lock contentions / 1000 requests | 474 | 367 | −23 % |

At 24 cores CPU per request stays within noise, because the time these
changes free now waits at the channel gate (see below).

Per item, pinned to 4 cores (5 rounds; CPU deltas of one item are within
run-to-run noise, allocation deltas are not):

| Item | CPU / request | Allocated / request |
|---|---|---|
| Q1 system context | +1.8 % | −1.7 % |
| Q2 request table | −4.3 % | −1.4 % |
| Q3 request timeouts | −2.8 % | −2.2 % |
| Q4 namespace metadata | −3.5 % | ±0 |
| Q5 read-path allocations | −2.4 % | −10.9 % |

Sampled thread time at 24 cores (cpu trace, first `Opc.Ua` frame above
the wait or copy):

| Frame | master | this PR |
|---|---|---|
| `ServerSystemContext.Copy` (incl. `MemberwiseClone`) | 12.4 % | 0.1 %
|
| `RequestManager` (`ConcurrentDictionary` inside
`EnterValidationScope`, scope `Dispose`, `RequestReceived`,
`CompleteRequest`) | 7.1 % | 0 |
| `RequestLifetime` (CTS registration spin,
ctor/`RegisterExternal`/`DisposeResources`) | 4.8 % | 0.2 % |
| `NamespaceMetadataRegistry.GetAsync` lock | 4.0 % | 0 |
| `RuntimeHelpers.InitializeArray` (`ReadValidationAttributes`) | 2.2 %
| 0 |
| Channel gate (`ChannelGate+AwaitEntryAsync`, inclusive) | 31 % | 46 %
|

## Changes

**Q1: one system context copy per request and node manager.**
`AsyncCustomNodeManager`'s service methods (Read, Write, Browse,
TranslateBrowsePaths, Call, HistoryRead/Update, monitored item and event
services) use the per-request copy from
`OperationContext.GetSystemContext` instead of copying the node
manager's context again. The per-request cache keeps the first node
manager's copy without an extra object and up to eight more, so a
request that touches several node managers no longer evicts the other
one's copy. `ServerSystemContext` copies itself with a copy constructor
instead of `MemberwiseClone` (which allocates through a runtime call)
unless it is a derived type; `SessionSystemContext` gains a protected
copy constructor. Copies per Read went from about three to one.

**Q2: lock-free request registration in `RequestManager`.** #4742 took
the server-wide lock off the request path, but the two
`ConcurrentDictionary`s for executing requests and open validation
scopes stayed the main waiting point after the channel gate (four writes
per request from every worker). `RequestTable<T>` places an entry with a
compare-and-swap into a free slot near a scrambled hash of its key, and
the entry remembers its slot so completion is one more compare-and-swap;
only when every probed slot is taken does an entry go to an overflow set
under a lock. Drains, Cancel, Session close, the timeout sweep and
`Dispose` read the slots one after the other, which keeps the existing
ordering arguments (registration fence vs. Session/Cancel marks, drain
count published before the snapshot). Sharding the dictionaries over
per-processor locks was tried first and only moved about 5 % of waits
into the shard locks.

**Q3: no timer or shared-token registrations per request.**
`RequestLifetime`'s timeout constructor, used by the endpoint for every
request, no longer creates a `Timer` and no longer registers with the
server's request-queue token and the transport token, which every worker
registered with and so contended on. The deadline and the two tokens are
watched by `RequestDeadlineMonitor`: one sweep every 20 ms over
intrusive lists sharded by processor, running only while requests are
monitored. The `params` constructor keeps its synchronous registrations.

**Q4: namespace metadata lookups without a lock.**
`NamespaceMetadataRegistry` reads its caches (found and missing, by URI
and now also by index) without its lock; only filling and invalidating
take it. The generation moves before the caches are cleared, so a lookup
racing an invalidation never reports a stale miss.

**Q5: fewer per-call allocations on the read path.**
`ReadValidationAttributes` fills its attribute-id array element by
element instead of with an array initializer
(`RuntimeHelpers.InitializeArray` per call); single-node Read and Write
requests skip the per-request attribute cache and validate with
`permissionsOnly` (the same three attributes, read once); `ReadAsync`
allocates the list of nodes to validate only when needed. (Review: a
shared static attribute array, a per-thread `Variant[]` scratch array
and a per-thread Read node cache were dropped again because they crossed
virtual calls; the measurements below were taken with them, so Q5's
allocation saving is a little smaller than shown.)

## Behaviour notes

- **Request timeouts (Q3):** a `TimeoutHint` now fires up to 20 ms late
(the sweep period), and cancelling the server's or the transport's token
reaches the request's `CancellationToken` up to 20 ms later instead of
synchronously. A token that is already cancelled cancels the request at
once, and `TryCancel` checks the tokens first, so a token cancelled
before an explicit Cancel still owns the terminal state. The
exactly-once terminal state of #4744 is unchanged: the sweep claims it
through the same compare-and-swap.
- **System contexts (Q1):** a custom node manager that changed fields of
the context it got in a service call now changes the copy shared by that
request's other calls into the same node manager; it never reaches other
requests or the node manager's own `SystemContext`.
- **Request ids (Q2):** `RequestManager.RequestReceived` no longer
throws for a different `OperationContext` with a request id that is
still registered. Requests are tracked by their context; ids come from a
server-wide counter.

## Testing

- New: `ServerSystemContextCopyTests` (every field copied, derived types
kept), `AsyncCustomNodeManagerTests` (one copy per node manager and
request, two managers in one request, not shared across requests;
single- vs multi-node Read verdicts identical incl. a node read twice),
`RequestTableTests` (idempotent add, exactly-once remove, overflow,
concurrent churn while enumerating; drain racing completion on many
threads; Session close and Cancel by handle exactly once while others
register), `RequestDeadlineMonitorTests` (timeout precision with a fake
clock, token propagation, ownership vs. explicit Cancel, no leaked
entries, sweep/cancel/complete race), `NamespaceMetadataRegistryTests`
(cached miss by index invalidated, lookups racing `Invalidate`,
permission changes visible through the cached node).
- Run locally (net10.0): Opc.Ua.Types.Tests, Opc.Ua.Core.Tests (Stack),
Opc.Ua.Server.Tests, Opc.Ua.Sessions.Tests, Opc.Ua.Subscriptions.Tests,
Opc.Ua.Redundancy.Server.Tests, Opc.Ua.Stress.Tests Contract +
Integration. net48 builds.
- Measured with `perf/ServerLoadHarness` scenarios `o6read` (o6 harness,
not part of this PR), `read`, `browse`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 06:25:31 +02:00

148 lines
7.3 KiB
C#

/* ========================================================================
* Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved.
*
* OPC Foundation MIT License 1.00
*
* Permission is hereby granted, free of charge, to any person
* obtaining a copy of this software and associated documentation
* files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following
* conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*
* The complete license agreement can be found here:
* http://opcfoundation.org/License/MIT/1.00/
* ======================================================================*/
#pragma warning disable CA2000
using System.Threading.Tasks;
using Moq;
using NUnit.Framework;
using Opc.Ua.Tests;
namespace Opc.Ua.Server.Tests
{
public partial class AsyncCustomNodeManagerTests
{
private static readonly NodeId s_readerRoleId = new("ReadPermissionReader", 0);
/// <summary>
/// Verifies that a Read through the service dispatcher gives the same per-node
/// permission verdicts and values whether a node is read alone (no per-request
/// attribute cache) or together with other nodes (cached attributes), including a
/// node read twice in one request.
/// </summary>
[Test]
public async Task ReadPermissionVerdictsAreTheSameForSingleAndMultiNodeRequestsAsync()
{
m_mockServer.Setup(server => server.Telemetry).Returns(NUnitTelemetryContext.Create());
using var resources = new ResourceManager(m_configuration);
m_mockServer.Setup(server => server.ResourceManager).Returns(resources);
ITestNodeManager manager = CreateManager();
var core = new Mock<ICoreNodeManager>();
core.Setup(value => value.SyncNodeManager).Returns(Mock.Of<INodeManager>());
var factory = new Mock<IMainNodeManagerFactory>();
factory.Setup(value => value.CreateConfigurationNodeManager())
.Returns(Mock.Of<IConfigurationNodeManager>());
factory.Setup(value => value.CreateCoreNodeManager(It.IsAny<ushort>()))
.Returns(core.Object);
m_mockServer.Setup(server => server.MainNodeManagerFactory).Returns(factory.Object);
await using var dispatcher = new MasterNodeManager(m_mockServer.Object, m_configuration, null, manager);
m_mockServer.Setup(server => server.NodeManager).Returns(dispatcher);
NodeId allowed = await AddPermissionVariableAsync(manager, "ReadAllowed", 7, PermissionType.Browse | PermissionType.Read)
.ConfigureAwait(false);
NodeId denied = await AddPermissionVariableAsync(manager, "ReadDenied", 8, PermissionType.Browse)
.ConfigureAwait(false);
var identity = new Mock<IUserIdentity>();
identity.Setup(i => i.GrantedRoleIds).Returns([s_readerRoleId]);
var session = new Mock<ISession>();
session.Setup(s => s.Id).Returns(new NodeId("ReadPermissionSession", 0));
session.Setup(s => s.EffectiveIdentity).Returns(identity.Object);
session.Setup(s => s.Identity).Returns(identity.Object);
session.Setup(s => s.PreferredLocales).Returns([]);
DataValue singleAllowed = await ReadOneAsync(dispatcher, session.Object, allowed).ConfigureAwait(false);
DataValue singleDenied = await ReadOneAsync(dispatcher, session.Object, denied).ConfigureAwait(false);
using var context = new OperationContext(session.Object, DiagnosticsMasks.None);
(ArrayOf<DataValue> many, _) = await dispatcher.ReadAsync(
context,
0,
TimestampsToReturn.Neither,
[
new ReadValueId { NodeId = allowed, AttributeId = Attributes.Value },
new ReadValueId { NodeId = denied, AttributeId = Attributes.Value },
new ReadValueId { NodeId = allowed, AttributeId = Attributes.Value }
]).ConfigureAwait(false);
Assert.Multiple(() =>
{
Assert.That(singleAllowed.StatusCode, Is.EqualTo(StatusCodes.Good));
Assert.That((int)singleAllowed.WrappedValue, Is.EqualTo(7));
Assert.That(singleDenied.StatusCode, Is.EqualTo(StatusCodes.BadUserAccessDenied));
Assert.That(many.Count, Is.EqualTo(3));
Assert.That(many[0].StatusCode, Is.EqualTo(singleAllowed.StatusCode));
Assert.That(many[0].WrappedValue, Is.EqualTo(singleAllowed.WrappedValue));
Assert.That(many[1].StatusCode, Is.EqualTo(singleDenied.StatusCode));
Assert.That(many[2].StatusCode, Is.EqualTo(singleAllowed.StatusCode));
Assert.That(many[2].WrappedValue, Is.EqualTo(singleAllowed.WrappedValue));
});
}
private static async Task<NodeId> AddPermissionVariableAsync(
ITestNodeManager manager,
string name,
int value,
PermissionType readerPermissions)
{
ServerSystemContext context = manager.SystemContext;
ushort nsIdx = manager.NamespaceIndexes[0];
var variable = new BaseDataVariableState(null);
variable.CreateAsPredefinedNode(context);
variable.NodeId = new NodeId(name, nsIdx);
variable.BrowseName = new QualifiedName(name, nsIdx);
variable.DataType = DataTypeIds.Int32;
variable.ValueRank = ValueRanks.Scalar;
variable.AccessLevel = AccessLevels.CurrentRead;
variable.UserAccessLevel = AccessLevels.CurrentRead;
variable.Value = value;
variable.RolePermissions =
[
new RolePermissionType { RoleId = s_readerRoleId, Permissions = (uint)readerPermissions }
];
await manager.AddNodeAsync(context, default, variable).ConfigureAwait(false);
return variable.NodeId;
}
private static async Task<DataValue> ReadOneAsync(
MasterNodeManager dispatcher,
ISession session,
NodeId nodeId)
{
using var context = new OperationContext(session, DiagnosticsMasks.None);
(ArrayOf<DataValue> values, _) = await dispatcher.ReadAsync(
context,
0,
TimestampsToReturn.Neither,
[new ReadValueId { NodeId = nodeId, AttributeId = Attributes.Value }]).ConfigureAwait(false);
Assert.That(values.Count, Is.EqualTo(1));
return values[0];
}
}
}