Files
UA-.NETStandard/docs/RateLimiting.md
T
Marc Schier 57c0a32ed4 Fix incomplete-message buffering and channel cleanup (#4522)
# Description

Fix incomplete-message buffering, channel cleanup, session resource
accounting, and enforcement of existing transport admission settings.

- Share a targeted `ChunkReassemblyBudget` across server listeners. The
reference default is 64 MiB, with a 32 MiB global occupancy threshold
for sessionless intermediate chunks. This is priority headroom for
activated sessions, not a guaranteed bootstrap partition. Single/final
chunks and unrelated response/client buffers remain outside this budget.
- Retain precise incoming/final payload quotas, valid Abort handling
without channel teardown, terminal late-final rejection, and
exactly-once buffer/reservation cleanup.
- Use one assembly timer per quota/clock scope, asynchronous cleanup, a
fixed deadline, a 30-second fallback for non-positive lifetimes, and
best-effort ERR `BadTimeout` before closure.
- Apply negotiated TCP/WebSocket/Pipe receive limits and size pipe
rentals to actual chunks.
- Add the optional `ISessionBindingProvider` and immutable
`SessionBindingContext` seam. Managed membership counts distinct
committed sessions, not responses, and handles reactivation, transfer,
closure, expiry, restoration and shutdown. Read-only snapshots require
token/channel/security validation and never replace service
authorization.
- Wire default and explicitly injected binding providers through raw
TCP, Kestrel and UACP WebSockets; preserve existing callback/manager
interfaces and direct construction.
- Enforce existing `ConnectionRateLimiter` and channel capacity settings
on Kestrel TCP and UACP WebSocket admission, with physical-transport
leases surviving reverse handoff and releasing on cancellation, close,
failed attachment and stop. HTTP/JSON requests are not treated as UASC
channels.
- Add regression coverage and documentation. Stabilize the CI
cancellation test by waiting for the resolver to enter before canceling.

This is the foundation only. New Balanced/FairShare/TrustedReservations
policies, tenant isolation and guaranteed bootstrap reserves belong in
the separate stacked follow-up, not this PR.

## Related Issues

Fixes #4521.

## Validation

| Scope | Result |
| --- | --- |
| Core transport, budget, binding-context and DI tests, net10.0 | 717
passed |
| Core transport, budget, binding-context and DI tests, net48 | 664
passed, 6 platform-specific skips |
| Session bindings, transfer/security/expiry/restore and hosted DI,
net10.0 | 69 passed |
| Same focused Server suite, net48 | 69 passed |
| ServerRedundancyHandlerTests, Release/net48 | 39 passed |
| ConsoleReferenceServer net10.0 | Build succeeded, zero warnings/errors
|
| Server and HTTPS binding netstandard2.1 | Both builds succeeded, zero
warnings/errors |

The new admission fixtures also passed five repeated runs after
correcting asynchronous connection-context disposal. Full-solution
validation, coverage comparison and CI for this update remain pending.

## Checklist

- [ ] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [ ] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [x] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [ ] I have addressed **all** PR feedback received.
2026-09-24 20:12:55 +02:00

12 KiB
Raw Blame History

Rate limiting and admission control

The server applies deterministic, configurable admission control so it sheds a connection or session-establishment storm with a fast, standard "busy" signal instead of collapsing (see Server Session Scalability for the underlying analysis). Clients react to that signal by backing off adaptively rather than retrying blindly. The primitives are built on System.Threading.RateLimiting, so the algorithm is pluggable and configurable through dependency injection.

Rate limiting is on by default with conservative limits sized so normal and bulk-but-well-behaved load is unaffected; only a storm is shed. Tune or disable it per deployment, or replace the whole limiter provider via DI.

Server side

What is limited

  • Inbound connections (opc.tcp listener): a per-second token bucket (with a burst) admits new connections; beyond the burst a connection is shed cheaply so the CPU-bound secure-channel handshake is protected. The listener socket backlog is configurable and defaults to 512 (raised from the previous hard-coded 10) so a burst of simultaneous connects is absorbed rather than dropped by the OS.
  • Session establishment (CreateSession / ActivateSession): a concurrency limiter bounds the number of in-flight establishment operations so a connect storm cannot saturate every core and starve steady-state publish delivery. When at capacity the server returns BadServerTooBusy with a retry-after hint in the fault, before doing the expensive certificate validation / signing.
  • Incomplete messages: one ChunkReassemblyBudget bounds retained intermediate-message buffers across the server's UA-TCP, Kestrel TCP, and UA Secure Conversation WebSocket listeners. This capacity limit is independent of ServerRateLimitOptions.Enabled.

Status codes

Situation Status code
Session establishment at capacity BadServerTooBusy (transient — the client should back off and retry)
Hard session cap reached (MaxSessionCount) BadTooManySessions
Connection shed at the listener the connection is dropped; the client sees a transport error and, together with any subsequent BadServerTooBusy, backs off
Incomplete-message budget exhausted BadTcpNotEnoughResources followed by channel closure; if the error cannot be sent, the client observes closure

Configuration

The deterministic limits live in ServerRateLimitOptions:

Option Default Meaning
Enabled true Master switch for all rate limiting.
ListenBacklog 512 Listener socket pending-connection backlog.
ConnectionRateLimitEnabled true Whether inbound connections are rate limited.
ConnectionsPerSecond 500 Sustained connection admission rate.
ConnectionBurst 1000 Connection burst capacity (token-bucket size).
SessionRateLimitEnabled true Whether session establishment is limited.
MaxConcurrentSessionEstablishment max(ProcessorCount * 8, 128) Concurrent in-flight CreateSession/ActivateSession.
SessionEstablishmentQueueLimit 0 Waiters before rejecting with BadServerTooBusy (0 = reject immediately).

Configure it through the DI hosting options:

services.AddOpcUa()
    .AddServer(options =>
    {
        options.ApplicationName = "MyServer";
        options.ConfigureRateLimits = limits =>
        {
            limits.ConnectionsPerSecond = 200;
            limits.MaxConcurrentSessionEstablishment = 64;
        };
    });

To plug in a completely different algorithm, register a custom IServerRateLimiterProvider in DI (it takes precedence over ConfigureRateLimits):

services.AddSingleton<IServerRateLimiterProvider>(sp =>
    new MyCustomRateLimiterProvider(...));

Without dependency injection, set the options or provider directly on the server before it starts:

var server = new StandardServer(telemetry);
server.RateLimitOptions = new ServerRateLimitOptions { ConnectionsPerSecond = 200 };
// or: server.RateLimiterProvider = new DefaultServerRateLimiterProvider(options);

Raw TCP, Kestrel TCP, and UACP WebSocket listeners consume the IConnectionRateLimiter supplied through TransportListenerSettings, injected by StandardServer.ConfigureTransportListenerSettings. A custom server can override that hook to supply its own limiter. The default TokenBucketConnectionRateLimiter wraps a System.Threading.RateLimiting.TokenBucketRateLimiter. Kestrel and UACP WebSocket listeners also enforce their configured channel cap across concurrent admissions and release capacity with the physical connection, including reverse handoff. The raw listener retains its existing admission and unused-channel reclamation behavior.

Incomplete messages

ChunkReassemblyBudget (Opc.Ua.Bindings) counts the backing-array length of each retained intermediate chunk, including pool rounding and metadata. All listeners of a server share the same budget. Completing, replacing, aborting, faulting, or closing a partial message releases its reservation.

The budget applies only while messages await further chunks, not to the whole process heap, socket buffers, final chunks, decoded requests, or outgoing responses. A request that fits in one chunk can still be served while the reassembly budget is full. A chunk that exceeds available capacity is rejected without waiting; the partial message is discarded and its channel is closed.

The default is sixteen times TransportQuotas.MaxMessageSize, clamped to 64 MiB through 1 GiB, then raised if necessary to accommodate four maximum-sized messages. A configuration with unlimited message size receives a 1 GiB budget. Both the stack's 2 MiB and the reference server's 4 MiB message limits select 64 MiB. ChunkReassemblyBudget.GetDefaultMaxBytes(maxMessageSize) returns this value, and ChunkReassemblyBudget.CreateDefault(configuration) creates a budget from an EndpointConfiguration using the same sizing policy.

Channels without an activated session may reserve only while total usage stays at or below MaxBytesWithoutSession, which defaults to half of MaxBytes. Channels with activated sessions can use the remaining headroom. This is a capacity policy, not an authentication boundary: an activated anonymous session also qualifies. The two-argument constructor sets the sessionless threshold explicitly, including the entire budget for sessionless workloads.

The threshold is not reserved space for new clients. For example, with a 100 MiB total and 50 MiB sessionless threshold, 60 MiB retained by activated sessions prevents further sessionless intermediate chunks. Single-chunk requests are still processed, but a large multi-chunk OpenSecureChannel, CreateSession, or ActivateSession may be refused until occupancy drops. Per-caller fairness and guaranteed bootstrap reservations require a separate isolation policy; the shared budget alone does not provide them.

Configure the hosted server through its fluent builder:

services.AddOpcUa()
    .AddServer(options => options.ApplicationName = "MyServer")
    .WithChunkReassemblyBudget(256L * 1024 * 1024);

For direct construction, assign the budget before startup:

var server = new StandardServer(telemetry)
{
    ChunkReassemblyBudget = new ChunkReassemblyBudget(256L * 1024 * 1024)
};

The same instance can be shared by several servers. A host that opens listeners itself passes it through TransportListenerSettings.ChunkReassemblyBudget; otherwise each standalone listener creates its own appropriately sized budget. For Dependency Injection (DI), the two-argument builder overload sets the sessionless threshold and validates both limits immediately:

services.AddOpcUa()
    .AddServer(options => options.ApplicationName = "MyServer")
    .WithChunkReassemblyBudget(256L * 1024 * 1024, 64L * 1024 * 1024);

A directly constructed ChunkReassemblyBudget(maxBytes, maxBytesWithoutSession) can also be registered as a singleton when several servers should share it.

Server channels also enforce a fixed assembly deadline using ChannelLifetime. Continuation traffic cannot restart it. A shared quota/clock timer schedules asynchronous cleanup independently of listener inactivity sweeps. Non-positive lifetimes use the 30-second default for assembly, and expiration reports BadTimeout before closure when possible; see incomplete-message resource limits.

HTTPS / Kestrel transport

The HTTPS/WSS transport is Kestrel-hosted and can use the ASP.NET Core rate limiter middleware. Because that listener builds its own isolated web host, attach the limiter through the stack's DI with AddHttpsRateLimiter (net8+):

services.AddOpcUa()
    .AddHttpsTransport()
    .AddHttpsRateLimiter(); // default: global 100 req/s fixed window, rejects with HTTP 429

Pass an Action<RateLimiterOptions> to fully configure the limiter with any System.Threading.RateLimiting policy. This registers a contributor that calls services.AddRateLimiter(...) and app.UseRateLimiter() on the listener's isolated Kestrel host; rejected requests return HTTP 429.

Client side

A client that gets a "server busy" signal must not hammer the server with retries — that is exactly what amplifies a connect storm. The default ReconnectPolicy (used by ManagedSession) is server-signal-aware: when the previous attempt failed with an overload signal (BadServerTooBusy, BadTcpServerTooBusy, BadTooManySessions, BadTooManyOperations, BadTooManyPublishRequests, or a transient timeout) it backs off more aggressively (4× the computed delay, capped at MaxDelay) and honors a server-provided retry-after hint as a lower bound.

This is exposed through IReconnectPolicy.TryGetNextDelay, which adapts the backoff to the previous attempt's status code and any server-provided retry-after hint. The connection state machine calls it automatically; a policy that returns false opts out of adaptive behavior and the state machine falls back to the basic attempt-based GetNextDelay, so a minimal custom policy only has to implement the plain delay. Because a failed initial connect funnels into the same reconnect loop, the adaptive backoff applies to both initial connects and reconnects.

ReconnectPolicy.IsServerBusySignal(StatusCode) classifies whether a status code is an overload signal, for use in custom policies.

To keep a bulk connect from bursting, a client-wide connect admission gate ramps concurrent initial connects: ManagedSessionBuilder.WithConnectRateLimiter(maxConcurrency) (or a shared RateLimiter / IClientConnectGate) bounds how many ManagedSession.CreateAsync calls establish at once; the excess wait in an unbounded queue rather than being rejected, so many sessions to one server ramp up smoothly instead of stampeding. It is off by default; through DI use the ConnectRateLimiterMaxConcurrency client option.

Server backpressure signals

  • Structured retry-after: the server carries a machine-readable retry-after to the client independently of diagnostics — as a RetryAfterMs value in ResponseHeader.additionalHeader on a BadServerTooBusy ServiceFault, and as the standard HTTP Retry-After header on the HTTPS 429. The client honors both via IReconnectPolicy.TryGetNextDelay. The UA-TCP client also honors a RetryAfterMs=N token in a transient server-busy Error message reason as a lower bound on channel-reconnect backoff. The legacy RetryAfterMs=N AdditionalInfo token remains a best-effort compatibility hint. See Server retry-after backpressure for how the client honors these signals during reconnect.
  • Load-based Server.ServiceLevel: the reference server computes Server.ServiceLevel from session-establishment headroom (255 at low load, scaling toward a floor as sessions approach MaxSessionCount, with hysteresis), so a client can read/subscribe to it as a proactive capacity signal.

See also