Files
UA-.NETStandard/tests/Opc.Ua.Client.Tests/Subscription/MessageProcessorTests.cs
romanett 87c35b84b6 Fix Opc.Ua.Client audit findings (security fallback, reconnect, subscriptions, redundancy, browse, files, WebApi) (#4572)
## Summary

Fixes the findings of a bug audit of `Opc.Ua.Client` and
`Opc.Ua.Client.ComplexTypes`, done on master plus #4516 (now merged).

- **Audit:** 82 findings across 12 areas, each checked against the OPC
UA specification. Final severities: 2 High, 37 Medium, 43 Low.
- **Fixes:** 76 fixed, 4 partial, 2 skipped (see [Not
fixed](#not-fixed--partial)).
- **Review:** a follow-up review of the fixes found 15 more defects in
them, all fixed here.
- **Tests:** every fix has a regression test, except a few races that
cannot be reproduced deterministically; the commit messages say which.

The finding IDs (`L<area>-<n>` for the audit, `G<n>` for the review)
appear in the commit titles.

## Behaviour and user-facing changes

### Security and endpoint selection
- **Reconnect no longer drops to weaker security (L3-1, High).** When
the server rejects the configured security policy or mode during
reconnect, `SessionReconnectHandler` no longer switches to whatever
endpoint the server advertises, None included. It only picks an endpoint
at least as strong as the configured one, judged by a locally computed
security level. The server-advertised `SecurityLevel` is not trusted. If
nothing qualifies, reconnect fails with the original error (new log
event `SessionReconnectHandler + 13`) and keeps retrying with the
original settings.
- **`CoreClientUtils.SelectEndpoint(useSecurity: true)` never returns a
SecurityMode None endpoint over opc.tcp (L9-9, High).** It now returns
`null` when no secure endpoint matches.
- When the discovery URL is `https`/`opc.https`, an HTTPS endpoint with
SecurityMode None is still accepted as a last resort, because TLS
protects it (Part 6 §7.4.1) (G17).
  - Endpoints with SecurityMode `Invalid` no longer count as secure.
- `GlobalDiscoveryServerClient` and `ServerPushConfigurationClient` now
throw `BadSecurityPolicyRejected` when no secure endpoint matches,
instead of passing `null` on. The ConsoleReferenceClient sample reports
this with a clear error.
- **No Bearer/Basic credentials over `http://` (L12-2).**
`WebApiClient.Create` and WebApi channel open/reconnect reject them with
`BadSecurityChecksFailed`. The public `WebApiClient(HttpClient, …)`
constructor also rejects them when the client's `BaseAddress` is not
https, and every send checks the address it actually goes to (covers a
`BaseAddress` set later).

### Session, keep-alive and identity
- **Faster detection of a silent server (L1-1).** The keep-alive Read is
now bounded by its `TimeoutHint` (2 × `KeepAliveInterval`), which is
what Part 4 §7.32 says the client should do. It then fails with
`BadNoCommunication`, so reconnect starts after about 2 ×
`KeepAliveInterval` instead of `OperationTimeout` (120 s by default).
- **Operations fail fast during a channel-manager recovery (L1-2, G9).**
`UpdateIdentityAsync`, `TransferSubscriptionsAsync` and
`ReactivateSubscriptionsAsync` fail with `BadSecureChannelClosed`
(retryable) instead of deadlocking. This covers the operation holding
the reconnect lock and any queued behind it.
- **Closing or disposing from a handler no longer waits on itself (L1-3,
G5, G6).**
- `Session` and `ManagedSession` Close/Dispose from a Notification
handler no longer wait 30 s for their own background drain.
- Disposing a `ManagedSession` from a keep-alive `PublishStatusChanged`
handler no longer deadlocks.
- Tasks spawned by a handler no longer count as "the handler" (the owner
marker is scoped to the work item).
- **Switching to an ECC user-token policy takes two ActivateSession
calls (L1-4, G4).** This follows Part 6 §6.8.2 (fetch the ephemeral key,
then activate).
- If the server returns no key, the switch fails with
`BadSecurityPolicyRejected`.
- If the second activation is rejected, the old identity stays on the
override policy with the key just fetched. Restoring the old key was
tried: against the reference server the next reconnect then failed to
decrypt the identity token.
- **`ManagedSession.ConfiguredEndpoint` follows failover (L2-6).** It
now reports the inner session's endpoint, so it changes after a failover
or network-path rotation.
- **An explicit identity update replaces the identity provider (L2-7).**
`ManagedSession.UpdateSessionAsync(identity)` stops the provider's
refresh loop and drops the provider for good. If the update fails, the
loop is restarted.
- **Reconnect waits for in-flight calls on a healthy channel (L2-3,
G8a).** Reconnect/failover on a healthy channel (certificate or
trust-list change, manual reconnect) waits for in-flight service calls.
The wait is bounded (max(`KeepAliveInterval`, 1 s)) only once the
channel is unhealthy (keep-alive stopped, channel not Ready), so a dead
channel no longer holds recovery for `OperationTimeout`.
- **Smaller fixes:**
- Restore reads the server certificate as a chain; this failed on macOS
(L1-5).
  - Session recovery can no longer run twice at once (L1-6).
  - Identity expiries more than 49.7 days out no longer throw (L2-2).
- Keep-alive reconnect suppression is recalculated on a lease swap
(L2-4).

### Reconnect state machine and reverse connect
- **⚠ `MaxTotalReconnectTime = 0` now means "no time limit" (L3-7).
Reviewer decision needed.** Before, 0 disabled reconnect entirely.
Negative values other than `Timeout.InfiniteTimeSpan` now throw
`ArgumentOutOfRangeException` when the state machine is constructed;
before, they crashed the worker. The XML docs of
`ReconnectPolicy`/`ReconnectPolicyOptions` are updated. If 0 should keep
meaning "no reconnect", this is a one-line revert.
- **A close racing the end of the reconnect budget is no longer lost
(L3-4).** It goes through the normal close instead of switching to
failover.
- **The "reconnect worker" marker covers one step only (L2-1).** It
lasts for a single connect/reconnect/failover/close step. Keep-alive,
publish and identity loops started in a step no longer count as the
worker afterwards.
- **Reverse-connect strategies `Undefined` and `Any` alone are removed
after the first callback (L8-1).** This matches the documentation;
before, they behaved like `Always`.
- **`AddEndpoint` for a URL that is already added disposes the replaced
listener (L8-2).**

### Service calls
- **Batched calls clean up after a failed later batch (L3-2).** When a
later batch of
`Browse`/`BrowseNext`/`HistoryRead`/`CreateMonitoredItems` fails, the
continuation points and monitored items created by earlier batches are
released or deleted before the exception is rethrown. This adds one
request. `AddNodes`/`RegisterNodes` are not undone.
- **Batched diagnostic infos point at the right strings (L3-3).** The
string-table indexes are rebased when diagnostics are padded.
- **`ReadBytesAsync` rejects oversized responses (L3-5).** It throws
`BadUnknownResponse` when the server returns more bytes than requested,
and `BadEncodingLimitsExceeded` beyond `int` range.
- **`ReadByteStringInChunksAsync` handles an unknown server limit
(L3-6).** When the server's `MaxByteStringLength` is 0 (absent) or
larger than `int`, it falls back to the client's
`MessageContext.MaxByteStringLength`. It throws only when both are
unknown.

### Browse and NodeCache
- **Empty browse pages with a continuation point are followed (L9-1,
L9-4, L9-5, G16).** Part 4 §5.9.2.1 allows a server to return an empty
page with a continuation point, so browsing keeps following it. After 10
consecutive empty rounds it gives up with `BadNoData` and releases the
continuation point.
- `BrowseAsync(NodeId)` now throws instead of silently returning a
truncated list.
  - `BrowseStreamAsync` yields a `BadNoData` result for that node.
- The same cap, through one shared helper, now also protects
`RoleManagementClient.ListRolesAsync` and the FileSystem
enumerate/empty-check loops.
- **Continuation points are released on failure (L9-3).**
`BrowseAsync(NodeId)` releases them on any exception, and
`BrowseStreamAsync` releases all points of the current page when the
consumer stops early.
- **NodeCache lookups propagate cancellation (L9-6, L9-8).**
`FindAsync`, `IsKnownAsync`, `FetchNodeAsync`,
`FindReferenceTypeNameAsync`, the reference-table fill and the
type-hierarchy search now let `OperationCanceledException` through.
Before, a cancelled lookup looked like "node does not exist".
- **A NodeClass mismatch fails only that node (L9-7).** It returns
`BadNodeClassInvalid` for that node instead of throwing
`KeyNotFoundException` for the whole batch.
- **Smaller fixes:**
- `MaxNodesPerBrowse` ≥ 2^31 no longer breaks the managed browse (L9-2).
  - Null discovery URLs are skipped (L9-10).
  - `ModelChangeTracker` no longer cancels a joined start (L9-11).
- **Log changes:** `BrowserContinuationPointExistsButBrowse` (Browser +
0) was removed, and `BrowserServerReturnedEmptyReferencesBut` now
includes the round count.

### Subscriptions (default engine)
- **Recovery after a transfer delivers each message exactly once (L5-1,
G22).** Messages already dispatched are acknowledged instead of
republished and delivered again. Messages whose earlier republish failed
are now republished and delivered, with the Republish flag and after
newer messages, instead of being acknowledged and lost.
- **A keep-alive that proves a gap triggers republish at once (L5-5,
G23).** The dedup mark moves only past messages that were actually
recovered, so a failed republish is retried by the next data message.
- **Item changes that fail are retried with backoff (L5-3).** They are
retried on the next apply pass after the state manager's 250 ms–5 s
backoff, instead of six times back to back. The six attempts now span
about 13 s.
- **Extreme revised intervals no longer throw (L5-4, L6-1).** Very large
or NaN intervals and counts revised by the server are capped (NaN falls
back to the requested value). Before, the overflow could stop every
publish worker.
- **`SetTriggeringAsync` from a callback no longer hangs (L5-7, G21).**
While the subscription is deleted, recreated or disposed, it returns
`BadOperationAbandoned`. A cancelled reset no longer makes later calls
fail.
- **Publish responses for unknown subscription ids (L6-3, L6-4, G20).**
The client no longer re-sends acknowledgements the server already
processed. An unknown id seen while a creation is pending is left alone
instead of being deleted.
- **Smaller fixes:**
- A null and an empty `IndexRange` are treated as equal, and restored
snapshots use `null` (L5-6).
  - A failed partition attach during restore is rolled back (L6-2).
- `StreamingSubscription` throws `ObjectDisposedException` after dispose
(L6-6).

### Subscriptions (classic `Subscription`)
- **Republish no longer abandons other missing messages (L7-1).** A
Republish response no longer overwrites `AvailableSequenceNumbers`, so
other missing messages are not given up while the server still holds
them.
- **Late copies are no longer delivered twice (L7-2).** Copies of
already-processed sequence numbers are ignored, and creating a new
server subscription resets the sequence state.
- **Disposing from a keep-alive handler no longer deadlocks (L7-3).**
`Dispose()` from a keep-alive `PublishStatusChanged` handler can return
while that callback is still finishing.
- **Transfer of a restored subscription (L7-4, G27).**
  - It succeeds even when some saved items were never created.
- The server subscription is deleted only when this session's own
TransferSubscriptions just moved it and adoption failed definitively
(item count mismatch or a non-transient status). Transient errors and
`ReactivateSubscriptionsAsync` never delete it.
  - The public `Subscription.TransferAsync` never deletes.
- **Smaller fixes:**
  - Cloned items keep their triggering links (L7-5).
  - `CreateAsync` clears stale server ids (L7-6).
- A `Good_SubscriptionTransferred` for the client's own transfer is
ignored (L7-7).
- An item removed while its create is in flight is deleted afterwards
(L7-8).
- The publish timeout for very long keep-alives no longer overflows
(L7-9).
  - A negative `MaxMessageCount` is handled (L7-10).
  - `LastMessage` tolerates a wrong notification type (L7-11).
  - Pending acks exactly 2^31 away no longer throw (L4-1).
- **Log changes:** new log events `Subscription + 68..70`.

### Redundancy
- **Warm/Hot demotion (L4-2, G1, G3).**
- Without token reuse, the demoted leader closes its session and
reconnects as a fresh standby.
- With token reuse, it abandons the shared server session locally,
without CloseSession, so the new leader can reactivate it and keep its
subscriptions. This uses the new internal methods
`ManagedSession.AbandonServerSessionAsync` and
`Session.AbandonServerSession`.
- `RoleChanged(false)` is now raised before the standby connect and no
longer depends on it, so `CurrentSession` can briefly be `null`.
- Exceptions from `RoleChanged` handlers are logged and do not stop the
teardown.
- **Promotion (L4-3, G2).** A promoted standby that adopts the leader's
session first closes its own server session. A retried promotion keeps
the session it already adopted.
- **Client failover takeover (L4-4, partial).** Subscriptions the backup
prepared in advance (not yet created, `TransferId` set to the active
client's id) are moved through the session's own transfer and keep
publishing. `docs/HighAvailability.md` is updated.
- **`ShouldFailover` through the interface (L4-6).** Calling it through
`IServerRedundancyHandler` no longer fails over away from a server that
is still Healthy.

### Fluent / dependency injection
- **Shared DI session (L10-5).** The shared session from `AddClient` is
reconnected after a caller disposes it, and is disposed with the
container. After that the connect delegate throws
`ObjectDisposedException`.
- **New public API:
`ManagedSessionBuilder.WithServerRedundancyOptions(ServerRedundancyOptions)`
(L10-6).** This is additive. The `Session.ServerRedundancy` options are
now actually applied.
- **`AddDiscoveryAndConnect` rewrites the advertised address (L10-7).**
It replaces the host and port the server advertises with those of the
discovery URL (Part 4 §5.6.2.1, NAT/containers), and prefers endpoints
with the same scheme.

### State machines and role management
- **`GetSubStateMachineAsync` returns the instance's own sub-state
machine (L10-1).** Before, it returned the type's placeholder.
- **`WaitForStateAsync` reports failures with OPC UA status codes
(L10-2).** It throws `BadNotFound` when `CurrentState/Id` cannot be
resolved, and `BadInvalidState` when observation ends without reaching
the state. Before, both surfaced as `OperationCanceledException`.
- **`RoleManagementClient.ListRolesAsync` pages correctly (L10-3).** It
follows continuation points, releases them on any failure, and throws on
a Bad browse status instead of returning an empty list.
- **Role properties are read into the right fields (L10-4).** Properties
whose namespace cannot be resolved are skipped instead of shifting the
other values.

### File system, temporary files and alias names
- **Path escaping (L11-8, G12).**
- Path segments escape `:` and `/` with `&`, and `&` itself as `&&`
before those characters or at the end of a name.
- `Format` and `Parse` round-trip all names. Typed input ending in `&/`
now reads the slash as part of the name.
- A segment without a namespace prefix also matches a namespace-0 child
when nothing matches in the parent's namespace.
- **Stale path cache (L11-1).** When a cached lookup fails,
`GetInfoAsync` retries once from the server. `ExistsAsync` and
`GetFileAsync` report "not found" instead of throwing.
- **`GenerateFileForReadAsync` waits for the file (L11-2).** It waits
until the file-generation state machine reaches ReadTransfer (`Idle`
counts as not started yet), and throws `IOException` if it reports
Error. The wait is bounded only by the caller's token.
- **Non-recursive delete refuses hidden children (L11-3).** It refuses
when the directory holds children that the subtype options hide from
enumeration. Before, the server deleted them all.
- **Alias names (L11-5, L11-6, L11-7).**
- `BadMethodInvalid` maps to `NotSupportedException` for all alias
calls, so `UseVerbose` falls back to `FindAlias`.
  - A duplicate alias name resolves to all its targets.
- If the refresh strategy fails to start, the resolver logs a warning
and keeps working without automatic invalidation. New log event block
`AliasNameResolver = 560`.
- **Null profile entries (L11-4).** Null `ServerProfileArray` entries no
longer crash `GetConformanceInfoAsync`.

### WebApi transport
- **Failures come back as `ServiceResultException` (L12-1).**
  - A timeout becomes `BadRequestTimeout`.
- HTTP 429/503 becomes `BadServerTooBusy` with the Retry-After hint, on
every target framework. `WebApiClient` checks the status itself (reusing
`HttpsTransportChannel.CreateServerTooBusyException`), so callers of
`WebApiClient` also get a `ServiceResultException` instead of an
`HttpRequestException` for these two statuses (G15).
  - An IO error becomes `BadNotConnected`.
- **Timeouts (L12-1, G14).**
- `OperationTimeout` applies to each request. When `RequestTimeout` is
unset, HttpClient's 100 s default no longer cuts off long Publish
requests.
- The GetEndpoints call at open is bounded by `OperationTimeout`, and
cancelling the open now throws.
- **Core change:** `HttpsTransportChannel.MapRequestFailure` went from
private to internal so the WebApi channel can reuse it.

### ComplexTypes
- **Out-of-range union selector (L12-3).** A union whose SwitchField is
larger than its field count now fails with `BadDecodingError` when
decoded and `BadEncodingError` when encoded (Part 6 §5.2.8).
- **Verbose JSON unions (L12-4).** They now decode their selected field
instead of `null`.

## Not fixed / partial
- **L2-5 (skipped):** an identity provider combined with a
caller-supplied reverse connection still can't connect to an
undiscovered endpoint. Fixing it means choosing the identity after
CreateSession, a design change.
- **L4-5 (skipped):** the one-time KeyCredential token needs a fresh
token on each reactivation. Today reactivation fails and the client
falls back to a new session.
- **L4-4 (partial):** only subscriptions the backup prepared in advance
survive a takeover.
- **L3-2 (partial):** `AddNodes`/`RegisterNodes` batches are not rolled
back.
- **L2-3 (partial):** service calls still hold the session lock for
their whole round trip.
- **Known from the review, not fixed here:**
  - G7: the batch cleanup request can't be cancelled.
- G11: the DI session's close at host shutdown never reaches the server,
and a closed session keeps being handed out.
- G13: a file renamed by another client is still found under its old
path.
  - G25: a clone-based transfer doesn't mark the original subscriptions.
  - G27b: the transfer item check matches by count, not by ID.
- **Outside this PR:** the reference server ignores `ECDHPolicyUri` on
ActivateSession (`SessionSecurityPolicyHelper`), which breaks Part 6
§6.8.2. As a result, the L1-4 two-step ECC switch ends in
`BadSecurityPolicyRejected` against our own server.

## Testing
All suites on net10.0, on the branch before the master merge. Every
suite had 0 failures:

| Suite | Passed | Skipped |
|---|---|---|
| Opc.Ua.Client.Tests | 2698 | 0 |
| Opc.Ua.Redundancy.Client.Tests | 134 | 0 |
| Opc.Ua.Subscriptions.Classic.Tests | 57 | 0 |
| Opc.Ua.Bindings.Https.WebApi.Tests | 385 | 0 |
| Opc.Ua.Sessions.Tests | 782 | 379 |
| Opc.Ua.Subscriptions.Tests | 640 | 6 |
| Opc.Ua.Client.ComplexTypes.Tests | 2858 | 0 |

After merging current master (net10.0), every suite had 0 failures
except one, a flaky new test fixed in 9f52952b2:

| Suite | Passed | Skipped | Failed |
|---|---|---|---|
| Opc.Ua.Client.Tests | 2719 | 0 | 1 (the flaky test) |
| Opc.Ua.Redundancy.Client.Tests | 134 | 0 | 0 |
| Opc.Ua.Client.ComplexTypes.Tests | 2858 | 0 | 0 |
| Opc.Ua.Subscriptions.Classic.Tests | 57 | 0 | 0 |
| Opc.Ua.Bindings.Https.WebApi.Tests | 397 | 0 | 0 |
| Opc.Ua.Sessions.Tests | 790 | 379 | 0 |
| Opc.Ua.Subscriptions.Tests | 640 | 6 | 0 |

The touched test projects also build on net48 with 0 warnings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: marcschier <11168470+marcschier@users.noreply.github.com>
Co-authored-by: Marc Schier <marcschier@users.noreply.github.com>
2026-09-29 09:34:03 +00:00

1917 lines
83 KiB
C#

/* ========================================================================
* Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved.
*
* OPC Foundation MIT License 1.00
*
* Permission is hereby granted, free of charge, to any person
* obtaining a copy of this software and associated documentation
* files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following
* conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*
* The complete license agreement can be found here:
* http://opcfoundation.org/License/MIT/1.00/
* ======================================================================*/
// CA2007: tests run without a SynchronizationContext; ConfigureAwait(false)
// adds noise without a behavioural benefit. Disabled file-level for the suite.
#pragma warning disable CA2007
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Moq;
using NUnit.Framework;
using Opc.Ua.Client.Subscriptions.Fakes;
using Opc.Ua.Tests;
namespace Opc.Ua.Client.Subscriptions
{
[TestFixture]
public sealed class MessageProcessorTests
{
[SetUp]
public void SetUp()
{
m_completion = new FakeMessageAckQueue();
m_telemetry = NUnitTelemetryContext.Create();
m_mockServices = new Mock<ISubscriptionServiceSetClientMethods>();
}
[Test]
public async Task DisposeAsyncShouldCompleteMessageWriterAndCancelTokenAsync()
{
// Arrange
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 3
};
// Act
await sut.DisposeAsync().ConfigureAwait(false);
// Assert
Assert.That(sut.PublishState, Is.EqualTo(PublishState.Completed));
Assert.That(m_completion.CompletedSubscriptions,
Is.EqualTo(new uint[] { 3 }));
// The subscription must be retired by identity, not only by id.
Assert.That(m_completion.CompletedProcessors,
Is.EqualTo(new IMessageProcessor[] { sut }));
}
/// <summary>
/// The ack-wait helper the tests below use must fail on its own
/// deadline rather than returning quietly. A silent return would push
/// the diagnosis down to whatever the caller asserts next - a bare
/// count mismatch that cannot tell an acknowledgement that was never
/// queued from one that merely arrived late - and would hide the race
/// entirely if that assertion were ever loosened.
/// </summary>
[Test]
public void WaitForQueuedAckAsyncShouldThrowWhenTheAckNeverArrives()
{
var queue = new FakeMessageAckQueue();
TimeoutException ex = Assert.ThrowsAsync<TimeoutException>(
async () => await queue.WaitForQueuedAckAsync(1, 200).ConfigureAwait(false));
Assert.That(ex.Message, Does.Contain("only 0 arrived"));
}
[Test]
public async Task OnPublishReceivedKeepAliveShouldDispatchKeepAliveAsync()
{
// Arrange
var message = new NotificationMessage
{
SequenceNumber = 3
};
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 3
};
await using (sut.ConfigureAwait(false))
{
// Act
await sut.OnPublishReceivedAsync(message, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
// Assert
Assert.That(sut.KeepAliveNotificationReceived.IsSet, Is.True);
Assert.That(sut.AvailableInRetransmissionQueue, Is.EqualTo(availableSequenceNumbers));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(3));
Assert.That(sut.DataChangeNotificationReceived.IsSet, Is.False);
Assert.That(m_completion.QueuedAcks, Is.Empty);
// Arrange
sut.KeepAliveNotificationReceived.Reset();
sut.DataChangeNotificationReceived.Reset();
message = new NotificationMessage
{
SequenceNumber = 4,
NotificationData =
[
new ExtensionObject(new DataChangeNotification
{
MonitoredItems =
[
new MonitoredItemNotification()
]
})
]
};
// Act
await sut.OnPublishReceivedAsync(message, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.DataChangeNotificationReceived.WaitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
// The processor dispatches the notification (which sets the
// event awaited above) and only then queues the acknowledgement,
// so asserting on QueuedAcks straight away races that last step.
await m_completion.WaitForQueuedAckAsync(1).ConfigureAwait(false);
// Assert
Assert.That(sut.AvailableInRetransmissionQueue, Is.EqualTo(availableSequenceNumbers));
Assert.That(sut.DataChangeNotificationReceived.IsSet, Is.True);
Assert.That(sut.KeepAliveNotificationReceived.IsSet, Is.False);
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(4));
Assert.That(m_completion.QueuedAcks, Has.Count.EqualTo(1));
Assert.That(m_completion.QueuedAcks[0].SubscriptionId, Is.EqualTo(3u));
Assert.That(m_completion.QueuedAcks[0].SequenceNumber, Is.EqualTo(4u));
}
}
[TestCase("data")]
[TestCase("event")]
[TestCase("keepalive")]
[TestCase("status")]
public async Task NotificationCallbacksHaveActiveReentryScopeAsync(string kind)
{
var observed = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry);
await using (sut.ConfigureAwait(false))
{
sut.NotificationCallback = _ =>
{
observed.TrySetResult(sut.IsDispatchingForTest);
return default;
};
await sut.OnPublishReceivedAsync(BuildNotificationMessage(kind, 1), null, [])
.ConfigureAwait(false);
Assert.That(await observed.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false), Is.True);
}
}
[TestCase("data")]
[TestCase("event")]
[TestCase("keepalive")]
[TestCase("status")]
public async Task ChildOfCompletedCallbackCanDisposeDuringLaterDispatchAsync(string kind)
{
var releaseChild = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var secondEntered = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var releaseSecond = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var disposalStarted = new TaskCompletionSource<InvalidOperationException?>(
TaskCreationOptions.RunContinuationsAsynchronously);
Task child = Task.CompletedTask;
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry);
await using (sut.ConfigureAwait(false))
{
sut.NotificationCallback = async sequenceNumber =>
{
if (sequenceNumber == 1)
{
child = Task.Run(async () =>
{
await releaseChild.Task.ConfigureAwait(false);
try
{
ValueTask disposing = sut.DisposeAsync();
disposalStarted.TrySetResult(null);
await disposing.ConfigureAwait(false);
}
catch (InvalidOperationException exception)
{
if (!disposalStarted.TrySetResult(exception))
{
throw;
}
}
});
}
else
{
secondEntered.TrySetResult(true);
await releaseSecond.Task.ConfigureAwait(false);
}
};
try
{
await sut.OnPublishReceivedAsync(BuildNotificationMessage(kind, 1), null, [])
.ConfigureAwait(false);
await sut.OnPublishReceivedAsync(BuildNotificationMessage(kind, 2), null, [])
.ConfigureAwait(false);
await secondEntered.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
releaseChild.TrySetResult(true);
Assert.That(
await disposalStarted.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false),
Is.Null,
"The completed callback's child is not re-entering the later callback.");
}
finally
{
releaseChild.TrySetResult(true);
releaseSecond.TrySetResult(true);
await child.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
}
}
}
[Test]
public async Task DeferredDataCallbackTaskDoesNotInheritDispatchGuardAsync()
{
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry)
{
Id = 3,
DeferredCallbackMode = DeferredCallbackMode.Data
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(
new NotificationMessage
{
SequenceNumber = 1,
NotificationData = [new ExtensionObject(new DataChangeNotification())]
},
null,
[]).ConfigureAwait(false);
await sut.DataChangeNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
await m_completion.WaitForQueuedAckAsync(1).ConfigureAwait(false);
sut.CallbackReturned.TrySetResult(true);
Assert.That(sut.DeferredDispatchGuard, Is.Not.Null);
Assert.That(
await sut.DeferredDispatchGuard!.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false),
Is.False);
}
}
[Test]
public async Task DeferredEventCallbackTaskDoesNotInheritDispatchGuardAsync()
{
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry)
{
Id = 3,
DeferredCallbackMode = DeferredCallbackMode.Event
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(
new NotificationMessage
{
SequenceNumber = 1,
NotificationData = [new ExtensionObject(new EventNotificationList())]
},
null,
[]).ConfigureAwait(false);
await sut.EventNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
await m_completion.WaitForQueuedAckAsync(1).ConfigureAwait(false);
sut.CallbackReturned.TrySetResult(true);
Assert.That(sut.DeferredDispatchGuard, Is.Not.Null);
Assert.That(
await sut.DeferredDispatchGuard!.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false),
Is.False);
}
}
[Test]
public async Task DeferredStatusCallbackTaskDoesNotInheritDispatchGuardAsync()
{
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry)
{
Id = 3,
DeferredCallbackMode = DeferredCallbackMode.Status
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(
new NotificationMessage
{
SequenceNumber = 1,
NotificationData =
[
new ExtensionObject(new StatusChangeNotification
{
Status = StatusCodes.BadTimeout
})
]
},
null,
[]).ConfigureAwait(false);
await sut.StatusChangeNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
await m_completion.WaitForQueuedAckAsync(1).ConfigureAwait(false);
sut.CallbackReturned.TrySetResult(true);
Assert.That(sut.DeferredDispatchGuard, Is.Not.Null);
Assert.That(
await sut.DeferredDispatchGuard!.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false),
Is.False);
}
}
[Test]
public async Task PublishingDrainDoesNotWaitForIngressBlockedByServiceWriterAsync()
{
var session = new FakeSubscriptionManagerContext();
var manager = new SubscriptionManager(session, m_telemetry.LoggerFactory, DiagnosticsMasks.None)
{
MinPublishWorkerCount = 1,
MaxPublishWorkerCount = 1
};
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry) { Id = 3 };
using var serviceGate = new SemaphoreSlim(1, 1);
var callbackEntered = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var ingressBlocked = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var drained = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
await serviceGate.WaitAsync().ConfigureAwait(false);
sut.NotificationCallback = async _ =>
{
callbackEntered.TrySetResult(true);
await serviceGate.WaitAsync().ConfigureAwait(false);
serviceGate.Release();
};
var subscription = new FakeManagedSubscription
{
Id = 3,
Created = true,
OnPublishReceivedAsyncFunc = (message, available, strings) =>
{
ValueTask receiving = sut.OnPublishReceivedAsync(message, available, strings);
ingressBlocked.TrySetResult(!receiving.IsCompleted);
return receiving;
}
};
session.CreateSubscriptionFactory = (_, _, _) => subscription;
session.OnPublishAsync = (_, _, _) => new ValueTask<PublishResponse>(new PublishResponse
{
SubscriptionId = 3,
MoreNotifications = true,
NotificationMessage = BuildNotificationMessage("keepalive", 1025)
});
Task quiescence = Task.CompletedTask;
try
{
for (uint sequenceNumber = 1; sequenceNumber <= 1024; sequenceNumber++)
{
await sut.OnPublishReceivedAsync(
BuildNotificationMessage("keepalive", sequenceNumber), null, []).ConfigureAwait(false);
}
await callbackEntered.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
manager.Add(Mock.Of<ISubscriptionNotificationHandler>(), OptionsFactory.Create<SubscriptionOptions>());
manager.Resume();
Assert.That(
await ingressBlocked.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false), Is.True);
quiescence = manager.RunWithPublishingQuiescedAsync(_ =>
{
manager.Pause();
drained.TrySetResult(true);
return default;
}, CancellationToken.None).AsTask();
await drained.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
Assert.That(serviceGate.CurrentCount, Is.Zero, "The service writer still owns the gate.");
Assert.That(session.PublishCalls, Has.Count.EqualTo(1));
}
finally
{
serviceGate.Release();
await quiescence.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
await manager.DisposeAsync().ConfigureAwait(false);
await sut.DisposeAsync().ConfigureAwait(false);
}
}
[Test]
public async Task CapacityWaitCannotRelabelRetiredMessageAsNewGenerationAsync()
{
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry);
var barrier = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
sut.NotificationCallback = sequenceNumber =>
{
if (sequenceNumber == 6000)
{
barrier.TrySetResult(true);
}
return default;
};
await sut.Block.WaitAsync().ConfigureAwait(false);
await using (sut.ConfigureAwait(false))
{
for (uint sequenceNumber = 1; sequenceNumber <= 1024; sequenceNumber++)
{
await sut.OnPublishReceivedAsync(
BuildNotificationMessage("keepalive", sequenceNumber), null, []).ConfigureAwait(false);
}
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
Task pending = sut.OnPublishReceivedAsync(BuildDataChangeMessage(5000), null, []).AsTask();
Assert.That(pending.IsCompleted, Is.False);
Task reset = sut.ResetGenerationAsync().AsTask();
sut.Block.Release();
await reset.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
await pending.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
await sut.OnPublishReceivedAsync(BuildNotificationMessage("keepalive", 6000), null, [])
.ConfigureAwait(false);
await barrier.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers, Does.Not.Contain(5000u));
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(1), null, []).ConfigureAwait(false);
await m_completion.WaitForQueuedAckAsync(1).ConfigureAwait(false);
Assert.That(m_completion.QueuedAcks.Single().SequenceNumber, Is.EqualTo(1u));
}
}
[Test]
public async Task BackpressuredBurstDeliversEveryNotificationAsync()
{
const int kMessageCount = 1100;
var firstEntered = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var releaseFirst = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry)
{
NotificationCallback = async sequenceNumber =>
{
if (sequenceNumber == 1)
{
firstEntered.TrySetResult(true);
await releaseFirst.Task.ConfigureAwait(false);
}
}
};
await using (sut.ConfigureAwait(false))
{
Task producing = ProduceAsync();
try
{
await firstEntered.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
Assert.That(producing.IsCompleted, Is.False);
releaseFirst.TrySetResult(true);
await producing.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
await m_completion.WaitForQueuedAckAsync(kMessageCount).ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(Enumerable.Range(1, kMessageCount).Select(value => (uint)value)));
Assert.That(m_completion.QueuedAcks.Select(acknowledgement => acknowledgement.SequenceNumber),
Is.EqualTo(Enumerable.Range(1, kMessageCount).Select(value => (uint)value)));
}
finally
{
releaseFirst.TrySetResult(true);
await producing.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
}
}
async Task ProduceAsync()
{
for (uint sequenceNumber = 1; sequenceNumber <= kMessageCount; sequenceNumber++)
{
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(sequenceNumber), null, [])
.ConfigureAwait(false);
}
}
}
[Test]
public async Task IngressBackpressureBlocksBeyondBoundedCapacityAsync()
{
var sut = new TestMessageProcessor(m_mockServices.Object, m_completion, m_telemetry)
{
Id = 3
};
await sut.Block.WaitAsync().ConfigureAwait(false);
await using (sut.ConfigureAwait(false))
{
var writes = new List<Task>();
for (uint sequence = 1; sequence <= 1024; sequence++)
{
writes.Add(sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = sequence },
null,
[]).AsTask());
}
await Task.WhenAll(writes).ConfigureAwait(false);
Task blocked = sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = 1025 },
null,
[]).AsTask();
Assert.That(blocked.IsCompleted, Is.False);
sut.Block.Release();
await blocked.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
}
}
[Test]
public async Task ProcessMessageAsyncShouldRepublishMissingMessagesAsync()
{
// Arrange
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 2
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 1
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
m_mockServices
.Setup(c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.Is<uint>(id => id == sut.Id),
It.Is<uint>(s => s == 2),
It.IsAny<CancellationToken>()))
.ReturnsAsync(new RepublishResponse
{
NotificationMessage = new NotificationMessage
{
SequenceNumber = 2,
NotificationData =
[
new ExtensionObject(new DataChangeNotification
{
MonitoredItems =
[
new MonitoredItemNotification()
]
})
]
}
})
.Verifiable(Times.Once);
// Act
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 3,
NotificationData =
[
new ExtensionObject(new EventNotificationList
{
Events =
[
new EventFieldList()
]
})
]
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.EventNotificationReceived.WaitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
// Assert
Assert.That(sut.EventNotificationReceived.IsSet, Is.True);
Assert.That(sut.KeepAliveNotificationReceived.IsSet, Is.True);
Assert.That(sut.DataChangeNotificationReceived.IsSet, Is.True);
m_mockServices.Verify();
}
}
[Test]
public async Task ProcessReceivedMessagesAsyncShouldProcessMessagesInOrderAsync()
{
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
// Arrange
NotificationMessage[] messages = [.. Enumerable.Range(2, 99).Select(i => new NotificationMessage
{
SequenceNumber = (uint)i
})];
Array.Reverse(messages);
var lastNotification = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 3,
NotificationCallback = sequenceNumber =>
{
if (sequenceNumber == 100)
{
lastNotification.TrySetResult(true);
}
return default;
}
};
await using (sut.ConfigureAwait(false))
{
await sut.Block.WaitAsync().ConfigureAwait(false);
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 1u
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
foreach (NotificationMessage message in messages)
{
await sut.OnPublishReceivedAsync(message, availableSequenceNumbers, stringTable).ConfigureAwait(false);
}
sut.Block.Release();
// Act
await lastNotification.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers, Is.EqualTo(
Enumerable.Range(1, 100).Select(i => (uint)i)));
Assert.That(sut.AvailableInRetransmissionQueue, Is.EqualTo(availableSequenceNumbers));
Assert.That(sut.DataChangeNotificationReceived.IsSet, Is.False);
Assert.That(sut.KeepAliveNotificationReceived.IsSet, Is.True);
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(100));
}
}
[Test]
public async Task DuplicateSequenceNumberShouldNotRedispatchAsync()
{
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 7
};
await using (sut.ConfigureAwait(false))
{
var message = new NotificationMessage
{
SequenceNumber = 5,
NotificationData =
[
new ExtensionObject(new DataChangeNotification
{
MonitoredItems =
[
new MonitoredItemNotification()
]
})
]
};
// First arrival
await sut.OnPublishReceivedAsync(message, availableSequenceNumbers, stringTable)
.ConfigureAwait(false);
await sut.DataChangeNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
int firstCount = sut.ReceivedSequenceNumbers.Count;
Assert.That(firstCount, Is.GreaterThanOrEqualTo(1));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(5));
// Reset signaling
sut.DataChangeNotificationReceived.Reset();
// Same sequence number arriving again
await sut.OnPublishReceivedAsync(message, availableSequenceNumbers, stringTable)
.ConfigureAwait(false);
// Give the processor a moment in case it tries to dispatch
await Task.Delay(50).ConfigureAwait(false);
// The duplicate should not re-fire the data-change handler.
Assert.That(
sut.DataChangeNotificationReceived.IsSet, Is.False,
"Duplicate sequence number must not re-dispatch the notification");
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(5));
}
}
[Test]
public async Task KeepAliveInterleavedWithNotificationsAsync()
{
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 5
};
await using (sut.ConfigureAwait(false))
{
// 1: notification
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 1,
NotificationData =
[
new ExtensionObject(new DataChangeNotification
{
MonitoredItems = [new MonitoredItemNotification()]
})
]
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.DataChangeNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
// 2: keep-alive (no NotificationData)
sut.KeepAliveNotificationReceived.Reset();
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 2
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
// 3: another notification
sut.DataChangeNotificationReceived.Reset();
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 3,
NotificationData =
[
new ExtensionObject(new DataChangeNotification
{
MonitoredItems = [new MonitoredItemNotification()]
})
]
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.DataChangeNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
// All three sequence numbers should appear in order.
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 1, 2, 3 }));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(3));
}
}
/// <summary>
/// A keep-alive carries the next sequence number to be sent (Part 4
/// §5.14.1.1), so a keep-alive two ahead of the last data message
/// proves the message in between was lost. It must be republished
/// right away instead of waiting for the next data message.
/// </summary>
[Test]
public async Task KeepAliveAfterGapRepublishesMissingMessageAsync()
{
SetupRepublish();
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 28
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(5),
[5], []).ConfigureAwait(false);
await WaitForLastSeqNumberAsync(sut, 5).ConfigureAwait(false);
// Data message 6 is lost; the keep-alive names 7 as next.
sut.KeepAliveNotificationReceived.Reset();
await sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = 7 },
[5, 6], []).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 5, 6, 7 }));
Assert.That(sut.MissingMessageCount, Is.EqualTo(1));
Assert.That(sut.RepublishMessageCount, Is.EqualTo(1));
Assert.That(sut.LastDataSequenceNumberProcessed, Is.EqualTo(6));
// A second keep-alive with the same number does not republish
// again, and the data message that reuses 7 is not a gap.
await sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = 7 },
[6], []).ConfigureAwait(false);
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(7),
[6, 7], []).ConfigureAwait(false);
await WaitForDataSeqNumberAsync(sut, 7).ConfigureAwait(false);
Assert.That(sut.MissingMessageCount, Is.EqualTo(1));
Assert.That(sut.RepublishMessageCount, Is.EqualTo(1));
m_mockServices.Verify(
c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.Is<uint>(s => s == 6),
It.IsAny<CancellationToken>()),
Times.Once);
}
}
[Test]
public async Task EmptyNotificationDataIsTreatedAsKeepAliveAsync()
{
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 9
};
await using (sut.ConfigureAwait(false))
{
// NotificationData is null/default.
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 7
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
Assert.That(sut.KeepAliveNotificationReceived.IsSet, Is.True);
Assert.That(sut.DataChangeNotificationReceived.IsSet, Is.False);
Assert.That(sut.EventNotificationReceived.IsSet, Is.False);
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(7));
}
}
[Test]
public async Task RepublishFailureLogsButContinuesProcessingAsync()
{
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 11
};
await using (sut.ConfigureAwait(false))
{
// First message at sequence 1.
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 1
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
// Republish for the gap (sequence 2) returns an error.
m_mockServices
.Setup(c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.Is<uint>(id => id == sut.Id),
It.Is<uint>(s => s == 2),
It.IsAny<CancellationToken>()))
.ThrowsAsync(new ServiceResultException(
StatusCodes.BadMessageNotAvailable))
.Verifiable(Times.AtLeastOnce);
// Skip to sequence 3 to force the gap.
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 3,
NotificationData =
[
new ExtensionObject(new EventNotificationList
{
Events = [new EventFieldList()]
})
]
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.EventNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
// Even though republish failed, the next message must still
// be dispatched.
Assert.That(sut.EventNotificationReceived.IsSet, Is.True);
m_mockServices.Verify();
}
}
[Test]
public async Task SequenceNumberWraparoundAdvancesLastProcessedAsync()
{
// Per OPC UA Part 4 §7.30.5, sequence numbers wrap from
// uint.MaxValue to 1 (skipping 0). The processor must accept
// this wrap as forward progress, not silently drop the
// post-wrap message as "old".
var availableSequenceNumbers = new List<uint>();
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 13
};
await using (sut.ConfigureAwait(false))
{
// Republish for the first message's "missing" gap (we start
// from LastSequenceNumberProcessed=0 so no gap is computed
// for this first one) — but for the second message, the gap
// is empty because we wrap directly from uint.MaxValue to 1.
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = uint.MaxValue
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
Assert.That(sut.LastSequenceNumberProcessed,
Is.EqualTo(uint.MaxValue));
sut.KeepAliveNotificationReceived.Reset();
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 1
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
Assert.That(sut.KeepAliveNotificationReceived.IsSet, Is.True,
"Wrapped sequence number must be accepted as forward progress, " +
"not dropped as duplicate/old.");
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(1));
}
}
[Test]
public async Task ReceivingTransferStatusUpdateShouldUpdatePublishStateAsync()
{
// Arrange
var availableSequenceNumbers = new List<uint> { 1, 2, 3 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 3
};
await using (sut.ConfigureAwait(false))
{
// Act
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 3,
NotificationData =
[
new ExtensionObject(new StatusChangeNotification
{
Status = StatusCodes.GoodSubscriptionTransferred
})
]
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.StatusChangeNotificationReceived.WaitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
// Assert
Assert.That(sut.StatusChangeNotificationReceived.IsSet, Is.True);
Assert.That(sut.ReceivedSequenceNumbers, Does.Contain(3));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(3));
Assert.That(sut.PublishState, Is.EqualTo(PublishState.Transferred));
sut.StatusChangeNotificationReceived.Reset();
// Act
await sut.OnPublishReceivedAsync(new NotificationMessage
{
SequenceNumber = 4,
NotificationData =
[
new ExtensionObject(new StatusChangeNotification
{
Status = StatusCodes.BadTimeout
})
]
}, availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.StatusChangeNotificationReceived.WaitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
// Assert
Assert.That(sut.StatusChangeNotificationReceived.IsSet, Is.True);
Assert.That(sut.ReceivedSequenceNumbers, Does.Contain(4));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(4));
Assert.That(sut.PublishState, Is.EqualTo(PublishState.Timeout));
}
}
/// <summary>
/// V2 equivalent of the classic-engine regression captured by
/// <c>OldMessagesAbandonedAfterRepublishTimeoutInSequentialModeAsync</c>
/// (currently <c>[Explicit]</c> in <c>Classic/SubscriptionUnitTests.cs</c>).
/// </summary>
/// <remarks>
/// Mirrors the classic test's exact arrival order — <c>[3]</c>,
/// <c>[2]</c>, <c>[4]</c>, <c>[1]</c>, <c>[5]</c> — but queues all five
/// while the worker is parked inside a sentinel keep-alive dispatch.
/// The new <see cref="MessageProcessor"/> uses
/// <see cref="System.Threading.Channels.Channel.CreateUnboundedPrioritized"/>
/// keyed by <see cref="NotificationMessage.SequenceNumber"/>, so any
/// items present in the channel at a single read are sorted by
/// sequence number. With the consumer held, the five data messages
/// land together and drain as <c>[1, 2, 3, 4, 5]</c> with no
/// republishes — the reliability guarantee the classic engine fails
/// to provide. (Without the sentinel, a fast consumer could read the
/// first arrival before the rest queue, which is what the priority
/// channel cannot recover from on its own.)
/// </remarks>
[Test]
public async Task OutOfOrderArrivalsReorderedByPriorityChannelAsync()
{
var availableSequenceNumbers = new List<uint> { 1, 2, 3, 4, 5 };
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 11
};
await using (sut.ConfigureAwait(false))
{
// Arrange: drain the gate, then park the worker inside a
// sentinel keep-alive dispatch so all five data messages
// queue up in the priority channel before any are read.
// The keep-alive does not advance the data dedup gate
// (LastDataSequenceNumberProcessed) so the burst is still
// processed as a fresh sequence starting at 1.
sut.Block.Wait();
await sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = 1u },
availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
foreach (uint seq in new uint[] { 3, 2, 4, 1, 5 })
{
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(seq),
availableSequenceNumbers, stringTable).ConfigureAwait(false);
}
sut.Block.Release();
// Act
await WaitForLastSeqNumberAsync(sut, 5).ConfigureAwait(false);
// Assert: the first entry is the sentinel keep-alive; the
// remaining entries must be the data burst in strict
// sequence-number order.
Assert.That(sut.ReceivedSequenceNumbers,
Has.Count.EqualTo(6));
Assert.That(sut.ReceivedSequenceNumbers[0], Is.EqualTo(1u),
"first entry should be the sentinel keep-alive");
Assert.That(sut.ReceivedSequenceNumbers.Skip(1),
Is.EqualTo(new uint[] { 1, 2, 3, 4, 5 }));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(5));
Assert.That(sut.MissingMessageCount, Is.Zero);
Assert.That(sut.RepublishMessageCount, Is.Zero);
m_mockServices.Verify(
c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.IsAny<uint>(),
It.IsAny<CancellationToken>()),
Times.Never);
}
}
/// <summary>
/// V2 equivalent of the classic-engine regression captured by
/// <c>OldMessagesAbandonedAfterRepublishTimeoutInSequentialModeAsync</c>,
/// stressed in serial timing instead of as a burst.
/// </summary>
/// <remarks>
/// Same arrival order — <c>[3]</c>, <c>[2]</c>, <c>[4]</c>, <c>[1]</c>,
/// <c>[5]</c> — but each forward-progressing push waits for delivery
/// before the next is queued. The V2 engine's monotonic dedup gate
/// (<c>LastDataSequenceNumberProcessed</c>) silently discards the
/// stale <c>[2]</c> and <c>[1]</c> instead of re-inserting them
/// out of order; <c>[3, 4, 5]</c> are delivered in strict ascending
/// order with no republishes.
/// </remarks>
[Test]
public async Task LateArrivalsDiscardedAfterDedupGateAdvancedAsync()
{
var availableSequenceNumbers = new List<uint>();
var stringTable = new List<string> { "test" };
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 12
};
await using (sut.ConfigureAwait(false))
{
// [3]: first data message after create. The empty
// AvailableInRetransmissionQueue suppresses the
// first-after-create speculative republish path, so [3]
// goes straight to the handler.
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(3),
availableSequenceNumbers, stringTable).ConfigureAwait(false);
await WaitForLastSeqNumberAsync(sut, 3).ConfigureAwait(false);
// [2] is strictly behind LastDataSeq=3. Queue [2] then [4]
// back-to-back; the priority channel picks [2] first
// (lowest SequenceNumber) so the worker reads [2], discards
// it silently, then reads [4] and delivers it. Waiting for
// LastSeq == 4 confirms both happened in that order.
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(2),
availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(4),
availableSequenceNumbers, stringTable).ConfigureAwait(false);
await WaitForLastSeqNumberAsync(sut, 4).ConfigureAwait(false);
// Same shape for [1] (even staler) followed by [5].
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(1),
availableSequenceNumbers, stringTable).ConfigureAwait(false);
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(5),
availableSequenceNumbers, stringTable).ConfigureAwait(false);
await WaitForLastSeqNumberAsync(sut, 5).ConfigureAwait(false);
// Assert: only forward-progressing messages were delivered,
// in strict order; the late [2] and [1] were discarded.
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 3, 4, 5 }));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(5));
Assert.That(sut.MissingMessageCount, Is.Zero);
Assert.That(sut.RepublishMessageCount, Is.Zero);
m_mockServices.Verify(
c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.IsAny<uint>(),
It.IsAny<CancellationToken>()),
Times.Never);
}
}
/// <summary>
/// A transferred subscription that stays quiet must still recover the
/// notifications the server kept in its retransmission queue: the
/// gap-walk republish only runs when a new data message arrives, so
/// the transfer itself has to drive the recovery.
/// </summary>
[Test]
public async Task RecoverTransferredMessagesRepublishesWithoutNewNotificationsAsync()
{
SetupRepublish();
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 21
};
await using (sut.ConfigureAwait(false))
{
// Act: transfer reports three messages still in the server
// retransmission queue, reported out of order.
await sut.RecoverTransferredMessagesAsync([4, 2, 3], default)
.ConfigureAwait(false);
// Assert: all three are recovered, in ascending order, and
// flagged as republished notifications.
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 2, 3, 4 }));
Assert.That(sut.PublishState, Is.EqualTo(PublishState.Republish));
Assert.That(sut.RepublishMessageCount, Is.EqualTo(3));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(4));
Assert.That(sut.AvailableInRetransmissionQueue, Is.Empty,
"transfer recovery must not publish its local snapshot");
await m_completion.WaitForQueuedAckAsync(3).ConfigureAwait(false);
// The dedup gate advanced past the recovered messages, so the
// next publish is not mistaken for a gap and nothing is
// republished a second time.
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(5),
[4, 2, 3], []).ConfigureAwait(false);
await WaitForLastSeqNumberAsync(sut, 5).ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 2, 3, 4, 5 }));
Assert.That(sut.MissingMessageCount, Is.Zero);
Assert.That(sut.RepublishMessageCount, Is.EqualTo(3));
}
}
[Test]
public async Task RecoverTransferredMessagesPreservesNewerAvailableSetAsync()
{
int republishCalls = 0;
var firstRepublishStarted = new TaskCompletionSource<bool>(
TaskCreationOptions.RunContinuationsAsynchronously);
var releaseFirstRepublish = new TaskCompletionSource<bool>(
TaskCreationOptions.RunContinuationsAsynchronously);
m_mockServices
.Setup(c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.IsAny<uint>(),
It.IsAny<CancellationToken>()))
.Returns(async (RequestHeader _, uint _, uint sequenceNumber,
CancellationToken _) =>
{
if (Interlocked.Increment(ref republishCalls) == 1)
{
firstRepublishStarted.SetResult(true);
await releaseFirstRepublish.Task
.WaitAsync(TimeSpan.FromSeconds(5), CancellationToken.None)
.ConfigureAwait(false);
}
return new RepublishResponse
{
ResponseHeader = new ResponseHeader(),
NotificationMessage = BuildDataChangeMessage(sequenceNumber)
};
});
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 25
};
await using (sut.ConfigureAwait(false))
{
Task recoverTask = sut.RecoverTransferredMessagesAsync([7, 8], default)
.AsTask();
await firstRepublishStarted.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
// The keep-alive names 9, the message right after the
// recovered ones, so processing it after the recovery does
// not start a keep-alive gap recovery that races the
// republish count asserted below.
await sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = 9 },
[11, 12],
[]).ConfigureAwait(false);
Assert.That(sut.AvailableInRetransmissionQueue,
Is.EqualTo(new uint[] { 11, 12 }));
releaseFirstRepublish.SetResult(true);
await recoverTask.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
Assert.That(sut.AvailableInRetransmissionQueue,
Is.EqualTo(new uint[] { 11, 12 }));
Assert.That(sut.RepublishMessageCount, Is.EqualTo(2));
}
}
[Test]
public async Task RecoverTransferredMessagesWithoutAvailableSequenceNumbersDoesNothingAsync()
{
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 22
};
await using (sut.ConfigureAwait(false))
{
await sut.RecoverTransferredMessagesAsync([], default)
.ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers, Is.Empty);
Assert.That(sut.RepublishMessageCount, Is.Zero);
Assert.That(sut.LastSequenceNumberProcessed, Is.Zero);
m_mockServices.Verify(
c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.IsAny<uint>(),
It.IsAny<CancellationToken>()),
Times.Never);
}
}
/// <summary>
/// Sequence numbers wrap from <see cref="uint.MaxValue"/> to 1
/// (Part 4 §7.30.5), so a retransmission queue spanning the wrap must
/// still be recovered oldest-first.
/// </summary>
[Test]
public async Task RecoverTransferredMessagesOrdersAcrossSequenceNumberWrapAsync()
{
SetupRepublish();
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 23
};
await using (sut.ConfigureAwait(false))
{
await sut.RecoverTransferredMessagesAsync(
[3, uint.MaxValue - 5, 5], default)
.ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[]
{
uint.MaxValue - 5, 3, 5
}));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(5));
}
}
/// <summary>
/// A reconnect that transfers the subscription onto the same object
/// keeps the dedup gate. Messages that were already dispatched but
/// not yet acknowledged are still listed as available by the server
/// (Part 4 §5.14.7.2); they must be acknowledged, not republished and
/// dispatched a second time.
/// </summary>
[Test]
public async Task RecoverTransferredMessagesSkipsAlreadyProcessedMessagesAsync()
{
SetupRepublish();
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 27
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(9),
[], []).ConfigureAwait(false);
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(10),
[], []).ConfigureAwait(false);
await WaitForLastSeqNumberAsync(sut, 10).ConfigureAwait(false);
await m_completion.WaitForQueuedAckAsync(2).ConfigureAwait(false);
m_completion.QueuedAcks.Clear();
await sut.RecoverTransferredMessagesAsync([9, 10, 11], default)
.ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 9, 10, 11 }));
Assert.That(sut.RepublishMessageCount, Is.EqualTo(1));
m_mockServices.Verify(
c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.Is<uint>(s => s == 9 || s == 10),
It.IsAny<CancellationToken>()),
Times.Never);
await m_completion.WaitForQueuedAckAsync(3).ConfigureAwait(false);
Assert.That(
m_completion.QueuedAcks.Select(a => a.SequenceNumber),
Is.EquivalentTo(new uint[] { 9, 10, 11 }));
Assert.That(sut.LastDataSequenceNumberProcessed, Is.EqualTo(11));
}
}
[Test]
public async Task RecoverTransferredMessagesAdvancesGateWhenRepublishFailsAsync()
{
m_mockServices
.Setup(c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.IsAny<uint>(),
It.IsAny<CancellationToken>()))
.ThrowsAsync(new ServiceResultException(
StatusCodes.BadMessageNotAvailable));
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 24
};
await using (sut.ConfigureAwait(false))
{
// A server that dropped the messages must not fail the
// recovery - the gate still advances past the sequence
// numbers the server reported as sent.
await sut.RecoverTransferredMessagesAsync([7, 8], default)
.ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers, Is.Empty);
Assert.That(sut.RepublishMessageCount, Is.EqualTo(2));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(8));
// The next message is therefore delivered without a gap walk.
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(9),
[], []).ConfigureAwait(false);
await WaitForLastSeqNumberAsync(sut, 9).ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 9 }));
Assert.That(sut.MissingMessageCount, Is.Zero);
}
}
/// <summary>
/// A gap republish that failed leaves the message undelivered although
/// the data gate moved past it. Transfer recovery must republish it
/// instead of acknowledging it as already dispatched, while messages
/// really dispatched are still only acknowledged.
/// </summary>
[Test]
public async Task RecoverTransferredMessagesRepublishesMessageWhoseGapRepublishFailedAsync()
{
SetupRepublishFailingFirst(11, 1);
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 29
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(10),
[], []).ConfigureAwait(false);
// Data 11 is lost and its republish fails.
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(12),
[11, 12], []).ConfigureAwait(false);
await WaitForReceivedCountAsync(sut, 2).ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 10, 12 }));
await sut.RecoverTransferredMessagesAsync([11, 12], default)
.ConfigureAwait(false);
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 10, 12, 11 }));
m_mockServices.Verify(
c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.Is<uint>(s => s == 11),
It.IsAny<CancellationToken>()),
Times.Exactly(2));
m_mockServices.Verify(
c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.Is<uint>(s => s == 12),
It.IsAny<CancellationToken>()),
Times.Never);
Assert.That(sut.LastDataSequenceNumberProcessed, Is.EqualTo(12));
}
}
/// <summary>
/// A keep-alive recovery whose republish fails must not advance the
/// data gate over the unrecovered message, so the gap-walk of the
/// next data message retries it.
/// </summary>
[Test]
public async Task KeepAliveRecoveryFailureLeavesGapForNextDataMessageAsync()
{
SetupRepublishFailingFirst(11, 1);
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 30
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(10),
[], []).ConfigureAwait(false);
await WaitForDataSeqNumberAsync(sut, 10).ConfigureAwait(false);
sut.KeepAliveNotificationReceived.Reset();
await sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = 12 },
[11], []).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
Assert.That(sut.LastDataSequenceNumberProcessed, Is.EqualTo(10));
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(12),
[11, 12], []).ConfigureAwait(false);
await WaitForReceivedCountAsync(sut, 4).ConfigureAwait(false);
// 10, the keep-alive 12, then 11 recovered by the gap-walk
// and data 12.
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 10, 12, 11, 12 }));
}
}
/// <summary>
/// A keep-alive that proves a gap nothing can be recovered for must
/// not advance the data gate: the missing message may still be in
/// flight and must be delivered when it arrives.
/// </summary>
[Test]
public async Task KeepAliveWithoutRecoverableMessagesKeepsDataGateAsync()
{
SetupRepublish();
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 31
};
await using (sut.ConfigureAwait(false))
{
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(10),
[], []).ConfigureAwait(false);
await WaitForDataSeqNumberAsync(sut, 10).ConfigureAwait(false);
// The keep-alive overtakes data 11, which the server does
// not list (no retransmission queue).
sut.KeepAliveNotificationReceived.Reset();
await sut.OnPublishReceivedAsync(
new NotificationMessage { SequenceNumber = 12 },
[], []).ConfigureAwait(false);
await sut.KeepAliveNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(11),
[], []).ConfigureAwait(false);
// Data 13 follows, so a dropped 11 shows in the content below
// instead of as a timeout.
await sut.OnPublishReceivedAsync(BuildDataChangeMessage(13),
[], []).ConfigureAwait(false);
DateTimeOffset deadline = DateTimeOffset.UtcNow.AddSeconds(5);
while (!sut.ReceivedSequenceNumbers.ToArray().Contains(13u) &&
DateTimeOffset.UtcNow < deadline)
{
await Task.Delay(10).ConfigureAwait(false);
}
// 10, the keep-alive 12, then the late original 11, then 13.
Assert.That(sut.ReceivedSequenceNumbers,
Is.EqualTo(new uint[] { 10, 12, 11, 13 }));
Assert.That(sut.RepublishMessageCount, Is.Zero);
}
}
[Test]
public async Task RecoverTransferredMessagesPropagatesCancellationDuringRepublishCallbackAsync()
{
SetupRepublish();
using var cts = new CancellationTokenSource();
var sut = new TestMessageProcessor(m_mockServices.Object,
m_completion, m_telemetry)
{
Id = 26
};
await using (sut.ConfigureAwait(false))
{
await sut.Block.WaitAsync().ConfigureAwait(false);
bool blockHeld = true;
try
{
Task recoverTask = sut.RecoverTransferredMessagesAsync([10], cts.Token)
.AsTask();
await sut.DataChangeNotificationReceived.WaitAsync()
.WaitAsync(TimeSpan.FromSeconds(5))
.ConfigureAwait(false);
cts.Cancel();
sut.Block.Release();
blockHeld = false;
Assert.ThrowsAsync<OperationCanceledException>(
async () => await recoverTask.ConfigureAwait(false));
Assert.That(sut.LastSequenceNumberProcessed, Is.EqualTo(10));
Assert.That(sut.RepublishMessageCount, Is.EqualTo(1));
}
finally
{
if (blockHeld)
{
sut.Block.Release();
}
}
}
}
private void SetupRepublishFailingFirst(uint failingSequenceNumber, int failures)
{
int remaining = failures;
m_mockServices
.Setup(c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.IsAny<uint>(),
It.IsAny<CancellationToken>()))
.Returns((RequestHeader _, uint _, uint sequenceNumber, CancellationToken _) =>
{
if (sequenceNumber == failingSequenceNumber &&
Interlocked.Decrement(ref remaining) >= 0)
{
throw new ServiceResultException(StatusCodes.BadSecureChannelClosed);
}
return new ValueTask<RepublishResponse>(new RepublishResponse
{
ResponseHeader = new ResponseHeader(),
NotificationMessage = BuildDataChangeMessage(sequenceNumber)
});
});
}
private void SetupRepublish()
{
m_mockServices
.Setup(c => c.RepublishAsync(
It.IsAny<RequestHeader>(),
It.IsAny<uint>(),
It.IsAny<uint>(),
It.IsAny<CancellationToken>()))
.ReturnsAsync((RequestHeader _, uint _, uint sequenceNumber,
CancellationToken _) => new RepublishResponse
{
ResponseHeader = new ResponseHeader(),
NotificationMessage = BuildDataChangeMessage(sequenceNumber)
});
}
private static NotificationMessage BuildNotificationMessage(string kind, uint sequenceNumber)
{
return new NotificationMessage
{
SequenceNumber = sequenceNumber,
NotificationData = kind switch
{
"data" => [new ExtensionObject(new DataChangeNotification())],
"event" => [new ExtensionObject(new EventNotificationList())],
"status" => [new ExtensionObject(new StatusChangeNotification { Status = StatusCodes.Good })],
"keepalive" => [],
_ => throw new ArgumentOutOfRangeException(nameof(kind))
}
};
}
private static NotificationMessage BuildDataChangeMessage(uint sequenceNumber)
{
return new NotificationMessage
{
SequenceNumber = sequenceNumber,
NotificationData =
[
new ExtensionObject(new DataChangeNotification
{
MonitoredItems =
[
new MonitoredItemNotification()
]
})
]
};
}
private static async Task WaitForLastSeqNumberAsync(
TestMessageProcessor sut,
uint expected,
int timeoutSeconds = 5)
{
DateTimeOffset deadline = DateTimeOffset.UtcNow.AddSeconds(timeoutSeconds);
while (sut.LastSequenceNumberProcessed < expected)
{
if (DateTimeOffset.UtcNow > deadline)
{
Assert.Fail(
$"Timeout: LastSequenceNumberProcessed = {sut.LastSequenceNumberProcessed}, " +
$"expected = {expected}, received = " +
$"[{string.Join(",", sut.ReceivedSequenceNumbers)}]");
}
await Task.Delay(10).ConfigureAwait(false);
}
}
private static async Task WaitForReceivedCountAsync(
TestMessageProcessor sut,
int expected,
int timeoutSeconds = 5)
{
DateTimeOffset deadline = DateTimeOffset.UtcNow.AddSeconds(timeoutSeconds);
while (sut.ReceivedSequenceNumbers.Count < expected)
{
if (DateTimeOffset.UtcNow > deadline)
{
Assert.Fail(
$"Timeout: received {sut.ReceivedSequenceNumbers.Count}, " +
$"expected = {expected}");
}
await Task.Delay(10).ConfigureAwait(false);
}
}
private static async Task WaitForDataSeqNumberAsync(
TestMessageProcessor sut,
uint expected,
int timeoutSeconds = 5)
{
DateTimeOffset deadline = DateTimeOffset.UtcNow.AddSeconds(timeoutSeconds);
while (sut.LastDataSequenceNumberProcessed != expected)
{
if (DateTimeOffset.UtcNow > deadline)
{
Assert.Fail(
$"Timeout: LastDataSequenceNumberProcessed = " +
$"{sut.LastDataSequenceNumberProcessed}, expected = {expected}");
}
await Task.Delay(10).ConfigureAwait(false);
}
}
private sealed class TestMessageProcessor : MessageProcessor
{
public TestMessageProcessor(ISubscriptionServiceSetClientMethods session,
IMessageAckQueue completion, ITelemetryContext telemetry)
: base(session, completion, telemetry)
{
}
public new IReadOnlyList<uint> AvailableInRetransmissionQueue
{
get => base.AvailableInRetransmissionQueue;
set => base.AvailableInRetransmissionQueue = value;
}
public SemaphoreSlim Block { get; } = new(1, 1);
public AsyncManualResetEvent DataChangeNotificationReceived { get; } = new();
public AsyncManualResetEvent EventNotificationReceived { get; } = new();
public AsyncManualResetEvent KeepAliveNotificationReceived { get; } = new();
public new uint LastSequenceNumberProcessed
{
get => base.LastSequenceNumberProcessed;
set => base.LastSequenceNumberProcessed = value;
}
public PublishState PublishState { get; set; }
public List<uint> ReceivedSequenceNumbers { get; } = [];
public AsyncManualResetEvent StatusChangeNotificationReceived { get; } = new();
public DeferredCallbackMode DeferredCallbackMode { get; init; }
public TaskCompletionSource<bool> CallbackReturned { get; } = new(
TaskCreationOptions.RunContinuationsAsynchronously);
public Task<bool>? DeferredDispatchGuard { get; private set; }
public bool IsDispatchingForTest => IsDispatchingNotification;
public Func<uint, ValueTask>? NotificationCallback { get; set; }
public new ValueTask RecoverTransferredMessagesAsync(
IReadOnlyList<uint> availableSequenceNumbers, CancellationToken ct)
{
return base.RecoverTransferredMessagesAsync(
availableSequenceNumbers, ct);
}
public async ValueTask WaitAsync()
{
await Block.WaitAsync().ConfigureAwait(false);
Block.Release();
}
public ValueTask ResetGenerationAsync()
{
return ResetMessageGenerationAsync(_ => default, _ => default, CancellationToken.None);
}
protected override async ValueTask DisposeAsync(bool disposing)
{
CallbackReturned.TrySetResult(true);
try
{
await base.DisposeAsync(disposing).ConfigureAwait(false);
}
finally
{
if (DeferredDispatchGuard != null)
{
await DeferredDispatchGuard.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
}
}
}
protected override ValueTask OnDataChangeNotificationAsync(uint sequenceNumber,
DateTime publishTime, DataChangeNotification notification,
PublishState publishStateMask, IReadOnlyList<string> stringTable)
{
ReceivedSequenceNumbers.Add(sequenceNumber);
if (publishStateMask != PublishState.None)
{
PublishState = publishStateMask;
}
DataChangeNotificationReceived.Set();
if (DeferredCallbackMode == DeferredCallbackMode.Data)
{
StartDeferredProbe();
}
return InvokeNotificationAsync(sequenceNumber);
}
protected override ValueTask OnEventDataNotificationAsync(uint sequenceNumber,
DateTime publishTime, EventNotificationList notification,
PublishState publishStateMask, IReadOnlyList<string> stringTable)
{
ReceivedSequenceNumbers.Add(sequenceNumber);
if (publishStateMask != PublishState.None)
{
PublishState = publishStateMask;
}
EventNotificationReceived.Set();
if (DeferredCallbackMode == DeferredCallbackMode.Event)
{
StartDeferredProbe();
}
return InvokeNotificationAsync(sequenceNumber);
}
protected override ValueTask OnKeepAliveNotificationAsync(uint sequenceNumber,
DateTime publishTime, PublishState publishStateMask)
{
ReceivedSequenceNumbers.Add(sequenceNumber);
if (publishStateMask != PublishState.None)
{
PublishState = publishStateMask;
}
KeepAliveNotificationReceived.Set();
return InvokeNotificationAsync(sequenceNumber);
}
protected override void OnPublishStateChanged(PublishState stateMask)
{
PublishState = stateMask;
base.OnPublishStateChanged(stateMask);
}
protected override async ValueTask OnStatusChangeNotificationAsync(uint sequenceNumber,
DateTime publishTime, StatusChangeNotification notification,
PublishState publishStateMask, IReadOnlyList<string> stringTable)
{
ReceivedSequenceNumbers.Add(sequenceNumber);
if (publishStateMask != PublishState.None)
{
PublishState = publishStateMask;
}
StatusChangeNotificationReceived.Set();
if (DeferredCallbackMode == DeferredCallbackMode.Status)
{
StartDeferredProbe();
}
await InvokeNotificationAsync(sequenceNumber).ConfigureAwait(false);
}
private void StartDeferredProbe()
{
DeferredDispatchGuard = Task.Run(async () =>
{
await CallbackReturned.Task.ConfigureAwait(false);
return IsDispatchingForTest;
});
}
private async ValueTask InvokeNotificationAsync(uint sequenceNumber)
{
if (NotificationCallback != null)
{
await NotificationCallback(sequenceNumber).ConfigureAwait(false);
}
await WaitAsync().ConfigureAwait(false);
}
}
private enum DeferredCallbackMode
{
None,
Data,
Event,
Status
}
private FakeMessageAckQueue m_completion;
private ITelemetryContext m_telemetry;
private Mock<ISubscriptionServiceSetClientMethods> m_mockServices;
}
}