Files
a451053758 Fix the recurring and permanent CI failures on master (#4446)
# Description

Almost every `master` build was reporting yellow or red, which meant the
pipeline signal had stopped being useful: a genuinely broken build
looked the same as a healthy one. I triaged every `master` build from
the last week across Azure DevOps (definition 14, 38 builds) and GitHub
Actions, clustered the errors by root cause, and fixed the nine that
were still live.

The headline finding is that the two loudest problems were not test bugs
at all. Codecov rejects the upload on **28 out of 28 builds**, and that
single warning is what turned every master build `partiallySucceeded`.
Separately, the weekly HA long-haul soak was mathematically incapable of
passing: the configured duration is per *scenario*, and the two
`[NonParallelizable]` soaks run back to back, so the floor is 2 x 60
minutes plus teardown against a 110-minute step budget.

A tenth cluster (the `Opc.Ua.Sessions.Tests` reverse-connect flake, 6
hits around Aug 31) was already fixed by #4366 and has not recurred, so
it needed no work.

## Clusters and root causes

| Cluster | Frequency | Root cause |
|---|---|---|
| codecov upload exits 1 | **28/28 builds** | Codecov returns
`Repository not found`; the resulting warning made every master build
yellow |
| `No space left on device` in coverage merge | nightly | Hosted Ubuntu
agent disk exhausted by the per-matrix coverage artifacts |
| Sample HA Long-Haul timeout | weekly | 2 soaks x 60 min against a 110
min budget, so it could never pass |
| `Artifact coverage-report already exists` | on stage re-run | Pipeline
artifact names must be unique per build |
| `NodeCache*` -> `address already in use` | macOS | `ServerFixture`
retried only `BadNoCommunication`; Kestrel https throws
`IOException`/`AddressInUseException` and escaped the retry, failing the
whole `OneTimeSetUp` |
| `MetricsAreEmittedForChannelLifetimeAsync` | macOS | Entry teardown
emits `close` *before* the `active(-1)` up/down counter, but the test
only waited for `close` |
| `OnPublishReceivedKeepAliveShouldDispatchKeepAliveAsync` | net472 |
The processor queues the ack *after* dispatching the notification the
test waits on |
| `ACarriedPartIsAuthoredAtTheGripperItRidesInAsync` | macOS | A single
starved sample (0.2125 m) tripped a 0.20 m absolute worst-case bound |
| `RobotIntentLiveChannelTests` fixture timeout | net472 | 30 s startup
budget too tight, and the failure text interpolated `lastException`
*before* the polling that assigns it, so it always read `Last error: .`
|

## Approach

**Pipeline.** Coverage reporting is best-effort by design, so the
codecov step now swallows the uploader's exit code and reports the
failure as plain log output rather than a warning that colours the run.
The coverage merge job reclaims the unused Android/GHC/Swift/CodeQL
toolchains before anything writes to the agent, and qualifies its
artifact name on a retry only, so the normal first attempt still
publishes `coverage-report`. The long-haul budgets were raised above the
real floor in both the workflow and the ADO pipeline, and the display
names now say `min/scenario` so the next reader does not repeat the same
miscalculation.

**Tests.** Each test fix targets the specific ordering or bound that was
wrong, rather than adding sleeps. The two metric/ack races now wait on
the measurement the assertions actually depend on. The OpenUSD test
gates the 95th percentile instead of the absolute worst case: the defect
it guards is systematic and moves *every* sample 0.25-0.49 m while the
robot drives, so it still fails that gate loudly, while one straddled
sample no longer can.

## Worth a careful look

- **The port-collision fix is the one real behavioural change.**
`ServerFixtureUtils.IsPortUnavailable` now classifies every transport's
form of the collision, including matching Kestrel's
`AddressInUseException` by type name so the test framework needs no
reference to the ASP.NET Core connection abstractions. It deliberately
does *not* match unrelated failures, which is covered by a negative
test, so a certificate error still surfaces immediately instead of being
retried 25 times on different ports.
- **I dropped an integration test I had written for that fix.** While
validating it I found the UA-TCP listener tolerates a partial bind on
Windows, so `StartAsync` succeeds on the same port and the test asserted
something untrue. I replaced it with deterministic unit tests of the
classifier instead of shipping a test that would itself become flaky.
- **The codecov fix is a mitigation, not a cure.** The `Repository not
found` response is account-side: the `CODECOV_TOKEN` is stale or the
repository is not activated. This change stops it from colouring the
build and logs the failure plainly, but uploads will not actually land
until someone refreshes the token. Flagging that explicitly so it is not
mistaken for a full fix.

## Related Issues

N/A - this addresses CI failures observed directly on `master` builds
rather than a filed issue.

## Checklist

_Put an `x` in the boxes that apply. You can complete these step by step
after opening the PR._

- [x] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [x] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [x] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [x] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [ ] I have addressed **all** PR feedback received.

### Notes on the checklist

- Documentation: no doc updates were needed. The changes are CI
configuration and test-harness fixes with no public API surface, and the
rationale is captured in comments at each changed site.
- Local verification: all touched projects build clean on **net10.0**
and **net48**, and the targeted tests pass (Server 5/5, Client 46/46,
Features 31/31, Robotics.Intent 20/20, plus the Di and Core.Tests
cases). One pre-existing local `net48` PKI-store failure in
`Opc.Ua.Robotics.Intent.Tests` reproduces on unmodified `master` and is
unrelated to this change; CI's `net472` leg runs that project fine.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Marc Schier <marcschier@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: marcschier <11168470+marcschier@users.noreply.github.com>
2026-09-09 09:16:35 +00:00

523 lines
23 KiB
C#

/* ========================================================================
* Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved.
*
* OPC Foundation MIT License 1.00
*
* Permission is hereby granted, free of charge, to any person
* obtaining a copy of this software and associated documentation
* files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following
* conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*
* The complete license agreement can be found here:
* http://opcfoundation.org/License/MIT/1.00/
* ======================================================================*/
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Opc.Ua.Security.Certificates;
namespace Opc.Ua.Server.TestFramework
{
/// <summary>
/// Extensions for server fixture.
/// </summary>
public static class ServerFixtureUtils
{
public const double DefaultSessionTimeout = 120000;
public const uint DefaultMaxResponseMessageSize = 128 * 1024;
public const int MinTestPort = 50000;
public const int MaxTestPort = 65000;
private const string kDefaultClientApplicationUri =
"urn:localhost:opcfoundation.org:ServerFixtureClient";
/// <summary>
/// Create and Activate a session without security.
/// </summary>
/// <remarks>
/// The request header is used to call services directly,
/// without establishing a session with a client.
/// </remarks>
/// <param name="server">The server to connect to.</param>
/// <param name="sessionName">A session name.</param>
/// <returns>The request header for the session.</returns>
/// <exception cref="NotSupportedException"></exception>
public static async Task<(RequestHeader, SecureChannelContext)> CreateAndActivateSessionAsync(
this SessionServerBase server,
string sessionName,
bool useSecurity = false,
UserIdentityToken identityToken = null,
double sessionTimeout = DefaultSessionTimeout,
uint maxResponseMessageSize = DefaultMaxResponseMessageSize,
string clientApplicationUri = null)
{
if (clientApplicationUri != null && clientApplicationUri.Length == 0)
{
throw new ArgumentException(
"The client ApplicationUri must not be empty.",
nameof(clientApplicationUri));
}
// Find TCP endpoint
ArrayOf<EndpointDescription> endpoints = server.GetEndpoints();
EndpointDescription endpoint = useSecurity
? endpoints.Find(e =>
e.TransportProfileUri
.Equals(Profiles.UaTcpTransport, StringComparison.Ordinal) &&
e.SecurityMode == MessageSecurityMode.Sign &&
e.SecurityPolicyUri == SecurityPolicies.Basic256Sha256) ??
endpoints.Find(e =>
e.TransportProfileUri
.Equals(Profiles.HttpsBinaryTransport, StringComparison.Ordinal) &&
e.SecurityMode == MessageSecurityMode.Sign &&
e.SecurityPolicyUri == SecurityPolicies.Basic256Sha256)
: endpoints.Find(e =>
e.TransportProfileUri
.Equals(Profiles.UaTcpTransport, StringComparison.Ordinal) ||
e.TransportProfileUri
.Equals(Profiles.HttpsBinaryTransport, StringComparison.Ordinal));
endpoint ??= endpoints.Find(e =>
e.TransportProfileUri
.Equals(Profiles.UaTcpTransport, StringComparison.Ordinal) ||
e.TransportProfileUri
.Equals(Profiles.HttpsBinaryTransport, StringComparison.Ordinal)) ??
throw new NotSupportedException("Unsupported transport profile or security policy.");
if (useSecurity)
{
endpoint.SecurityMode = MessageSecurityMode.Sign;
endpoint.SecurityPolicyUri = SecurityPolicies.Basic256Sha256;
}
else
{
endpoint.SecurityMode = MessageSecurityMode.None;
endpoint.SecurityPolicyUri = SecurityPolicies.None;
}
Certificate clientCertificate = null;
try
{
ByteString clientNonce = default;
ByteString clientCertificateData = default;
byte[] clientChannelCertificate = null;
byte[] serverChannelCertificate = null;
byte[] channelThumbprint = null;
if (useSecurity)
{
clientCertificate = CertificateBuilder
.Create("CN=ServerFixtureClient")
.AddExtension(
new X509SubjectAltNameExtension(
clientApplicationUri ?? kDefaultClientApplicationUri,
[Utils.GetHostName()]))
.SetRSAKeySize(CertificateFactory.DefaultKeySize)
.CreateForRSA();
clientNonce = Nonce.CreateRandomNonceData(32).ToByteString();
clientCertificateData = clientCertificate.RawData.ToByteString();
clientChannelCertificate = clientCertificate.RawData;
channelThumbprint = Nonce.CreateRandomNonceData(32);
if (!endpoint.ServerCertificate.IsEmpty)
{
using CertificateCollection serverCertificateChain =
Utils.ParseCertificateChainBlob(
endpoint.ServerCertificate,
server.MessageContext.Telemetry);
serverChannelCertificate = serverCertificateChain[0].RawData;
}
}
var secureChannelContext = new SecureChannelContext(
sessionName,
endpoint,
RequestEncoding.Binary,
clientChannelCertificate,
serverChannelCertificate,
channelThumbprint);
var requestHeader = new RequestHeader();
ApplicationDescription clientDescription = clientApplicationUri == null
? null
: new ApplicationDescription
{
ApplicationUri = clientApplicationUri,
ApplicationName = new LocalizedText("ServerFixtureClient"),
ApplicationType = ApplicationType.Client
};
CreateSessionResponse createSessionResponse = await server.CreateSessionAsync(
secureChannelContext,
requestHeader,
clientDescription,
null,
null,
sessionName,
clientNonce,
clientCertificateData,
sessionTimeout,
maxResponseMessageSize,
RequestLifetime.None).ConfigureAwait(false);
ValidateResponse(createSessionResponse.ResponseHeader);
SignatureData clientSignature = null;
if (useSecurity)
{
SecurityPolicyInfo securityPolicy =
SecurityPolicies.Default.GetInfo(endpoint.SecurityPolicyUri!)!;
using CertificateCollection serverCertificateChain =
Utils.ParseCertificateChainBlob(
createSessionResponse.ServerCertificate,
server.MessageContext.Telemetry);
byte[] dataToSign = securityPolicy.GetClientSignatureData(
secureChannelContext.ChannelThumbprint,
createSessionResponse.ServerNonce.ToArray(),
serverCertificateChain[0].RawData,
secureChannelContext.ServerChannelCertificate,
secureChannelContext.ClientChannelCertificate,
clientNonce.ToArray());
clientSignature = SecurityPolicies.Default.CreateSignatureData(
securityPolicy,
clientCertificate!,
dataToSign);
}
requestHeader.AuthenticationToken = createSessionResponse.AuthenticationToken;
ActivateSessionResponse activateSessionResponse = await server.ActivateSessionAsync(
secureChannelContext,
requestHeader,
clientSignature,
[],
[],
identityToken != null ? new ExtensionObject(identityToken) : default,
null,
RequestLifetime.None).ConfigureAwait(false);
ValidateResponse(activateSessionResponse.ResponseHeader);
return (requestHeader, secureChannelContext);
}
finally
{
clientCertificate?.Dispose();
}
}
/// <summary>
/// Close a session.
/// </summary>
/// <param name="server">The server where the session is active.</param>
/// <param name="requestHeader">The request header of the session.</param>
public static async Task CloseSessionAsync(
this SessionServerBase server,
SecureChannelContext secureChannelContext,
RequestHeader requestHeader,
CancellationToken ct)
{
// close session
using var requestLifetime = new RequestLifetime(ct);
CloseSessionResponse response = await server.CloseSessionAsync(
secureChannelContext,
requestHeader,
true,
requestLifetime).ConfigureAwait(false);
ValidateResponse(response.ResponseHeader);
}
/// <summary>
/// Validate the response of a service call.
/// </summary>
/// <param name="header">The response header of the service call.</param>
/// <exception cref="ServiceResultException"></exception>
public static void ValidateResponse(ResponseHeader header)
{
if (header == null)
{
throw new ServiceResultException(
StatusCodes.BadUnknownResponse,
"Null header in response.");
}
if (StatusCode.IsBad(header.ServiceResult))
{
throw new ServiceResultException(
new ServiceResult(
header.ServiceResult,
header.ServiceDiagnostics,
header.StringTable));
}
}
/// <summary>
/// Validate the response of a service call and validate the number of items returned.
/// </summary>
/// <typeparam name="TRequest"></typeparam>
/// <typeparam name="TResponse"></typeparam>
/// <remarks>
/// On the client the generated code already validates the response but the
/// check is duplicated here to catch also issues when running tests within
/// the shared client/server context.
/// </remarks>
/// <param name="header">The response header of the service call.</param>
/// <param name="response">The list of returned values by the service call.</param>
/// <param name="request">The list of requests passed to the service call.</param>
/// <exception cref="ArgumentException"></exception>
/// <exception cref="ServiceResultException"></exception>
public static void ValidateResponse<TRequest, TResponse>(
ResponseHeader header,
ArrayOf<TResponse> response,
ArrayOf<TRequest> request)
{
ValidateResponse(header);
if (response is ArrayOf<DiagnosticInfo>)
{
throw new ArgumentException(
"Must call ValidateDiagnosticInfos() for ArrayOf<DiagnosticInfo>.",
nameof(response));
}
if (response.Count != request.Count)
{
throw ServiceResultException.Unexpected(
"The server returned {0} responses but {1} requests were made.",
response.Count,
request.Count);
}
for (int ii = 0; ii < response.Count; ii++)
{
if (response[ii] is null)
{
throw ServiceResultException.Unexpected(
"The server returned responses contain a null response at index {0}.", ii);
}
}
}
/// <summary>
/// Validate the diagnostic response of a service call.
/// </summary>
/// <typeparam name="TRequest"></typeparam>
/// <exception cref="ServiceResultException"></exception>
public static void ValidateDiagnosticInfos<TRequest>(
ArrayOf<DiagnosticInfo> response,
ArrayOf<TRequest> request,
ArrayOf<string> stringTable,
ILogger logger)
{
// returning an empty list for diagnostic info arrays is allowed.
if (response.Count != 0)
{
if (response.Count != request.Count)
{
throw ServiceResultException.Unexpected(
"The server forgot to fill in the DiagnosticInfos array correctly when returning an operation level error.");
}
// now validate the string table
if (!stringTable.IsEmpty)
{
for (int ii = 0; ii < response.Count; ii++)
{
if (response[ii] is DiagnosticInfo diagnosticInfo &&
!diagnosticInfo.IsNullDiagnosticInfo)
{
if (diagnosticInfo.NamespaceUri >= stringTable.Count ||
diagnosticInfo.SymbolicId >= stringTable.Count ||
diagnosticInfo.Locale >= stringTable.Count ||
diagnosticInfo.LocalizedText >= stringTable.Count)
{
throw ServiceResultException.Unexpected(
"The server forgot to fill in string table for the DiagnosticInfos " +
"array correctly when returning an operation level error.");
}
var serviceResult = new ServiceResult(
StatusCodes.Good,
ii,
response,
stringTable);
if (logger.IsEnabled(LogLevel.Information))
{
logger.LogInformation("DiagnosticInfo: {ServiceResult}", serviceResult);
}
}
}
}
}
}
/// <summary>
/// Create a browse description from a node id collection.
/// </summary>
/// <param name="nodeIdCollection">The node id collection.</param>
/// <param name="template">The template for the browse description for each node id.</param>
public static ArrayOf<BrowseDescription> CreateBrowseDescriptionCollectionFromNodeId(
ArrayOf<NodeId> nodeIdCollection,
BrowseDescription template)
{
var browseDescriptionCollection = new List<BrowseDescription>();
foreach (NodeId nodeId in nodeIdCollection)
{
var browseDescription = (BrowseDescription)template.MemberwiseClone();
browseDescription.NodeId = nodeId;
browseDescriptionCollection.Add(browseDescription);
}
return browseDescriptionCollection;
}
/// <summary>
/// Create the continuation point collection from the browse result
/// collection for the BrowseNext service.
/// </summary>
/// <param name="browseResultCollection">The browse result collection to use.</param>
/// <returns>The collection of continuation points for the BrowseNext service.</returns>
public static ArrayOf<ByteString> PrepareBrowseNext(
ArrayOf<BrowseResult> browseResultCollection)
{
var continuationPoints = new List<ByteString>();
foreach (BrowseResult browseResult in browseResultCollection)
{
if (!browseResult.ContinuationPoint.IsEmpty)
{
continuationPoints.Add(browseResult.ContinuationPoint);
}
}
return continuationPoints;
}
/// <summary>
/// A dictionary of all node attributes.
/// </summary>
public static readonly ReadOnlyDictionary<uint, DataValue> AttributesIds = new(
new SortedDictionary<uint, DataValue>
{
{ Attributes.NodeId, default },
{ Attributes.NodeClass, default },
{ Attributes.BrowseName, default },
{ Attributes.DisplayName, default },
{ Attributes.Description, default },
{ Attributes.WriteMask, default },
{ Attributes.UserWriteMask, default },
{ Attributes.DataType, default },
{ Attributes.ValueRank, default },
{ Attributes.ArrayDimensions, default },
{ Attributes.AccessLevel, default },
{ Attributes.UserAccessLevel, default },
{ Attributes.Historizing, default },
{ Attributes.MinimumSamplingInterval, default },
{ Attributes.EventNotifier, default },
{ Attributes.Executable, default },
{ Attributes.UserExecutable, default },
{ Attributes.IsAbstract, default },
{ Attributes.InverseName, default },
{ Attributes.Symmetric, default },
{ Attributes.ContainsNoLoops, default },
{ Attributes.DataTypeDefinition, default },
{ Attributes.RolePermissions, default },
{ Attributes.UserRolePermissions, default },
{ Attributes.AccessRestrictions, default },
{ Attributes.AccessLevelEx, default }
});
/// <summary>
/// Get free IP Port.
/// </summary>
public static int GetNextFreeIPPort()
{
var endpoint = new IPEndPoint(IPAddress.Any, 0);
using var socket = new Socket(
endpoint.AddressFamily,
SocketType.Stream,
ProtocolType.Tcp);
socket.Bind(endpoint);
if (socket.LocalEndPoint is IPEndPoint ep)
{
return ep.Port;
}
return 0;
}
/// <summary>
/// Whether starting a server failed because the port it was asked to
/// bind is already taken, in which case picking another one and
/// retrying is worthwhile.
/// </summary>
/// <remarks>
/// The port handed out by <see cref="GetNextFreeIPPort"/> is only free
/// at the instant it is queried - the socket is closed again before the
/// server binds - so a parallel fixture or an unrelated process on the
/// agent can take it in between. How that surfaces depends on the
/// transport: the UA-TCP listener reports
/// <see cref="StatusCodes.BadNoCommunication"/>, while the HTTPS
/// listeners bind through Kestrel, which throws an
/// <see cref="System.IO.IOException"/> wrapping an
/// <c>AddressInUseException</c> and a
/// <see cref="SocketError.AddressAlreadyInUse"/>
/// <see cref="SocketException"/>. Only the former used to be retried,
/// so the https and opc.https fixtures failed their entire
/// OneTimeSetUp on a port collision.
/// </remarks>
/// <param name="exception">The exception the server start threw.</param>
public static bool IsPortUnavailable(Exception exception)
{
for (Exception current = exception; current != null; current = current.InnerException)
{
if (current is ServiceResultException sre &&
sre.StatusCode == StatusCodes.BadNoCommunication)
{
return true;
}
if (current is SocketException socket &&
(socket.SocketErrorCode == SocketError.AddressAlreadyInUse ||
socket.SocketErrorCode == SocketError.AccessDenied))
{
return true;
}
// Kestrel reports the collision as
// Microsoft.AspNetCore.Connections.AddressInUseException. It
// normally carries the SocketException matched above as its
// inner exception, but match the type by name too so the
// detection does not depend on that - and so this file needs no
// reference to the ASP.NET Core connection abstractions.
if (current.GetType().Name == "AddressInUseException")
{
return true;
}
if (current is AggregateException aggregate)
{
foreach (Exception inner in aggregate.InnerExceptions)
{
if (IsPortUnavailable(inner))
{
return true;
}
}
}
}
return false;
}
}
}