GithubStar/UA-.NETStandard
1
0
Fork 0
mirror of https://github.com/OPCFoundation/UA-.NETStandard.git synced 2026-10-16 21:02:51 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fail CI when a test fixture's setup or teardown fails (#4723)

Browse Source 
# Description

`get-test-verdict.ps1` accepts a non-zero `dotnet test` exit whenever
the TRX counters show no failure, on the assumption that the host died
at exit after every teardown had run. A failed fixture setup or teardown
breaks that assumption. The assembly-level `[OneTimeTearDown]`
certificate leak check fails after its tests passed, no TRX counter
records it, and the host exits 1. In the last 10 master runs, CI
accepted exactly that as an "at-exit stall" for `Opc.Ua.Gds.Tests` and
`Opc.Ua.Server.Tests` on every leg (net48, and net10.0 on Windows,
Linux, and macOS). #4719 fixed the Server.Tests leak; #4721 fixes the
GDS leak.

The NUnit adapter reports such a failure as a TRX run message, `<RunInfo
outcome="Error"><Text>TearDown failed for test fixture
Opc.Ua.Gds.Tests.LeakDetectionSetup</Text></RunInfo>`, followed by a
second one with the failure message. For a setup failure it writes
`Setup failed for test fixture …`.

Changes:

- `Measure-TestResults` collects these messages. `Get-TestRunVerdict
-FixtureFailures` rejects a run that has any of them, whatever the exit
code, and names the fixture in the reason.
- A host that dies at exit after a clean run is still tolerated. The
only other tolerated case in the last 10 master runs,
`Opc.Ua.Client.Tests` on net48 in run 37835429177, logged "Test host
process crashed" and stays tolerated.
- `Measure-TestResults` and `Get-CounterValue` move from
`run-dotnet-tests.ps1` to the dot-sourced `get-test-verdict.ps1`, so
`CiTestVerdictTests` can feed them TRX files. The source check of the
counter set becomes a behavioral test.
- `docs/DeveloperGuide.md` describes the rule.

**Merge order:** after #4727 (#4719 and #4721 are on master). With #4721
merged, this PR's CI still rejects `Opc.Ua.Gds.Tests` on net48: eight
`PushTest` fixtures fail in `TearDown` and leak 88 certificates when
their setup is ignored, which master accepted as an "at-exit stall".
#4727 fixes them. On every other leg, `Opc.Ua.Gds.Tests` now passes
under the rule.

### Validation

| Command | Result |
| --- | --- |
| `dotnet test tests/Opc.Ua.Tools.Tests -f net10.0 --filter
FullyQualifiedName~CiTestVerdictTests` with the scripts from master | 6
new tests fail, 13 pass |
| Same with this change | 19 passed |
| `Measure-TestResults` and `Get-TestRunVerdict` on the TRX of a full
`Opc.Ua.Gds.Tests` run without #4721, exit code 1 | Rejected: `1 test
fixture setup or teardown failure(s): TearDown failed for test fixture
Opc.Ua.Gds.Tests.LeakDetectionSetup.` |
| Same on the TRX of a run with #4721, exit code 1 | Tolerated, as
before |
| CI of this PR before #4721 merged | `Opc.Ua.Gds.Tests` rejected on all
four legs for `TearDown failed for test fixture
Opc.Ua.Gds.Tests.LeakDetectionSetup`; every other project passed |
| CI after merging master with #4721 (0e8ef033a) | `Opc.Ua.Gds.Tests`
passes on Linux, Windows and macOS net10.0; on net48 it is rejected for
9 fixture failures in `PushTest` and `LeakDetectionSetup`, fixed by
#4727 |
| `dotnet test tests/Opc.Ua.Tools.Tests -f net10.0` | 947 passed, 1
skipped, 1 failed:
`CiMatrixScriptTests.JobTimeoutBoundsEveryProjectsCombinedBudgetAsync("full")`,
which this change does not touch; it passes on rerun, alone and with
`CiMatrixScriptTests` and `CiTestVerdictTests` (40 passed) |

## Related Issues

- No tracking issue. Follows #4719 and #4721.

## Checklist

- [ ] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [x] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [ ] I have addressed **all** PR feedback received.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: marcschier <11168470+marcschier@users.noreply.github.com>
This commit is contained in:
Marc Schier
2026-10-10 00:24:49 +02:00
committed by GitHub
parent 2cdf0de97f
commit 9099aee13e
5 changed files with 360 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/scripts/get-test-verdict.ps1
+106 -7
View File
@@ -7,15 +7,17 @@
tests/Opc.Ua.Tools.Tests/CiTestVerdictTests.cs. It is separated from the
executor because it is the single decision that determines whether continuous
integration reports green: a rule that is too strict reports a false red, and
one that is too lax lets a genuinely broken suite merge. Keeping it in its own
file lets it be tested without building or running anything.
one that is too lax lets a genuinely broken suite merge. Keeping it, and the
TRX parsing it consumes, in their own file lets both be tested without
building or running anything.
The emitted TRX decides the verdict, not the 'dotnet test' exit code. A
non-zero exit is tolerated when - and only when - the results record at least
one *passing* test and no failure of any kind, which means the host died
during process exit after the last test and every teardown had already run.
A host that dies mid-run leaves a non-zero failed/aborted/passedButRunAborted counter and is
rejected, and a suite whose tests were all skipped is rejected as well.
one *passing* test, no failure of any kind, and no test fixture whose setup or
teardown failed, which means the host died during process exit after the last
test and every teardown had already run. A host that dies mid-run leaves a
non-zero failed/aborted/passedButRunAborted counter and is rejected, and a
suite whose tests were all skipped is rejected as well.
#>
<#
@@ -38,6 +40,11 @@
inconclusive, notRunnable, disconnected, warning, completed, inProgress,
and pending - a fail-closed set.
.PARAMETER FixtureFailures
The test fixtures whose setup or teardown failed, one message per fixture,
as Measure-TestResults reads them from the TRX run messages. No counter
records these failures.
.PARAMETER ExitCode
Exit code of the 'dotnet test' process.
@@ -62,7 +69,8 @@ function Get-TestRunVerdict
[Parameter(Mandatory = $true)] [int] $Failed,
[Parameter(Mandatory = $true)] [int] $ExitCode,
[Parameter(Mandatory = $true)] [bool] $TimedOut,
[Parameter(Mandatory = $false)] [int] $TimeoutMinutes = 0
[Parameter(Mandatory = $false)] [int] $TimeoutMinutes = 0,
[Parameter(Mandatory = $false)] [string[]] $FixtureFailures = @()
)
# A timeout is never benign: the executor killed a process that was still
@@ -97,6 +105,20 @@ function Get-TestRunVerdict
}
}
# A failed fixture setup or teardown - for example an assembly-level leak
# check in [OneTimeTearDown] - fails after its tests passed, so no counter
# records it, and the host then exits non-zero. Evaluated before the exit
# code so that this exit is not tolerated as an at-exit stall below.
$fixtureFailureList = @($FixtureFailures | Where-Object { -not [string]::IsNullOrWhiteSpace($_) })
if ($fixtureFailureList.Count -gt 0) {
return [pscustomobject]@{
Passed = $false
Tolerated = $false
Reason = ("$($fixtureFailureList.Count) test fixture setup or teardown failure(s): " +
"$($fixtureFailureList -join '; ').")
}
}
if ($Total -le 0) {
return [pscustomobject]@{
Passed = $false
@@ -129,3 +151,80 @@ function Get-TestRunVerdict
return [pscustomobject]@{ Passed = $true; Tolerated = $false; Reason = '' }
}
<#
.SYNOPSIS
Sums the counters of every TRX below a directory and collects the test
fixtures whose setup or teardown failed.
.DESCRIPTION
A failed fixture setup or teardown is not a test result, so no counter
records it. The NUnit adapter reports it as a run message that the TRX
stores in a RunInfo, for example "TearDown failed for test fixture
Opc.Ua.Gds.Tests.LeakDetectionSetup", followed by a second RunInfo with the
failure message. Only the first one names the fixture, so each failed
fixture is reported once.
#>
function Measure-TestResults([string] $directory)
{
$total = 0
$passed = 0
$failed = 0
$fixtureFailures = [System.Collections.Generic.List[string]]::new()
$nonPassingCounters = @(
'failed',
'error',
'timeout',
'aborted',
'passedButRunAborted',
'inconclusive',
'notRunnable',
'disconnected',
'warning',
'completed',
'inProgress',
'pending')
$trxFiles = @(Get-ChildItem -LiteralPath $directory -Recurse -File -Filter *.trx -ErrorAction SilentlyContinue)
foreach ($trxFile in $trxFiles) {
# XmlDocument.Load rather than [xml](Get-Content): PubSub emits several
# megabytes of results and the array-of-lines cast fails on files that
# size.
$document = [System.Xml.XmlDocument]::new()
$document.XmlResolver = $null
$document.Load($trxFile.FullName)
foreach ($counters in $document.GetElementsByTagName('Counters')) {
$total += Get-CounterValue $counters 'total'
$passed += Get-CounterValue $counters 'passed'
foreach ($name in $nonPassingCounters) {
$failed += Get-CounterValue $counters $name
}
}
foreach ($runInfo in $document.GetElementsByTagName('RunInfo')) {
# The NUnit adapter writes "Setup" or "TearDown"; accept either
# spelling of both.
if ($runInfo.InnerText -match '^\s*(set ?up|tear ?down) failed for test fixture [^\r\n]+') {
$fixtureFailures.Add($Matches[0].Trim())
}
}
}
return [pscustomobject]@{
Files = $trxFiles.Count
Total = $total
Passed = $passed
Failed = $failed
FixtureFailures = $fixtureFailures.ToArray()
}
}
<#
.SYNOPSIS
Reads a TRX counter attribute, treating an absent attribute as zero.
#>
function Get-CounterValue([System.Xml.XmlElement] $element, [string] $name)
{
$raw = $element.GetAttribute($name)
if ([string]::IsNullOrEmpty($raw)) {
return 0
}
return [int]$raw
}
.github/scripts/run-dotnet-tests.ps1
+4 -56
View File
@@ -100,8 +100,9 @@ Param(
$ErrorActionPreference = 'Stop'
# The verdict rule lives in its own file so it can be tested without building or
# running anything - see tests/Opc.Ua.Tools.Tests/CiTestVerdictTests.cs.
# The verdict rule and the TRX parsing it consumes live in their own file so they
# can be tested without building or running anything - see
# tests/Opc.Ua.Tools.Tests/CiTestVerdictTests.cs.
. (Join-Path $PSScriptRoot 'get-test-verdict.ps1')
$projectList = @($Projects -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ })
@@ -233,60 +234,6 @@ function Get-ProjectProperties([string] $project)
return ($output.Substring($start, $end - $start + 1) | ConvertFrom-Json).Properties
}
<#
.SYNOPSIS
Sums the counters of every TRX below a directory.
#>
function Measure-TestResults([string] $directory)
{
$total = 0
$passed = 0
$failed = 0
$nonPassingCounters = @(
'failed',
'error',
'timeout',
'aborted',
'passedButRunAborted',
'inconclusive',
'notRunnable',
'disconnected',
'warning',
'completed',
'inProgress',
'pending')
$trxFiles = @(Get-ChildItem -LiteralPath $directory -Recurse -File -Filter *.trx -ErrorAction SilentlyContinue)
foreach ($trxFile in $trxFiles) {
# XmlDocument.Load rather than [xml](Get-Content): PubSub emits several
# megabytes of results and the array-of-lines cast fails on files that
# size.
$document = [System.Xml.XmlDocument]::new()
$document.XmlResolver = $null
$document.Load($trxFile.FullName)
foreach ($counters in $document.GetElementsByTagName('Counters')) {
$total += Get-CounterValue $counters 'total'
$passed += Get-CounterValue $counters 'passed'
foreach ($name in $nonPassingCounters) {
$failed += Get-CounterValue $counters $name
}
}
}
return [pscustomobject]@{ Files = $trxFiles.Count; Total = $total; Passed = $passed; Failed = $failed }
}
<#
.SYNOPSIS
Reads a TRX counter attribute, treating an absent attribute as zero.
#>
function Get-CounterValue([System.Xml.XmlElement] $element, [string] $name)
{
$raw = $element.GetAttribute($name)
if ([string]::IsNullOrEmpty($raw)) {
return 0
}
return [int]$raw
}
$records = @()
foreach ($project in $projectList) {
$stem = [System.IO.Path]::GetFileNameWithoutExtension($project)
@@ -423,6 +370,7 @@ foreach ($project in $projectList) {
-Total $results.Total `
-Passed $results.Passed `
-Failed $results.Failed `
-FixtureFailures $results.FixtureFailures `
-ExitCode $test.ExitCode `
-TimedOut $test.TimedOut `
-TimeoutMinutes $PerProjectTimeoutMinutes
docs/DeveloperGuide.md
+4 -3
View File
@@ -100,7 +100,8 @@ Conventions and requirements:
their allocation stacks, and the NUnit fixture where available. Tracking is enabled by default
in Debug builds and opt-in in Release because stack capture adds per-certificate overhead. The
equivalent AppContext switch is `Opc.Ua.Security.Certificates.CertificateLeakTracking`; set it
before the first `Certificate` is created. For example, in PowerShell:
before the first `Certificate` is created. A leak fails the assembly's `[OneTimeTearDown]`,
which fails the CI run. For example, in PowerShell:
```powershell
$env:OPCUA_CERTIFICATE_LEAK_TRACKING = '1'
@@ -425,11 +426,11 @@ gh run download <run-id> --pattern 'dotnet-results-*' --dir ./ci-results
GitHub refuses to start a run whose matrices expand past 256 jobs, and the limit cannot be raised. If the packed matrix would not fit, the target is raised until it does, so the matrix adapts as test projects are added instead of silently truncating. [`CiMatrixScriptTests`](../tests/Opc.Ua.Tools.Tests/CiMatrixScriptTests.cs) asserts that both scopes fit, that every `*.Tests.csproj` on disk is either scheduled or explicitly excluded, that each profile runs each of its projects exactly once, that only a project heavier than the target exceeds it, and that a narrowed matrix can never expand to nothing.
The executor fails a project when its build fails, when the TRX counters report any failure, when it produces **no** TRX at all, when nothing in it actually ran, or when it outlives its per-project ceiling. A "no tests ran" outcome is a failure, not a pass — and that includes a suite whose tests were *all* skipped, which records a non-zero total while verifying nothing.
The executor fails a project when its build fails, when the TRX counters report any failure, when a test fixture's setup or teardown fails, when it produces **no** TRX at all, when nothing in it actually ran, or when it outlives its per-project ceiling. A "no tests ran" outcome is a failure, not a pass — and that includes a suite whose tests were *all* skipped, which records a non-zero total while verifying nothing.
That per-project ceiling is a **single combined budget** covering the project's build *and* its test run, measured by one stopwatch. The matrix derives each job's `timeout-minutes` from it as `20 + projectCount × perProjectTimeout`, so spending it twice per project would let a batch outlive its job: GitHub would cancel the run, and a cancelled job produces neither the executor's per-project annotation nor its results. A batch whose budgets add up past the job ceiling is an error in `get-ci-matrix.ps1` rather than a clamped `timeout-minutes`, for the same reason. `CiMatrixScriptTests` pins both halves of that arithmetic.
The verdict comes from the emitted TRX rather than from the `dotnet test` exit code, and lives in [`.github/scripts/get-test-verdict.ps1`](../.github/scripts/get-test-verdict.ps1) so it can be tested on its own — see [`CiTestVerdictTests`](../tests/Opc.Ua.Tools.Tests/CiTestVerdictTests.cs). A non-zero exit is tolerated when, and only when, the results record at least one **passing** test and no failure, error, timeout, abort or `passedButRunAborted`. That combination means the host died during process **exit**, after the last test and every teardown had already run; failing it would report a false red. It is not a macOS quirk — Windows hosts do it too (run 35714133848, `test-windows-net48 (5/30)`: `Opc.Ua.Client.Tests` reported 256 passed, 0 failed, host exit 1). A host that dies mid-run leaves a non-zero counter and is still rejected, and a run in which every test was skipped is rejected whatever the exit code says. Every tolerated run raises a warning annotation and is labelled in the job summary, so a host that keeps dying stays visible.
The verdict comes from the emitted TRX rather than from the `dotnet test` exit code, and lives in [`.github/scripts/get-test-verdict.ps1`](../.github/scripts/get-test-verdict.ps1) so it can be tested on its own — see [`CiTestVerdictTests`](../tests/Opc.Ua.Tools.Tests/CiTestVerdictTests.cs). A failed fixture setup or teardown, such as the assembly-level certificate leak check in a `[OneTimeTearDown]`, is not a test result, so no counter records it: the NUnit adapter writes it to the TRX as a run message (`TearDown failed for test fixture …`), and the verdict rejects it whatever the exit code says. A non-zero exit is tolerated when, and only when, the results record at least one **passing** test, no failure, error, timeout, abort or `passedButRunAborted`, and no failed fixture setup or teardown. That combination means the host died during process **exit**, after the last test and every teardown had already run; failing it would report a false red. It is not a macOS quirk — Windows hosts do it too (run 35714133848, `test-windows-net48 (5/30)`: `Opc.Ua.Client.Tests` reported 256 passed, 0 failed, host exit 1). A host that dies mid-run leaves a non-zero counter and is still rejected, and a run in which every test was skipped is rejected whatever the exit code says. Every tolerated run raises a warning annotation and is labelled in the job summary, so a host that keeps dying stays visible.
#### Why a project can be skipped
tests/Opc.Ua.Gds.Tests/PushTest.cs
+19 -6
View File
@@ -295,25 +295,38 @@ namespace Opc.Ua.Gds.Tests
/// <summary>
/// Tear down the Global Discovery Server and disconnect the Client
/// </summary>
/// <remarks>
/// OneTimeSetUpAsync can end early through Assert.Ignore, for example when
/// the GDS test server does not advertise the security policy of the
/// fixture. The push server is then not registered and the fields set
/// after that point are null, but the GDS server is running and must be
/// disposed.
/// </remarks>
[OneTimeTearDown]
protected async Task OneTimeTearDownAsync()
{
try
{
await ConnectGDSClientAsync(true).ConfigureAwait(false);
await UnRegisterPushServerApplicationAsync().ConfigureAwait(false);
if (m_applicationRecord != null)
{
await ConnectGDSClientAsync(true).ConfigureAwait(false);
await UnRegisterPushServerApplicationAsync().ConfigureAwait(false);
}
await m_gdsClient.DisconnectClientAsync().ConfigureAwait(false);
await m_pushClient.DisconnectClientAsync().ConfigureAwait(false);
await m_server.DisposeAsync().ConfigureAwait(false);
}
catch
{
}
finally
{
m_pushClient.Dispose();
m_gdsClient.Dispose();
m_selfSignedServerCert.Dispose();
if (m_server != null)
{
await m_server.DisposeAsync().ConfigureAwait(false);
}
m_pushClient?.Dispose();
m_gdsClient?.Dispose();
m_selfSignedServerCert?.Dispose();
m_selfSignedServerCert = null!;
m_caCert?.Dispose();
m_caCert = null!;
tests/Opc.Ua.Tools.Tests/CiTestVerdictTests.cs
+227 -29
View File
@@ -29,12 +29,15 @@
#if NET10_0
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using System.Xml.Linq;
using NUnit.Framework;
namespace Opc.Ua.Tools.Tests
@@ -126,17 +129,9 @@ namespace Opc.Ua.Tools.Tests
/// complete set before asking for a verdict.
/// </summary>
[Test]
public async Task ExecutorCountsEveryNonPassingTrxCounterAsync()
public async Task EveryNonPassingTrxCounterIsCountedAsync()
{
string executor = Path.Combine(FindRepositoryRoot(), ".github", "scripts", "run-dotnet-tests.ps1");
string source = await File.ReadAllTextAsync(executor).ConfigureAwait(false);
int start = source.IndexOf("function Measure-TestResults", StringComparison.Ordinal);
int end = source.IndexOf("\nfunction Get-CounterValue", start, StringComparison.Ordinal);
Assert.That(start, Is.GreaterThanOrEqualTo(0));
Assert.That(end, Is.GreaterThan(start));
string measure = source[start..end];
string[] expected =
string[] nonPassing =
[
"failed",
"error",
@@ -151,19 +146,128 @@ namespace Opc.Ua.Tools.Tests
"inProgress",
"pending"
];
var counters = new Dictionary<string, int> { ["total"] = 20, ["passed"] = 8 };
foreach (string counter in nonPassing)
{
counters[counter] = 1;
}
TrxResults results = await MeasureAsync(new TrxFile("run.trx", counters, [])).ConfigureAwait(false);
Assert.Multiple(() =>
{
foreach (string counter in expected)
{
Assert.That(
measure,
Does.Contain($"'{counter}'"),
$"Measure-TestResults must reject the TRX '{counter}' counter.");
}
Assert.That(results.Files, Is.EqualTo(1));
Assert.That(results.Total, Is.EqualTo(20));
Assert.That(results.Passed, Is.EqualTo(8));
Assert.That(results.Failed, Is.EqualTo(nonPassing.Length));
Assert.That(results.FixtureFailures, Is.Empty);
});
}
/// <summary>
/// A fixture whose setup or teardown fails - in this repository usually
/// the assembly-level certificate leak check in [OneTimeTearDown] - fails
/// after its tests passed. No TRX counter records it; the NUnit adapter
/// reports it as run messages instead, and the host exits non-zero.
/// Observed on master for Opc.Ua.Gds.Tests and Opc.Ua.Server.Tests on
/// every leg, accepted as an at-exit stall until this rule existed.
/// </summary>
[Test]
public async Task FixtureSetupAndTeardownFailuresAreReadFromTheRunMessagesAsync()
{
var counters = new Dictionary<string, int> { ["total"] = 10, ["passed"] = 10 };
TrxResults results = await MeasureAsync(
new TrxFile(
"teardown.trx",
counters,
[
("Warning", "ClientTest FinishAsync\n\tTEST 13:48:23 validation suppressed"),
("Error", "TearDown failed for test fixture Opc.Ua.Gds.Tests.LeakDetectionSetup"),
("Error", "One or more child tests were ignored\nCertificate leak detected: 1 instance(s)")
]),
new TrxFile(
"setup.trx",
counters,
[
("Error", "Setup failed for test fixture Opc.Ua.Server.Tests.ServerFixture"),
("Error", "OneTimeSetUp: System.InvalidOperationException : the server did not start")
])).ConfigureAwait(false);
string[] expected =
[
"TearDown failed for test fixture Opc.Ua.Gds.Tests.LeakDetectionSetup",
"Setup failed for test fixture Opc.Ua.Server.Tests.ServerFixture"
];
Assert.Multiple(() =>
{
Assert.That(results.Files, Is.EqualTo(2));
Assert.That(results.Total, Is.EqualTo(20));
Assert.That(results.Failed, Is.Zero);
Assert.That(results.FixtureFailures, Is.EquivalentTo(expected));
});
}
/// <summary>
/// Other run messages are not fixture failures. A host that crashes at
/// exit leaves "Test host process crashed" behind, and test output that
/// merely mentions a teardown must not count either; otherwise the
/// at-exit tolerance would no longer apply to the case it exists for.
/// </summary>
[Test]
public async Task OtherRunMessagesAreNotFixtureFailuresAsync()
{
TrxResults results = await MeasureAsync(
new TrxFile(
"crash.trx",
new Dictionary<string, int> { ["total"] = 332, ["passed"] = 332 },
[
("Error", "The active test run was aborted. Reason: Test host process crashed"),
("Warning", "PushTest Run\n\tTEST 14:08:56 TearDown failed for test fixture Other.Fixture")
])).ConfigureAwait(false);
Assert.That(results.FixtureFailures, Is.Empty);
}
/// <summary>
/// A failed fixture setup or teardown is rejected whatever the exit code
/// says, and the reason names the fixture so the job summary shows it.
/// </summary>
[TestCase(0)]
[TestCase(1)]
public async Task FixtureSetupOrTeardownFailureIsRejectedAsync(int exitCode)
{
Verdict verdict = await InvokeAsync(
trxFileCount: 1,
total: 1314,
passed: 1258,
failed: 0,
exitCode: exitCode,
timedOut: false,
fixtureFailures: ["TearDown failed for test fixture Opc.Ua.Gds.Tests.LeakDetectionSetup"])
.ConfigureAwait(false);
Assert.Multiple(() =>
{
Assert.That(verdict.Passed, Is.False);
Assert.That(verdict.Tolerated, Is.False);
Assert.That(verdict.Reason, Does.Contain("Opc.Ua.Gds.Tests.LeakDetectionSetup"));
});
}
/// <summary>
/// The executor has to hand the fixture failures it measured to the
/// verdict; a rule that never receives them protects nothing.
/// </summary>
[Test]
public async Task ExecutorPassesFixtureFailuresToTheVerdictAsync()
{
string executor = Path.Combine(FindRepositoryRoot(), ".github", "scripts", "run-dotnet-tests.ps1");
string source = await File.ReadAllTextAsync(executor).ConfigureAwait(false);
Assert.That(source, Does.Contain("-FixtureFailures $results.FixtureFailures"));
}
/// <summary>
/// A run that produced no TRX at all is a broken run, not an empty one.
/// Every mainline project runs on VSTest and must emit one, so silence
@@ -309,29 +413,105 @@ namespace Opc.Ua.Tools.Tests
int failed,
int exitCode,
bool timedOut,
int timeoutMinutes = 30)
int timeoutMinutes = 30,
string[]? fixtureFailures = null)
{
string root = FindRepositoryRoot();
string script = Path.Combine(root, ".github", "scripts", "get-test-verdict.ps1");
string script = Path.Combine(FindRepositoryRoot(), ".github", "scripts", "get-test-verdict.ps1");
// Dot-source the helper and emit the verdict as JSON, which is what
// makes the rule testable without building or running any project.
string command = string.Format(
CultureInfo.InvariantCulture,
". '{0}'; Get-TestRunVerdict -TrxFileCount {1} -Total {2} -Passed {3} -Failed {4} -ExitCode {5} " +
"-TimedOut ${6} -TimeoutMinutes {7} | ConvertTo-Json -Compress",
script.Replace("'", "''", StringComparison.Ordinal),
". {0}; Get-TestRunVerdict -TrxFileCount {1} -Total {2} -Passed {3} -Failed {4} -ExitCode {5} " +
"-TimedOut ${6} -TimeoutMinutes {7}{8} | ConvertTo-Json -Compress",
Quote(script),
trxFileCount,
total,
passed,
failed,
exitCode,
timedOut ? "true" : "false",
timeoutMinutes);
timeoutMinutes,
fixtureFailures == null
? string.Empty
: " -FixtureFailures @(" + string.Join(",", fixtureFailures.Select(Quote)) + ")");
string output = await RunPowerShellAsync(command, "Get-TestRunVerdict").ConfigureAwait(false);
Verdict? verdict = JsonSerializer.Deserialize<Verdict>(output.Trim(), s_json);
Assert.That(verdict, Is.Not.Null, $"No verdict was emitted: {output}");
return verdict!;
}
/// <summary>
/// Writes each TRX into a fresh directory and measures it with
/// Measure-TestResults, the way the executor reads a project's results.
/// </summary>
private static async Task<TrxResults> MeasureAsync(
params TrxFile[] trxFiles)
{
string directory = Path.Combine(Path.GetTempPath(), "CiTestVerdictTests", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(directory);
try
{
foreach (TrxFile trxFile in trxFiles)
{
WriteTrx(Path.Combine(directory, trxFile.FileName), trxFile.Counters, trxFile.RunInfos);
}
string script = Path.Combine(FindRepositoryRoot(), ".github", "scripts", "get-test-verdict.ps1");
string command = string.Format(
CultureInfo.InvariantCulture,
". {0}; Measure-TestResults {1} | ConvertTo-Json -Compress",
Quote(script),
Quote(directory));
string output = await RunPowerShellAsync(command, "Measure-TestResults").ConfigureAwait(false);
TrxResults? results = JsonSerializer.Deserialize<TrxResults>(output.Trim(), s_json);
Assert.That(results, Is.Not.Null, $"No results were emitted: {output}");
return results!;
}
finally
{
Directory.Delete(directory, true);
}
}
/// <summary>
/// Writes a TRX with the given counters and run messages, the two parts
/// of a VSTest result file that Measure-TestResults reads.
/// </summary>
private static void WriteTrx(
string path,
Dictionary<string, int> counters,
(string Outcome, string Text)[] runInfos)
{
XNamespace ns = "http://microsoft.com/schemas/VisualStudio/TeamTest/2010";
var document = new XDocument(
new XElement(
ns + "TestRun",
new XElement(
ns + "ResultSummary",
new XAttribute("outcome", "Completed"),
new XElement(
ns + "Counters",
counters.Select(counter => new XAttribute(
counter.Key,
counter.Value.ToString(CultureInfo.InvariantCulture)))),
new XElement(
ns + "RunInfos",
runInfos.Select(runInfo => new XElement(
ns + "RunInfo",
new XAttribute("computerName", "ci"),
new XAttribute("outcome", runInfo.Outcome),
new XElement(ns + "Text", runInfo.Text)))))));
document.Save(path);
}
private static async Task<string> RunPowerShellAsync(string command, string function)
{
using var process = new Process();
process.StartInfo.FileName = "pwsh";
process.StartInfo.WorkingDirectory = root;
process.StartInfo.WorkingDirectory = FindRepositoryRoot();
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
PowerShellScriptOutput.ConfigureDeterministicOutput(process.StartInfo);
@@ -350,11 +530,16 @@ namespace Opc.Ua.Tools.Tests
Assert.That(
process.ExitCode,
Is.Zero,
$"Get-TestRunVerdict failed: {PowerShellScriptOutput.Normalize(output + error)}");
$"{function} failed: {PowerShellScriptOutput.Normalize(output + error)}");
return output;
}
Verdict? verdict = JsonSerializer.Deserialize<Verdict>(output.Trim(), s_json);
Assert.That(verdict, Is.Not.Null, $"No verdict was emitted: {output}");
return verdict!;
/// <summary>
/// Quotes a value as a single-quoted PowerShell string literal.
/// </summary>
private static string Quote(string value)
{
return "'" + value.Replace("'", "''", StringComparison.Ordinal) + "'";
}
private static string FindRepositoryRoot()
@@ -377,6 +562,19 @@ namespace Opc.Ua.Tools.Tests
/// </summary>
public sealed record Verdict(bool Passed, bool Tolerated, string Reason);
/// <summary>
/// A TRX for Measure-TestResults: its counters and its run messages.
/// </summary>
private sealed record TrxFile(
string FileName,
Dictionary<string, int> Counters,
(string Outcome, string Text)[] RunInfos);
/// <summary>
/// The result of Measure-TestResults. Deserialized by reflection, hence public.
/// </summary>
public sealed record TrxResults(int Files, int Total, int Passed, int Failed, string[] FixtureFailures);
private static readonly JsonSerializerOptions s_json = new()
{
PropertyNameCaseInsensitive = true