mirror of
https://github.com/OPCFoundation/UA-.NETStandard.git
synced 2026-10-26 20:55:01 +08:00
Marc Schier
994d7b34da
# Description Partition the long-running sample and WoT test suites into bounded, non-overlapping CI shards, and give all workloads the standard 45-minute test-execution budget. Account for setup/build overhead separately in the job timeout so builds do not consume test timeouts. Remove the former sample, integration, WoT, and server-specific timeout exceptions while retaining complete test scheduling. Focused matrix tests pass (25/25); full and PR matrix generation succeeds at 238 and 98 jobs, respectively. PowerShell syntax parsing and `git diff --check` pass. The full solution test suite was not run locally. ## Related Issues - Fixes #4858 ## Checklist _Put an `x` in the boxes that apply. You can complete these step by step after opening the PR._ - [ ] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [x] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - [ ] I have added all necessary documentation. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [ ] I have addressed **all** PR feedback received.
846 lines
37 KiB
PowerShell
846 lines
37 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Expands the GitHub Actions build and test workload for a CI scope.
|
|
|
|
.DESCRIPTION
|
|
Produces the job matrices consumed by .github/workflows/buildandtest.yml
|
|
(pull-request scope) and .github/workflows/nightly.yml (full scope), plus a
|
|
manifest that records every (project x profile x shard) workload the run is
|
|
expected to cover.
|
|
|
|
The workload is described once, here, as a table of profiles. A profile is a
|
|
(runner OS, CustomTestTarget, test-host target framework, configuration,
|
|
category filter, tier) tuple. Projects are discovered from the file system
|
|
with a '*.Tests.csproj' sweep, so neither matrix has to be hand-maintained
|
|
as projects are added or renamed.
|
|
|
|
CustomTestTarget, not '--framework', is the mechanism that pins the stack to
|
|
a single target framework (see targets.props). A profile carries both the
|
|
CustomTestTarget it builds with and the framework its tests actually run on.
|
|
|
|
The project fan-out is batched: each matrix entry carries several projects
|
|
that the executor runs in sequence, keeping per-project results. Batches are
|
|
sized by measured duration rather than by project count. Every batch pays a
|
|
fixed cost of several minutes - checkout, SDK install, and the cold build of
|
|
the shared dependency graph - while a further project in the same batch only
|
|
adds its incremental build and its tests. Two projects per job spent almost
|
|
half of the pull-request matrix compiling the same assemblies over and over,
|
|
so projects are packed, in path order, into batches of about
|
|
BatchTargetMinutes each. Path order keeps related projects (PubSub.*,
|
|
Redundancy.*, WotCon.*) together, so their shared dependencies are built
|
|
once per batch.
|
|
|
|
GitHub Actions also refuses to start a workflow run whose matrices expand
|
|
past 256 jobs, and that limit cannot be raised. If the packed matrix does not
|
|
fit, the target is raised until it does, so the matrix adapts as test
|
|
projects are added instead of silently truncating.
|
|
|
|
.PARAMETER Scope
|
|
'pr' expands the workload that gates a pull request. 'full' expands the
|
|
complete workload Azure Pipelines used to run on a schedule.
|
|
|
|
.PARAMETER ExcludeMacOS
|
|
Drop the macOS test profile. The batch target is deliberately computed from
|
|
the full profile table regardless of this switch, so excluding macOS does not
|
|
reshuffle the batches of the remaining profiles.
|
|
|
|
.PARAMETER OnlyProject
|
|
Restrict the test matrix to the single test project whose path ends with this
|
|
value. Used by the protected private-fuzz-corpus job so it covers exactly the
|
|
profiles this table already enables for that project, instead of repeating
|
|
the profile list in a second place. The non-mainline tiers are dropped, since
|
|
they cover named projects of their own.
|
|
|
|
.PARAMETER MaxEntries
|
|
The documented GitHub Actions ceiling on matrix jobs per workflow run.
|
|
Exceeding it is an error, never a truncation.
|
|
|
|
.PARAMETER ReservedEntries
|
|
Entries reserved for the other full-scope jobs: discovery, four NativeAOT
|
|
legs, eight crash-corpus replays, coverage, the summary, and one spare.
|
|
|
|
.PARAMETER MaxJobTimeoutMinutes
|
|
Upper bound on a batch job's 'timeout-minutes', below the 360-minute ceiling
|
|
GitHub enforces on hosted runners. A batch whose per-project budgets add up
|
|
past this is an error rather than a clamp: clamping would let the executor
|
|
outlive the job, which is the one case that produces no annotation and no
|
|
results. The packer therefore never puts more projects in one batch than
|
|
this ceiling can budget for.
|
|
|
|
.PARAMETER BatchTargetMinutes
|
|
The measured duration, in minutes, a batch is packed up to, on top of its
|
|
fixed cost. Weights come from DurationsPath and are the slowest pull-request
|
|
profile's figures, so batches on the faster runners finish sooner. A project
|
|
heavier than the target gets a batch of its own. Raised in steps of five
|
|
when the packed matrix would not fit MaxEntries.
|
|
|
|
.PARAMETER DurationsPath
|
|
JSON table of per-project weights in minutes. Defaults to
|
|
.github/ci-test-durations.json. A project missing from it is weighted with
|
|
the table's defaultMinutes, so a new project is still scheduled; the table
|
|
only steers balance, never what runs. Regenerate it with
|
|
.github/scripts/update-test-durations.ps1.
|
|
|
|
.PARAMETER RepositoryRoot
|
|
Repository root. Defaults to the root two levels above this script.
|
|
|
|
.PARAMETER ManifestPath
|
|
Optional path to write the expanded workload manifest to. The workflows
|
|
upload it so a run's intended coverage can be compared against another CI
|
|
system's inventory.
|
|
#>
|
|
|
|
Param(
|
|
[ValidateSet('pr', 'full')]
|
|
[string] $Scope = 'pr',
|
|
[switch] $ExcludeMacOS,
|
|
[string] $OnlyProject = '',
|
|
[int] $MaxEntries = 256,
|
|
[int] $ReservedEntries = 16,
|
|
[int] $MaxJobTimeoutMinutes = 350,
|
|
[int] $BatchTargetMinutes = 30,
|
|
[string] $DurationsPath = '',
|
|
[string] $RepositoryRoot = '',
|
|
[string] $ManifestPath = ''
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
if ([string]::IsNullOrWhiteSpace($RepositoryRoot)) {
|
|
$RepositoryRoot = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot '..' '..')).Path
|
|
}
|
|
else {
|
|
$RepositoryRoot = (Resolve-Path -LiteralPath $RepositoryRoot).Path
|
|
}
|
|
if ([string]::IsNullOrWhiteSpace($DurationsPath)) {
|
|
$DurationsPath = Join-Path $RepositoryRoot '.github' 'ci-test-durations.json'
|
|
}
|
|
if ($BatchTargetMinutes -lt 1) {
|
|
throw "BatchTargetMinutes must be positive, not $BatchTargetMinutes."
|
|
}
|
|
|
|
# Runner images: GitHub-hosted labels.
|
|
$RunnerImages = @{
|
|
windows = 'windows-latest'
|
|
linux = 'ubuntu-latest'
|
|
macos = 'macos-latest'
|
|
}
|
|
|
|
# Hosted runner architectures, recorded in the manifest so a comparison against
|
|
# another CI system's inventory is not silently architecture-blind.
|
|
$RunnerArchitectures = @{
|
|
windows = 'x64'
|
|
linux = 'x64'
|
|
macos = 'arm64'
|
|
}
|
|
|
|
$LatestSdk = '10.0.402'
|
|
|
|
# The category filter the fast legs use. Tiers that lift it declare so below.
|
|
$DefaultFilter = 'TestCategory!=LongRunning&TestCategory!=Stress'
|
|
|
|
# Projects that are not part of the mainline sweep:
|
|
# Aot - published and run as native executables rather than
|
|
# by 'dotnet test'; the workflows have their own jobs.
|
|
# Stress - opt-in tier owned by .github/workflows/stress-test.yml.
|
|
# Subscriptions.Durable - its own tier, run unfiltered below.
|
|
# OneFuzz.Validator - pins net10.0 to match the net10.0-only drop it
|
|
# validates, so a legacy leg fails with NETSDK1005
|
|
# before the no-op shell can take effect. It is covered
|
|
# by the fuzz-public-drop job instead.
|
|
$MainlineExclusions = @(
|
|
'Opc.Ua.Aot.Tests.csproj',
|
|
'Opc.Ua.Stress.Tests.csproj',
|
|
'Opc.Ua.Subscriptions.Durable.Tests.csproj',
|
|
'Opc.Ua.OneFuzz.Validator.Tests.csproj')
|
|
|
|
$DurableProject = 'Opc.Ua.Subscriptions.Durable.Tests.csproj'
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Sorts strings with an ordinal comparison so the batches a run produces do not
|
|
depend on the runner's culture.
|
|
#>
|
|
function Sort-Ordinal([string[]] $values)
|
|
{
|
|
$list = [System.Collections.Generic.List[string]]::new()
|
|
foreach ($value in $values) {
|
|
$list.Add($value)
|
|
}
|
|
$list.Sort([System.StringComparer]::Ordinal)
|
|
return $list.ToArray()
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Converts an absolute path below the repository root into a forward-slashed
|
|
repository-relative path.
|
|
#>
|
|
function ConvertTo-RelativePath([string] $fullName)
|
|
{
|
|
return $fullName.Substring($RepositoryRoot.Length).TrimStart('\', '/').Replace('\', '/')
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Discovers files below the repository root, skipping build output.
|
|
#>
|
|
function Find-RepositoryFile([string] $filter)
|
|
{
|
|
$found = Get-ChildItem -LiteralPath $RepositoryRoot -Recurse -File -Filter $filter |
|
|
Where-Object { $_.FullName -notmatch '[\\/](bin|obj|\.git)[\\/]' } |
|
|
ForEach-Object { ConvertTo-RelativePath $_.FullName }
|
|
return Sort-Ordinal @($found)
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Returns the SDK versions a profile needs, newest last so the latest SDK
|
|
drives the build while the older runtime is available to host its tests.
|
|
#>
|
|
function Get-SdkVersions([string] $framework)
|
|
{
|
|
switch ($framework) {
|
|
'net8.0' { return @('8.0.x', $LatestSdk) }
|
|
'net9.0' { return @('9.0.x', $LatestSdk) }
|
|
default { return @($LatestSdk) }
|
|
}
|
|
}
|
|
|
|
# The workload. Each profile fans out over the project set named by its tier.
|
|
#
|
|
# customTestTarget - passed as /p:CustomTestTarget for restore, build and test
|
|
# framework - the target framework the tests actually execute on
|
|
# scopes - which CI scopes include the profile
|
|
#
|
|
# The Linux net10.0 Release mainline profile is pull-request only on purpose: in
|
|
# the full scope the long-running tier runs the identical tuple with the category
|
|
# filter lifted, so keeping both would run the same tests twice.
|
|
$Profiles = @(
|
|
@{ id = 'windows-net48'; os = 'windows'; customTestTarget = 'net48'; framework = 'net48'; configuration = 'Release'; scopes = @('pr', 'full') }
|
|
@{ id = 'windows-net10.0'; os = 'windows'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Release'; scopes = @('pr', 'full') }
|
|
@{ id = 'linux-net10.0'; os = 'linux'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Release'; scopes = @('pr') }
|
|
@{ id = 'macos-net10.0'; os = 'macos'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Release'; scopes = @('pr', 'full') }
|
|
@{ id = 'windows-net10.0-debug'; os = 'windows'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Debug'; scopes = @('full') }
|
|
@{ id = 'linux-net10.0-debug'; os = 'linux'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Debug'; scopes = @('full') }
|
|
@{ id = 'windows-net9.0'; os = 'windows'; customTestTarget = 'net9.0'; framework = 'net9.0'; configuration = 'Release'; scopes = @('full') }
|
|
@{ id = 'linux-net9.0'; os = 'linux'; customTestTarget = 'net9.0'; framework = 'net9.0'; configuration = 'Release'; scopes = @('full') }
|
|
@{ id = 'windows-net8.0'; os = 'windows'; customTestTarget = 'net8.0'; framework = 'net8.0'; configuration = 'Release'; scopes = @('full') }
|
|
@{ id = 'linux-net8.0'; os = 'linux'; customTestTarget = 'net8.0'; framework = 'net8.0'; configuration = 'Release'; scopes = @('full') }
|
|
@{ id = 'linux-long-running'; os = 'linux'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Release'; scopes = @('full')
|
|
tier = 'long-running'; filter = ''; hangTimeout = '30m'; coverage = $false
|
|
}
|
|
@{ id = 'windows-durable'; os = 'windows'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Release'; scopes = @('full')
|
|
tier = 'durable'; filter = ''; hangTimeout = '30m'; coverage = $false
|
|
}
|
|
@{ id = 'linux-durable'; os = 'linux'; customTestTarget = 'net10.0'; framework = 'net10.0'; configuration = 'Release'; scopes = @('full')
|
|
tier = 'durable'; filter = ''; hangTimeout = '30m'; coverage = $false
|
|
}
|
|
)
|
|
|
|
# Solution builds. 'solutions' is either '*' for every discovered .slnx or an
|
|
# explicit list. Every solution across every
|
|
# supported target framework in both configurations for the full scope, narrowed
|
|
# for pull requests, plus the Linux leg that proves UA.slnx still compiles for
|
|
# every Linux-capable target framework.
|
|
$BuildProfiles = @(
|
|
@{ id = 'pr-windows'; os = 'windows'; scopes = @('pr'); solutions = '*'
|
|
tfms = @('net48', 'net10.0'); configurations = @('Debug', 'Release')
|
|
}
|
|
@{ id = 'pr-linux'; os = 'linux'; scopes = @('pr'); solutions = @('UA.slnx')
|
|
tfms = @('net8.0', 'net9.0', 'net10.0'); configurations = @('Release')
|
|
}
|
|
@{ id = 'full-windows'; os = 'windows'; scopes = @('full'); solutions = '*'
|
|
tfms = @('net48', 'net8.0', 'net9.0', 'net10.0')
|
|
configurations = @('Debug', 'Release')
|
|
}
|
|
@{ id = 'full-linux'; os = 'linux'; scopes = @('full'); solutions = @('UA.slnx')
|
|
tfms = @('net8.0', 'net9.0', 'net10.0'); configurations = @('Release')
|
|
}
|
|
)
|
|
|
|
# Every batch carries this much on top of its packed weight: checkout, SDK
|
|
# install, restore, and the cold build of the dependency graph its first
|
|
# project pulls in. Run 37520359929 measured 3.8-4.9 minutes for that first
|
|
# build against 0.4-0.9 minutes for each further project in the same batch.
|
|
$BatchFixedMinutes = 6
|
|
|
|
# Test execution gets a consistent 45-minute ceiling for every project. The job
|
|
# timeout adds a fixed 20-minute allowance for checkout, restore, and builds.
|
|
# Keep 45 minutes for mainline too; projects whose test execution exceeds this
|
|
# standard ceiling are split into filtered shards.
|
|
$PerProjectTimeoutMinutes = 45
|
|
$DedicatedTestProjects = @('Opc.Ua.Server.Tests', 'Opc.Ua.WotCon.Tests', 'Opc.Ua.Types.Tests')
|
|
|
|
# 20 minutes of fixed cost plus one per-project ceiling for each project has to
|
|
# stay below MaxJobTimeoutMinutes, which caps how many projects one batch takes.
|
|
$MaxProjectsPerBatch = [System.Math]::Floor(($MaxJobTimeoutMinutes - 20) / $PerProjectTimeoutMinutes)
|
|
if ($MaxProjectsPerBatch -lt 1) {
|
|
throw "MaxJobTimeoutMinutes=$MaxJobTimeoutMinutes cannot budget even one $PerProjectTimeoutMinutes-minute project."
|
|
}
|
|
|
|
if (-not (Test-Path -LiteralPath $DurationsPath -PathType Leaf)) {
|
|
throw "The per-project duration table '$DurationsPath' does not exist."
|
|
}
|
|
$durationTable = Get-Content -LiteralPath $DurationsPath -Raw | ConvertFrom-Json -AsHashtable
|
|
$DefaultProjectMinutes = [int]$durationTable.defaultMinutes
|
|
if ($DefaultProjectMinutes -lt 1) {
|
|
throw "'$DurationsPath' must declare a positive defaultMinutes."
|
|
}
|
|
$ProjectMinutes = @{}
|
|
foreach ($entry in $durationTable.projects.GetEnumerator()) {
|
|
if ([int]$entry.Value -lt 1) {
|
|
throw "'$DurationsPath' gives '$($entry.Key)' a weight of $($entry.Value); every weight must be at least one minute."
|
|
}
|
|
$ProjectMinutes[$entry.Key] = [int]$entry.Value
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Returns the packing weight of a project, in minutes.
|
|
#>
|
|
function Get-ProjectMinutes([string] $project)
|
|
{
|
|
$stem = [System.IO.Path]::GetFileNameWithoutExtension($project)
|
|
if ($ProjectMinutes.ContainsKey($stem)) {
|
|
return $ProjectMinutes[$stem]
|
|
}
|
|
return $DefaultProjectMinutes
|
|
}
|
|
|
|
$allTestProjects = Find-RepositoryFile '*.Tests.csproj'
|
|
$mainlineProjects = @($allTestProjects | Where-Object { $MainlineExclusions -notcontains (Split-Path $_ -Leaf) })
|
|
$durableProjects = @($allTestProjects | Where-Object { (Split-Path $_ -Leaf) -eq $DurableProject })
|
|
$solutions = Find-RepositoryFile '*.slnx'
|
|
|
|
if ($mainlineProjects.Count -eq 0) {
|
|
throw "Discovery found no mainline test projects beneath '$RepositoryRoot'. A matrix that runs nothing would report success without testing anything."
|
|
}
|
|
if ($durableProjects.Count -eq 0) {
|
|
throw "Discovery found no '$DurableProject'. The durable-subscription tier would silently disappear."
|
|
}
|
|
if ($solutions.Count -eq 0) {
|
|
throw "Discovery found no '*.slnx' beneath '$RepositoryRoot'."
|
|
}
|
|
|
|
if (-not [string]::IsNullOrWhiteSpace($OnlyProject)) {
|
|
$normalized = $OnlyProject.Replace('\', '/')
|
|
$mainlineProjects = @($mainlineProjects | Where-Object { $_.EndsWith($normalized, [System.StringComparison]::OrdinalIgnoreCase) })
|
|
if ($mainlineProjects.Count -eq 0) {
|
|
throw "No discovered test project path ends with '$OnlyProject'. A matrix that runs nothing would report success without testing anything."
|
|
}
|
|
# The named tiers cover named projects, so narrowing to one project leaves
|
|
# them with nothing to run rather than silently re-running it unfiltered.
|
|
$durableProjects = @()
|
|
}
|
|
|
|
$projectsByTier = @{
|
|
mainline = $mainlineProjects
|
|
'long-running' = if ([string]::IsNullOrWhiteSpace($OnlyProject)) { $mainlineProjects } else { @() }
|
|
durable = $durableProjects
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Expands the solution build matrix for a scope.
|
|
#>
|
|
function Expand-BuildMatrix([string] $scope)
|
|
{
|
|
$entries = @()
|
|
foreach ($buildProfile in $BuildProfiles) {
|
|
if ($buildProfile.scopes -notcontains $scope) {
|
|
continue
|
|
}
|
|
|
|
$profileSolutions = $buildProfile.solutions
|
|
if ($profileSolutions -eq '*') {
|
|
$profileSolutions = $solutions
|
|
}
|
|
foreach ($solution in $profileSolutions) {
|
|
if ($solutions -notcontains $solution) {
|
|
throw "Build profile '$($buildProfile.id)' names solution '$solution', which was not discovered."
|
|
}
|
|
$stem = [System.IO.Path]::GetFileNameWithoutExtension($solution)
|
|
foreach ($tfm in $buildProfile.tfms) {
|
|
foreach ($configuration in $buildProfile.configurations) {
|
|
$entries += [ordered]@{
|
|
id = "$($buildProfile.id)-$stem-$tfm-$configuration"
|
|
name = "$stem $tfm $configuration ($($buildProfile.os))"
|
|
os = $buildProfile.os
|
|
runsOn = $RunnerImages[$buildProfile.os]
|
|
arch = $RunnerArchitectures[$buildProfile.os]
|
|
solution = $solution
|
|
customTestTarget = $tfm
|
|
configuration = $configuration
|
|
dotnet = (Get-SdkVersions $tfm) -join "`n"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
return $entries
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Packs an ordered project list into contiguous batches whose weights add up
|
|
to at most $targetMinutes, never more than $MaxProjectsPerBatch projects.
|
|
|
|
.DESCRIPTION
|
|
Contiguous rather than best-fit: path order keeps projects that share
|
|
dependencies together, and the result only depends on the list and the
|
|
table, so a run is reproducible. A project heavier than the target is
|
|
placed in a batch of its own rather than split or dropped.
|
|
#>
|
|
function Split-IntoBatches([string[]] $projects, [int] $targetMinutes)
|
|
{
|
|
$batches = @()
|
|
$current = @()
|
|
$currentMinutes = 0
|
|
foreach ($project in $projects) {
|
|
$minutes = Get-ProjectMinutes $project
|
|
$stem = [System.IO.Path]::GetFileNameWithoutExtension($project)
|
|
if ($stem -eq 'Opc.Ua.WotCon.Samples.Tests' -or $DedicatedTestProjects -contains $stem) {
|
|
if ($current.Count -gt 0) {
|
|
$batches += , @($current)
|
|
$current = @()
|
|
$currentMinutes = 0
|
|
}
|
|
$batches += , @($project)
|
|
continue
|
|
}
|
|
if ($current.Count -gt 0 -and
|
|
($currentMinutes + $minutes -gt $targetMinutes -or $current.Count -ge $MaxProjectsPerBatch)) {
|
|
$batches += , @($current)
|
|
$current = @()
|
|
$currentMinutes = 0
|
|
}
|
|
$current += $project
|
|
$currentMinutes += $minutes
|
|
}
|
|
if ($current.Count -gt 0) {
|
|
$batches += , @($current)
|
|
}
|
|
return $batches
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Returns exhaustive, non-overlapping filters for projects whose full test
|
|
suite exceeds the per-project execution budget.
|
|
#>
|
|
function Get-TestProjectShards([string] $project, [string] $framework)
|
|
{
|
|
$stem = [System.IO.Path]::GetFileNameWithoutExtension($project)
|
|
switch ($stem) {
|
|
'Opc.Ua.WotCon.Samples.Tests' {
|
|
if ($framework -eq 'net48') {
|
|
return @(
|
|
@{
|
|
id = 'all'
|
|
name = 'all'
|
|
filter = ''
|
|
hangTimeout = ''
|
|
}
|
|
)
|
|
}
|
|
$prefix = 'Opc.Ua.WotCon.Samples.Tests.'
|
|
$generationScenario = "${prefix}WotSampleEndToEndTests.RealSamplesAggregateSubscribeAndReplaceGenerationAsync"
|
|
$pumpComparison = "${prefix}WotPumpAddressSpaceComparisonTests.WotPumpInstanceMatchesNativePumpSubsetAsync"
|
|
$nativeTypes = "${prefix}WotPumpAddressSpaceComparisonTests.CompanionTypeDefinitionsMatchNativePumpServerAsync"
|
|
return @(
|
|
@{
|
|
id = 'sample-core'
|
|
name = 'sample-core'
|
|
filter = @(
|
|
"FullyQualifiedName!~$generationScenario"
|
|
"FullyQualifiedName!~$pumpComparison"
|
|
"FullyQualifiedName!~$nativeTypes"
|
|
) -join '&'
|
|
hangTimeout = ''
|
|
}
|
|
@{
|
|
id = 'sample-generation'
|
|
name = 'sample-generation'
|
|
filter = "FullyQualifiedName~$generationScenario"
|
|
hangTimeout = '45m'
|
|
}
|
|
@{
|
|
id = 'sample-native-comparisons'
|
|
name = 'sample-native-comparisons'
|
|
filter = "(FullyQualifiedName~$pumpComparison|FullyQualifiedName~$nativeTypes)"
|
|
hangTimeout = '30m'
|
|
}
|
|
)
|
|
}
|
|
'Opc.Ua.WotCon.Tests' {
|
|
$materialization = 'Opc.Ua.WotCon.Tests.Materialization.'
|
|
if ($framework -ne 'net48') {
|
|
return @(
|
|
@{
|
|
id = 'wot-materialization'
|
|
name = 'wot-materialization'
|
|
filter = "FullyQualifiedName~${materialization}"
|
|
hangTimeout = ''
|
|
}
|
|
@{
|
|
id = 'wot-other'
|
|
name = 'wot-other'
|
|
filter = "FullyQualifiedName!~${materialization}"
|
|
hangTimeout = ''
|
|
}
|
|
)
|
|
}
|
|
return @(
|
|
@{
|
|
id = 'wot-materialization-core'
|
|
name = 'wot-materialization-core'
|
|
filter = "FullyQualifiedName~${materialization}&TestCategory!=Integration"
|
|
hangTimeout = ''
|
|
}
|
|
@{
|
|
id = 'wot-materialization-integration'
|
|
name = 'wot-materialization-integration'
|
|
filter = "FullyQualifiedName~${materialization}&TestCategory=Integration"
|
|
hangTimeout = ''
|
|
}
|
|
@{
|
|
id = 'wot-other-core'
|
|
name = 'wot-other-core'
|
|
filter = "FullyQualifiedName!~${materialization}&TestCategory!=Integration"
|
|
hangTimeout = ''
|
|
}
|
|
@{
|
|
id = 'wot-other-integration'
|
|
name = 'wot-other-integration'
|
|
filter = "FullyQualifiedName!~${materialization}&TestCategory=Integration"
|
|
hangTimeout = ''
|
|
}
|
|
)
|
|
}
|
|
'Opc.Ua.Server.Tests' {
|
|
if ($framework -ne 'net48') {
|
|
return @(
|
|
@{
|
|
id = 'all'
|
|
name = 'all'
|
|
filter = ''
|
|
hangTimeout = ''
|
|
}
|
|
)
|
|
}
|
|
|
|
$groups = @(
|
|
@{
|
|
id = 'server-node-management'
|
|
name = 'server-node-management'
|
|
terms = @('NodeManager', 'SamplingGroup', 'SamplingInterval', 'NodeState', 'MonitoredNode')
|
|
}
|
|
@{
|
|
id = 'server-historian'
|
|
name = 'server-historian'
|
|
terms = @('Historian', 'Aggregate', 'History')
|
|
}
|
|
@{
|
|
id = 'server-subscriptions'
|
|
name = 'server-subscriptions'
|
|
terms = @('Subscription', 'MonitoredItem', 'Publish', 'KeepAlive')
|
|
}
|
|
@{
|
|
id = 'server-sessions'
|
|
name = 'server-sessions'
|
|
terms = @('Session', 'Identity', 'UserManagement', 'UserDatabase', 'Role', 'Authentication')
|
|
}
|
|
@{
|
|
id = 'server-configuration'
|
|
name = 'server-configuration'
|
|
terms = @('Configuration', 'Certificate', 'TrustList', 'KeyCredential', 'Security')
|
|
}
|
|
@{
|
|
id = 'server-runtime'
|
|
name = 'server-runtime'
|
|
terms = @('Fluent', 'Hosting', 'RuntimeNodeSet', 'NodeSet', 'StateMachine',
|
|
'ComplexType', 'FileSystem', 'AliasNames', 'Program')
|
|
}
|
|
@{
|
|
id = 'server-transport'
|
|
name = 'server-transport'
|
|
terms = @('NativeTcp', 'ReverseConnect', 'Transport', 'ReferenceServer', 'Discovery',
|
|
'Quickstart')
|
|
}
|
|
@{
|
|
id = 'server-other'
|
|
name = 'server-other'
|
|
terms = @()
|
|
}
|
|
)
|
|
|
|
$assignedTerms = @()
|
|
foreach ($group in $groups) {
|
|
$filterTerms = @(
|
|
$assignedTerms | ForEach-Object { "FullyQualifiedName!~$_" }
|
|
)
|
|
if ($group.terms.Count -gt 0) {
|
|
$groupFilter = @(
|
|
$group.terms | ForEach-Object { "FullyQualifiedName~$_" }
|
|
) -join '|'
|
|
$filterTerms += "($groupFilter)"
|
|
}
|
|
$group.filter = $filterTerms -join '&'
|
|
$group.hangTimeout = ''
|
|
$assignedTerms += $group.terms
|
|
}
|
|
return $groups
|
|
}
|
|
default {
|
|
return @(
|
|
@{
|
|
id = 'all'
|
|
name = 'all'
|
|
filter = ''
|
|
hangTimeout = ''
|
|
}
|
|
)
|
|
}
|
|
}
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Counts the test matrix entries a batch target produces for a scope, ignoring
|
|
which optional runners are enabled so the target is stable.
|
|
#>
|
|
function Measure-TestEntries([string] $scope, [int] $targetMinutes)
|
|
{
|
|
$total = 0
|
|
foreach ($testProfile in $Profiles) {
|
|
if ($testProfile.scopes -notcontains $scope) {
|
|
continue
|
|
}
|
|
$tier = 'mainline'
|
|
if ($testProfile.ContainsKey('tier')) {
|
|
$tier = $testProfile.tier
|
|
}
|
|
foreach ($batch in @(Split-IntoBatches $projectsByTier[$tier] $targetMinutes)) {
|
|
if ($batch.Count -eq 1) {
|
|
$total += @(Get-TestProjectShards $batch[0] $testProfile.framework).Count
|
|
}
|
|
else {
|
|
$total++
|
|
}
|
|
}
|
|
}
|
|
return [int]$total
|
|
}
|
|
|
|
<#
|
|
.SYNOPSIS
|
|
Expands the test matrix and the per-tuple manifest for a scope.
|
|
#>
|
|
function Expand-TestMatrix([string] $scope, [int] $targetMinutes, [bool] $includeMacOS)
|
|
{
|
|
$entries = @()
|
|
$manifest = @()
|
|
foreach ($testProfile in $Profiles) {
|
|
if ($testProfile.scopes -notcontains $scope) {
|
|
continue
|
|
}
|
|
if ($testProfile.os -eq 'macos' -and -not $includeMacOS) {
|
|
continue
|
|
}
|
|
|
|
$tier = 'mainline'
|
|
if ($testProfile.ContainsKey('tier')) {
|
|
$tier = $testProfile.tier
|
|
}
|
|
$filter = $DefaultFilter
|
|
if ($testProfile.ContainsKey('filter')) {
|
|
$filter = $testProfile.filter
|
|
}
|
|
$hangTimeout = '10m'
|
|
if ($testProfile.ContainsKey('hangTimeout')) {
|
|
$hangTimeout = $testProfile.hangTimeout
|
|
}
|
|
# coverlet.collector 10.x ships build assets for net8.0 and newer only, so
|
|
# a .NET Framework test host cannot load the 'XPlat Code Coverage'
|
|
# collector at all - VSTest only warns and writes no report. The
|
|
# long-running and durable tiers opt out explicitly:
|
|
# they re-run projects the filtered legs already covered, so folding their
|
|
# numbers into the merged report would double-count them.
|
|
$coverage = -not $testProfile.framework.StartsWith('net4')
|
|
if ($testProfile.ContainsKey('coverage')) {
|
|
$coverage = [bool]$testProfile.coverage
|
|
}
|
|
# Each project gets one test-execution ceiling; the job timeout also
|
|
# includes fixed setup/build overhead.
|
|
$perProjectTimeout = $PerProjectTimeoutMinutes
|
|
|
|
$batches = @(Split-IntoBatches $projectsByTier[$tier] $targetMinutes)
|
|
for ($index = 0; $index -lt $batches.Count; $index++) {
|
|
$batch = $batches[$index]
|
|
$batchHangTimeout = $hangTimeout
|
|
$shards = @(@{
|
|
id = 'all'
|
|
name = 'all'
|
|
filter = ''
|
|
hangTimeout = ''
|
|
})
|
|
if ($batch.Count -eq 1) {
|
|
$shards = @(Get-TestProjectShards $batch[0] $testProfile.framework)
|
|
}
|
|
foreach ($shard in $shards) {
|
|
$batchId = "$($testProfile.id)-$(($index + 1).ToString('00'))-$($shard.id)"
|
|
$batchTimeout = $perProjectTimeout
|
|
$batchFilter = $filter
|
|
if (-not [string]::IsNullOrWhiteSpace($shard.filter)) {
|
|
if ([string]::IsNullOrWhiteSpace($batchFilter)) {
|
|
$batchFilter = $shard.filter
|
|
}
|
|
else {
|
|
$batchFilter = "$batchFilter&$($shard.filter)"
|
|
}
|
|
}
|
|
$shardHangTimeout = $batchHangTimeout
|
|
if (-not [string]::IsNullOrWhiteSpace($shard.hangTimeout)) {
|
|
$shardHangTimeout = $shard.hangTimeout
|
|
}
|
|
# Twenty minutes of fixed cost - checkout, SDK install, restore,
|
|
# and builds - plus the test-execution ceiling for every project
|
|
# in the batch. This leaves each test its full budget while
|
|
# keeping the job timeout a true upper bound.
|
|
$jobTimeout = 20 + ($batch.Count * $batchTimeout)
|
|
if ($jobTimeout -gt $MaxJobTimeoutMinutes) {
|
|
throw ("Batch '$batchId' of $($batch.Count) projects needs $jobTimeout minutes, past the " +
|
|
"$MaxJobTimeoutMinutes-minute ceiling. Clamping it would let the executor outlive the " +
|
|
'job, so lower the per-project ceiling or raise ReservedEntries so smaller batches fit.')
|
|
}
|
|
$name = "$($testProfile.id) ($($index + 1)/$($batches.Count))"
|
|
if ($shard.id -ne 'all') {
|
|
$name = "$name - $($shard.name)"
|
|
}
|
|
$entries += [ordered]@{
|
|
id = $batchId
|
|
name = $name
|
|
os = $testProfile.os
|
|
runsOn = $RunnerImages[$testProfile.os]
|
|
arch = $RunnerArchitectures[$testProfile.os]
|
|
profile = $testProfile.id
|
|
tier = $tier
|
|
shard = $shard.name
|
|
customTestTarget = $testProfile.customTestTarget
|
|
framework = $testProfile.framework
|
|
configuration = $testProfile.configuration
|
|
filter = $batchFilter
|
|
hangTimeout = $shardHangTimeout
|
|
coverage = $coverage
|
|
projects = $batch -join ';'
|
|
perProjectTimeout = $batchTimeout
|
|
timeoutMinutes = $jobTimeout
|
|
# The batch's summed weight from the duration table: what the
|
|
# packer balanced, on top of the fixed per-batch cost.
|
|
packedMinutes = [int]($batch | ForEach-Object { Get-ProjectMinutes $_ } | Measure-Object -Sum).Sum
|
|
dotnet = (Get-SdkVersions $testProfile.framework) -join "`n"
|
|
}
|
|
|
|
foreach ($project in $batch) {
|
|
# Every automatic entry uses the corpus checked into the
|
|
# repository. The private crash corpus is only ever replayed by
|
|
# the protected job in nightly.yml.
|
|
$corpus = 'none'
|
|
if ($project -like 'fuzzing/*') {
|
|
$corpus = 'public-checked-in'
|
|
}
|
|
$manifest += [ordered]@{
|
|
project = $project
|
|
profile = $testProfile.id
|
|
tier = $tier
|
|
os = $testProfile.os
|
|
runsOn = $RunnerImages[$testProfile.os]
|
|
arch = $RunnerArchitectures[$testProfile.os]
|
|
customTestTarget = $testProfile.customTestTarget
|
|
framework = $testProfile.framework
|
|
configuration = $testProfile.configuration
|
|
filter = $batchFilter
|
|
shard = $shard.name
|
|
coverage = $coverage
|
|
corpus = $corpus
|
|
batch = $batchId
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
return @{ entries = $entries; manifest = $manifest }
|
|
}
|
|
|
|
$buildEntries = @(Expand-BuildMatrix $Scope)
|
|
$budget = $MaxEntries - $buildEntries.Count - $ReservedEntries
|
|
if ($budget -lt 1) {
|
|
throw "The '$Scope' solution build matrix alone needs $($buildEntries.Count) of the $MaxEntries entries GitHub Actions allows per workflow run, leaving no room for tests."
|
|
}
|
|
|
|
# Raising the target shrinks the matrix until it fits. Once every batch is
|
|
# already at the per-batch project cap, raising it further changes nothing, so
|
|
# that is reported rather than looped on.
|
|
$heaviestProject = (@($ProjectMinutes.Values) + $DefaultProjectMinutes | Measure-Object -Maximum).Maximum
|
|
$targetCeiling = $MaxProjectsPerBatch * $heaviestProject
|
|
$targetMinutes = $BatchTargetMinutes
|
|
while ((Measure-TestEntries $Scope $targetMinutes) -gt $budget) {
|
|
if ($targetMinutes -ge $targetCeiling) {
|
|
throw ("No batch target fits the '$Scope' test workload into $budget matrix entries; " +
|
|
"$MaxProjectsPerBatch projects per batch is the most MaxJobTimeoutMinutes allows.")
|
|
}
|
|
$targetMinutes += 5
|
|
}
|
|
|
|
$expanded = Expand-TestMatrix $Scope $targetMinutes (-not $ExcludeMacOS)
|
|
$testEntries = @($expanded.entries)
|
|
$totalEntries = $testEntries.Count + $buildEntries.Count
|
|
$largestBatch = ($testEntries | ForEach-Object { @($_.projects -split ';').Count } | Measure-Object -Maximum).Maximum
|
|
|
|
Write-Host "Scope: $Scope"
|
|
Write-Host "Repository root: $RepositoryRoot"
|
|
Write-Host "Mainline projects: $($mainlineProjects.Count)"
|
|
Write-Host "Solutions: $($solutions -join ', ')"
|
|
Write-Host "Batch target: $targetMinutes min (+$BatchFixedMinutes fixed), at most $MaxProjectsPerBatch projects"
|
|
Write-Host "Largest batch: $largestBatch projects"
|
|
Write-Host "Test entries: $($testEntries.Count)"
|
|
Write-Host "Build entries: $($buildEntries.Count)"
|
|
Write-Host "Reserved entries: $ReservedEntries"
|
|
Write-Host "Total matrix jobs: $totalEntries (limit $MaxEntries)"
|
|
|
|
if ($totalEntries -gt $MaxEntries) {
|
|
throw "The '$Scope' workload expands to $totalEntries matrix jobs, past the $MaxEntries GitHub Actions allows per workflow run. Raise BatchTargetMinutes so fewer, larger batches are chosen rather than dropping entries."
|
|
}
|
|
|
|
if (-not [string]::IsNullOrWhiteSpace($ManifestPath)) {
|
|
$manifestDirectory = Split-Path -Parent $ManifestPath
|
|
if (-not [string]::IsNullOrWhiteSpace($manifestDirectory) -and -not (Test-Path -LiteralPath $manifestDirectory)) {
|
|
$null = New-Item -ItemType Directory -Path $manifestDirectory -Force
|
|
}
|
|
$document = [ordered]@{
|
|
scope = $Scope
|
|
batchTargetMinutes = $targetMinutes
|
|
includeMacOS = (-not $ExcludeMacOS)
|
|
builds = $buildEntries
|
|
tests = @($expanded.manifest)
|
|
}
|
|
$document | ConvertTo-Json -Depth 6 | Out-File -LiteralPath $ManifestPath -Encoding utf8
|
|
Write-Host "Wrote the workload manifest to '$ManifestPath'."
|
|
}
|
|
|
|
$testsJson = ConvertTo-Json -InputObject $testEntries -Depth 5 -Compress
|
|
$buildsJson = ConvertTo-Json -InputObject $buildEntries -Depth 5 -Compress
|
|
|
|
if (-not [string]::IsNullOrWhiteSpace($env:GITHUB_OUTPUT)) {
|
|
"tests=$testsJson" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
|
"builds=$buildsJson" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
|
"batch_target_minutes=$targetMinutes" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
|
"test_entry_count=$($testEntries.Count)" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
|
"build_entry_count=$($buildEntries.Count)" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
|
}
|
|
else {
|
|
Write-Output $testsJson
|
|
Write-Output $buildsJson
|
|
}
|