mirror of
https://github.com/OPCFoundation/UA-.NETStandard.git
synced 2026-10-17 05:11:13 +08:00
Cut CI runner minutes: duration-packed test batches, CodeQL on Linux/daily, cancellable branch packing (#4694)
Reduces GitHub Actions runner minutes without removing any OS, target framework, test project or check. All figures come from master CI run 37520359929 (158 jobs, ~1,880 runner-minutes) and the last two weeks of run history. ## 1. Test batches packed by duration (`get-ci-matrix.ps1`) Previously the expander picked the **smallest** batch size that fit the 256-job limit, which meant 2 projects per job and 33 jobs per profile. Job logs show ~45% of the test matrix (674 of ~1,516 min) went to compiling. Each batch rebuilds the shared dependency graph: the first project's build took 3.8–4.9 min, the second only 0.4–0.9 min. - Projects are now packed in path order (which keeps `PubSub.*`, `Redundancy.*`, `WotCon.*` together) up to a 30-minute weight target, using `.github/ci-test-durations.json`. Each weight is the slowest PR profile's test minutes, rounded up, plus 1. - A batch never holds more projects than the job timeout can budget for: at most 7, given 20 + n×45 ≤ 350. - If the matrix would exceed the job limit, the target is raised instead. - **PR test matrix: 132 → 60 jobs** (15 per profile; macOS 33 → 15). Each batch is estimated at ≤36 min on the slowest runner, comparable to today's slowest jobs (37–43 min), so wall-clock should be about the same. Expected saving is roughly 300+ runner-min per run, plus less queueing for the scarce macOS runners. - The weekly full scope shrinks the same way (still 13 profiles). - The table only steers balance: a project missing from it gets `defaultMinutes`, so it is still scheduled. - `run-dotnet-tests.ps1` now records per-project `buildSeconds`/`testSeconds` (batch summary and job summary). The new `update-test-durations.ps1` refreshes the table from any run's `dotnet-results-*` artifacts. - `discover` output `batch_size` is renamed to `batch_target_minutes`. Nothing consumed it. New `CiMatrixScriptTests` cases: - each profile runs each of its projects exactly once; - only a lone project may exceed the target; - PR profiles average ≥3 projects per job; - the duration table names only existing projects. ## 2 + 3. CodeQL (`codeql-analysis.yml`) - **No longer runs on every push to master.** Instead, a daily schedule on master (was weekly) gives an analysis that always completes. Each analysis takes ~3 h, and with `cancel-in-progress: true` per branch, 49 of 76 push runs in two weeks were cancelled by the next merge before uploading anything. `master378`/`release/*`/`develop/*` keep their push trigger. PRs are still analyzed as before. - Only PR runs are cancelled when superseded. Push and scheduled runs always finish. - **Runs on `ubuntu-latest` instead of `windows-2025`.** Only net10.0 is built, so the extracted sources are identical. The traced Windows build alone took 70 min, against 12 min for the untraced Linux `UA.slnx` net10.0 leg. This PR touches a `.cs` file, so its own CodeQL run is the first measurement. ## 4. NuGet packing on feature branches (`nuget-publish.yml`) 298 of ~400 runs in two weeks were feature-branch pushes, each about 59 Windows-min across two signed builds. - A superseded feature-branch run is now cancelled. Protected lines and tags are never cancelled: they publish previews, and `release.yml` promotes a run by ID. Each manual (`workflow_dispatch`) run gets its own concurrency group, so a later push cannot cancel it. - A new `changes` job skips packing when a feature branch, as a whole, changes only documentation. It diffs against the merge base with master, not the previous push, so a docs-only push that cancels a code push cannot mask that code change. It uses the same `get-path-relevance.ps1` rule as CI. - Protected refs, tags, manual runs, and any branch whose merge base cannot be resolved always pack. Checked locally for docs-only branch (skip), code branch, protected ref, manual run and unknown commit (all pack). ## Verification - `CiMatrixScriptTests`, `CiPathRelevanceTests` and the new `CiTestDurationsScriptTests` (covering `update-test-durations.ps1`): all `*.Ci*` tests in `Opc.Ua.Tools.Tests` pass locally, 63/63. - This PR's own CI run shows the new matrix. Compare its total runner-minutes and longest test job against run 37520359929. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
romanett
@@ -18,20 +18,30 @@
|
||||
a single target framework (see targets.props). A profile carries both the
|
||||
CustomTestTarget it builds with and the framework its tests actually run on.
|
||||
|
||||
GitHub Actions refuses to start a workflow run whose matrices expand past 256
|
||||
jobs, and that limit cannot be raised. The project fan-out is therefore
|
||||
batched: each matrix entry carries several projects that the executor runs in
|
||||
sequence, keeping per-project results. The batch size is the smallest one
|
||||
that fits the budget, so it adapts as test projects are added instead of
|
||||
silently truncating the matrix.
|
||||
The project fan-out is batched: each matrix entry carries several projects
|
||||
that the executor runs in sequence, keeping per-project results. Batches are
|
||||
sized by measured duration rather than by project count. Every batch pays a
|
||||
fixed cost of several minutes - checkout, SDK install, and the cold build of
|
||||
the shared dependency graph - while a further project in the same batch only
|
||||
adds its incremental build and its tests. Two projects per job spent almost
|
||||
half of the pull-request matrix compiling the same assemblies over and over,
|
||||
so projects are packed, in path order, into batches of about
|
||||
BatchTargetMinutes each. Path order keeps related projects (PubSub.*,
|
||||
Redundancy.*, WotCon.*) together, so their shared dependencies are built
|
||||
once per batch.
|
||||
|
||||
GitHub Actions also refuses to start a workflow run whose matrices expand
|
||||
past 256 jobs, and that limit cannot be raised. If the packed matrix does not
|
||||
fit, the target is raised until it does, so the matrix adapts as test
|
||||
projects are added instead of silently truncating.
|
||||
|
||||
.PARAMETER Scope
|
||||
'pr' expands the workload that gates a pull request. 'full' expands the
|
||||
complete workload Azure Pipelines used to run on a schedule.
|
||||
|
||||
.PARAMETER ExcludeMacOS
|
||||
Drop the macOS test profile. The batch size is deliberately computed from the
|
||||
full profile table regardless of this switch, so excluding macOS does not
|
||||
Drop the macOS test profile. The batch target is deliberately computed from
|
||||
the full profile table regardless of this switch, so excluding macOS does not
|
||||
reshuffle the batches of the remaining profiles.
|
||||
|
||||
.PARAMETER OnlyProject
|
||||
@@ -55,7 +65,22 @@
|
||||
GitHub enforces on hosted runners. A batch whose per-project budgets add up
|
||||
past this is an error rather than a clamp: clamping would let the executor
|
||||
outlive the job, which is the one case that produces no annotation and no
|
||||
results.
|
||||
results. The packer therefore never puts more projects in one batch than
|
||||
this ceiling can budget for.
|
||||
|
||||
.PARAMETER BatchTargetMinutes
|
||||
The measured duration, in minutes, a batch is packed up to, on top of its
|
||||
fixed cost. Weights come from DurationsPath and are the slowest pull-request
|
||||
profile's figures, so batches on the faster runners finish sooner. A project
|
||||
heavier than the target gets a batch of its own. Raised in steps of five
|
||||
when the packed matrix would not fit MaxEntries.
|
||||
|
||||
.PARAMETER DurationsPath
|
||||
JSON table of per-project weights in minutes. Defaults to
|
||||
.github/ci-test-durations.json. A project missing from it is weighted with
|
||||
the table's defaultMinutes, so a new project is still scheduled; the table
|
||||
only steers balance, never what runs. Regenerate it with
|
||||
.github/scripts/update-test-durations.ps1.
|
||||
|
||||
.PARAMETER RepositoryRoot
|
||||
Repository root. Defaults to the root two levels above this script.
|
||||
@@ -74,6 +99,8 @@ Param(
|
||||
[int] $MaxEntries = 256,
|
||||
[int] $ReservedEntries = 40,
|
||||
[int] $MaxJobTimeoutMinutes = 350,
|
||||
[int] $BatchTargetMinutes = 30,
|
||||
[string] $DurationsPath = '',
|
||||
[string] $RepositoryRoot = '',
|
||||
[string] $ManifestPath = ''
|
||||
)
|
||||
@@ -86,6 +113,12 @@ if ([string]::IsNullOrWhiteSpace($RepositoryRoot)) {
|
||||
else {
|
||||
$RepositoryRoot = (Resolve-Path -LiteralPath $RepositoryRoot).Path
|
||||
}
|
||||
if ([string]::IsNullOrWhiteSpace($DurationsPath)) {
|
||||
$DurationsPath = Join-Path $RepositoryRoot '.github' 'ci-test-durations.json'
|
||||
}
|
||||
if ($BatchTargetMinutes -lt 1) {
|
||||
throw "BatchTargetMinutes must be positive, not $BatchTargetMinutes."
|
||||
}
|
||||
|
||||
# Runner images: GitHub-hosted labels.
|
||||
$RunnerImages = @{
|
||||
@@ -227,6 +260,56 @@ $BuildProfiles = @(
|
||||
}
|
||||
)
|
||||
|
||||
# Every batch carries this much on top of its packed weight: checkout, SDK
|
||||
# install, restore, and the cold build of the dependency graph its first
|
||||
# project pulls in. Run 37520359929 measured 3.8-4.9 minutes for that first
|
||||
# build against 0.4-0.9 minutes for each further project in the same batch.
|
||||
$BatchFixedMinutes = 6
|
||||
|
||||
# One combined ceiling per project, shared by that project's build and test
|
||||
# invocation in run-dotnet-tests.ps1. The job timeout budgets it exactly once
|
||||
# per project, so the executor must not spend it twice. Keep 45 minutes for
|
||||
# mainline too: on hosted Windows, the net48 Opc.Ua.Server.Tests build plus
|
||||
# its otherwise healthy test run normally takes about 25 minutes and has
|
||||
# exceeded 30 under runner variance.
|
||||
$PerProjectTimeoutMinutes = 45
|
||||
|
||||
# 20 minutes of fixed cost plus one per-project ceiling for each project has to
|
||||
# stay below MaxJobTimeoutMinutes, which caps how many projects one batch takes.
|
||||
$MaxProjectsPerBatch = [System.Math]::Floor(($MaxJobTimeoutMinutes - 20) / $PerProjectTimeoutMinutes)
|
||||
if ($MaxProjectsPerBatch -lt 1) {
|
||||
throw "MaxJobTimeoutMinutes=$MaxJobTimeoutMinutes cannot budget even one $PerProjectTimeoutMinutes-minute project."
|
||||
}
|
||||
|
||||
if (-not (Test-Path -LiteralPath $DurationsPath -PathType Leaf)) {
|
||||
throw "The per-project duration table '$DurationsPath' does not exist."
|
||||
}
|
||||
$durationTable = Get-Content -LiteralPath $DurationsPath -Raw | ConvertFrom-Json -AsHashtable
|
||||
$DefaultProjectMinutes = [int]$durationTable.defaultMinutes
|
||||
if ($DefaultProjectMinutes -lt 1) {
|
||||
throw "'$DurationsPath' must declare a positive defaultMinutes."
|
||||
}
|
||||
$ProjectMinutes = @{}
|
||||
foreach ($entry in $durationTable.projects.GetEnumerator()) {
|
||||
if ([int]$entry.Value -lt 1) {
|
||||
throw "'$DurationsPath' gives '$($entry.Key)' a weight of $($entry.Value); every weight must be at least one minute."
|
||||
}
|
||||
$ProjectMinutes[$entry.Key] = [int]$entry.Value
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Returns the packing weight of a project, in minutes.
|
||||
#>
|
||||
function Get-ProjectMinutes([string] $project)
|
||||
{
|
||||
$stem = [System.IO.Path]::GetFileNameWithoutExtension($project)
|
||||
if ($ProjectMinutes.ContainsKey($stem)) {
|
||||
return $ProjectMinutes[$stem]
|
||||
}
|
||||
return $DefaultProjectMinutes
|
||||
}
|
||||
|
||||
$allTestProjects = Find-RepositoryFile '*.Tests.csproj'
|
||||
$mainlineProjects = @($allTestProjects | Where-Object { $MainlineExclusions -notcontains (Split-Path $_ -Leaf) })
|
||||
$durableProjects = @($allTestProjects | Where-Object { (Split-Path $_ -Leaf) -eq $DurableProject })
|
||||
@@ -302,24 +385,43 @@ function Expand-BuildMatrix([string] $scope)
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Splits an ordered project list into contiguous batches of at most $size.
|
||||
Packs an ordered project list into contiguous batches whose weights add up
|
||||
to at most $targetMinutes, never more than $MaxProjectsPerBatch projects.
|
||||
|
||||
.DESCRIPTION
|
||||
Contiguous rather than best-fit: path order keeps projects that share
|
||||
dependencies together, and the result only depends on the list and the
|
||||
table, so a run is reproducible. A project heavier than the target is
|
||||
placed in a batch of its own rather than split or dropped.
|
||||
#>
|
||||
function Split-IntoBatches([string[]] $projects, [int] $size)
|
||||
function Split-IntoBatches([string[]] $projects, [int] $targetMinutes)
|
||||
{
|
||||
$batches = @()
|
||||
for ($index = 0; $index -lt $projects.Count; $index += $size) {
|
||||
$take = [System.Math]::Min($size, $projects.Count - $index)
|
||||
$batches += , @($projects[$index..($index + $take - 1)])
|
||||
$current = @()
|
||||
$currentMinutes = 0
|
||||
foreach ($project in $projects) {
|
||||
$minutes = Get-ProjectMinutes $project
|
||||
if ($current.Count -gt 0 -and
|
||||
($currentMinutes + $minutes -gt $targetMinutes -or $current.Count -ge $MaxProjectsPerBatch)) {
|
||||
$batches += , @($current)
|
||||
$current = @()
|
||||
$currentMinutes = 0
|
||||
}
|
||||
$current += $project
|
||||
$currentMinutes += $minutes
|
||||
}
|
||||
if ($current.Count -gt 0) {
|
||||
$batches += , @($current)
|
||||
}
|
||||
return $batches
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Counts the test matrix entries a batch size produces for a scope, ignoring
|
||||
which optional runners are enabled so the batch size is stable.
|
||||
Counts the test matrix entries a batch target produces for a scope, ignoring
|
||||
which optional runners are enabled so the target is stable.
|
||||
#>
|
||||
function Measure-TestEntries([string] $scope, [int] $size)
|
||||
function Measure-TestEntries([string] $scope, [int] $targetMinutes)
|
||||
{
|
||||
$total = 0
|
||||
foreach ($testProfile in $Profiles) {
|
||||
@@ -330,7 +432,7 @@ function Measure-TestEntries([string] $scope, [int] $size)
|
||||
if ($testProfile.ContainsKey('tier')) {
|
||||
$tier = $testProfile.tier
|
||||
}
|
||||
$total += [System.Math]::Ceiling($projectsByTier[$tier].Count / [double]$size)
|
||||
$total += @(Split-IntoBatches $projectsByTier[$tier] $targetMinutes).Count
|
||||
}
|
||||
return [int]$total
|
||||
}
|
||||
@@ -339,7 +441,7 @@ function Measure-TestEntries([string] $scope, [int] $size)
|
||||
.SYNOPSIS
|
||||
Expands the test matrix and the per-tuple manifest for a scope.
|
||||
#>
|
||||
function Expand-TestMatrix([string] $scope, [int] $size, [bool] $includeMacOS)
|
||||
function Expand-TestMatrix([string] $scope, [int] $targetMinutes, [bool] $includeMacOS)
|
||||
{
|
||||
$entries = @()
|
||||
$manifest = @()
|
||||
@@ -373,15 +475,10 @@ function Expand-TestMatrix([string] $scope, [int] $size, [bool] $includeMacOS)
|
||||
if ($testProfile.ContainsKey('coverage')) {
|
||||
$coverage = [bool]$testProfile.coverage
|
||||
}
|
||||
# One combined ceiling per project, shared by that project's build and
|
||||
# test invocation in run-dotnet-tests.ps1. The job timeout below budgets
|
||||
# it exactly once per project, so the executor must not spend it twice.
|
||||
# Keep 45 minutes for mainline too: on hosted Windows, the net48
|
||||
# Opc.Ua.Server.Tests build plus its otherwise healthy test run normally
|
||||
# takes about 25 minutes and has exceeded 30 under runner variance.
|
||||
$perProjectTimeout = 45
|
||||
# See $PerProjectTimeoutMinutes for why this is one combined ceiling.
|
||||
$perProjectTimeout = $PerProjectTimeoutMinutes
|
||||
|
||||
$batches = Split-IntoBatches $projectsByTier[$tier] $size
|
||||
$batches = @(Split-IntoBatches $projectsByTier[$tier] $targetMinutes)
|
||||
for ($index = 0; $index -lt $batches.Count; $index++) {
|
||||
$batch = $batches[$index]
|
||||
$batchId = "$($testProfile.id)-$(($index + 1).ToString('00'))"
|
||||
@@ -413,6 +510,9 @@ function Expand-TestMatrix([string] $scope, [int] $size, [bool] $includeMacOS)
|
||||
projects = $batch -join ';'
|
||||
perProjectTimeout = $perProjectTimeout
|
||||
timeoutMinutes = $jobTimeout
|
||||
# The batch's summed weight from the duration table: what the
|
||||
# packer balanced, on top of the fixed per-batch cost.
|
||||
packedMinutes = [int]($batch | ForEach-Object { Get-ProjectMinutes $_ } | Measure-Object -Sum).Sum
|
||||
dotnet = (Get-SdkVersions $testProfile.framework) -join "`n"
|
||||
}
|
||||
|
||||
@@ -451,30 +551,38 @@ if ($budget -lt 1) {
|
||||
throw "The '$Scope' solution build matrix alone needs $($buildEntries.Count) of the $MaxEntries entries GitHub Actions allows per workflow run, leaving no room for tests."
|
||||
}
|
||||
|
||||
$batchSize = 1
|
||||
while ((Measure-TestEntries $Scope $batchSize) -gt $budget) {
|
||||
$batchSize++
|
||||
if ($batchSize -gt $mainlineProjects.Count) {
|
||||
throw "No batch size fits the '$Scope' test workload into $budget matrix entries."
|
||||
# Raising the target shrinks the matrix until it fits. Once every batch is
|
||||
# already at the per-batch project cap, raising it further changes nothing, so
|
||||
# that is reported rather than looped on.
|
||||
$heaviestProject = (@($ProjectMinutes.Values) + $DefaultProjectMinutes | Measure-Object -Maximum).Maximum
|
||||
$targetCeiling = $MaxProjectsPerBatch * $heaviestProject
|
||||
$targetMinutes = $BatchTargetMinutes
|
||||
while ((Measure-TestEntries $Scope $targetMinutes) -gt $budget) {
|
||||
if ($targetMinutes -ge $targetCeiling) {
|
||||
throw ("No batch target fits the '$Scope' test workload into $budget matrix entries; " +
|
||||
"$MaxProjectsPerBatch projects per batch is the most MaxJobTimeoutMinutes allows.")
|
||||
}
|
||||
$targetMinutes += 5
|
||||
}
|
||||
|
||||
$expanded = Expand-TestMatrix $Scope $batchSize (-not $ExcludeMacOS)
|
||||
$expanded = Expand-TestMatrix $Scope $targetMinutes (-not $ExcludeMacOS)
|
||||
$testEntries = @($expanded.entries)
|
||||
$totalEntries = $testEntries.Count + $buildEntries.Count
|
||||
$largestBatch = ($testEntries | ForEach-Object { @($_.projects -split ';').Count } | Measure-Object -Maximum).Maximum
|
||||
|
||||
Write-Host "Scope: $Scope"
|
||||
Write-Host "Repository root: $RepositoryRoot"
|
||||
Write-Host "Mainline projects: $($mainlineProjects.Count)"
|
||||
Write-Host "Solutions: $($solutions -join ', ')"
|
||||
Write-Host "Batch size: $batchSize"
|
||||
Write-Host "Batch target: $targetMinutes min (+$BatchFixedMinutes fixed), at most $MaxProjectsPerBatch projects"
|
||||
Write-Host "Largest batch: $largestBatch projects"
|
||||
Write-Host "Test entries: $($testEntries.Count)"
|
||||
Write-Host "Build entries: $($buildEntries.Count)"
|
||||
Write-Host "Reserved entries: $ReservedEntries"
|
||||
Write-Host "Total matrix jobs: $totalEntries (limit $MaxEntries)"
|
||||
|
||||
if ($totalEntries -gt $MaxEntries) {
|
||||
throw "The '$Scope' workload expands to $totalEntries matrix jobs, past the $MaxEntries GitHub Actions allows per workflow run. Lower the budget so a larger batch size is chosen rather than dropping entries."
|
||||
throw "The '$Scope' workload expands to $totalEntries matrix jobs, past the $MaxEntries GitHub Actions allows per workflow run. Raise BatchTargetMinutes so fewer, larger batches are chosen rather than dropping entries."
|
||||
}
|
||||
|
||||
if (-not [string]::IsNullOrWhiteSpace($ManifestPath)) {
|
||||
@@ -483,11 +591,11 @@ if (-not [string]::IsNullOrWhiteSpace($ManifestPath)) {
|
||||
$null = New-Item -ItemType Directory -Path $manifestDirectory -Force
|
||||
}
|
||||
$document = [ordered]@{
|
||||
scope = $Scope
|
||||
batchSize = $batchSize
|
||||
includeMacOS = (-not $ExcludeMacOS)
|
||||
builds = $buildEntries
|
||||
tests = @($expanded.manifest)
|
||||
scope = $Scope
|
||||
batchTargetMinutes = $targetMinutes
|
||||
includeMacOS = (-not $ExcludeMacOS)
|
||||
builds = $buildEntries
|
||||
tests = @($expanded.manifest)
|
||||
}
|
||||
$document | ConvertTo-Json -Depth 6 | Out-File -LiteralPath $ManifestPath -Encoding utf8
|
||||
Write-Host "Wrote the workload manifest to '$ManifestPath'."
|
||||
@@ -499,7 +607,7 @@ $buildsJson = ConvertTo-Json -InputObject $buildEntries -Depth 5 -Compress
|
||||
if (-not [string]::IsNullOrWhiteSpace($env:GITHUB_OUTPUT)) {
|
||||
"tests=$testsJson" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
||||
"builds=$buildsJson" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
||||
"batch_size=$batchSize" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
||||
"batch_target_minutes=$targetMinutes" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
||||
"test_entry_count=$($testEntries.Count)" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
||||
"build_entry_count=$($buildEntries.Count)" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
||||
}
|
||||
|
||||
@@ -302,6 +302,10 @@ foreach ($project in $projectList) {
|
||||
total = 0
|
||||
passed = 0
|
||||
failed = 0
|
||||
# Wall-clock seconds, read by update-test-durations.ps1 to refresh
|
||||
# the weights get-ci-matrix.ps1 packs batches with.
|
||||
buildSeconds = 0
|
||||
testSeconds = 0
|
||||
}
|
||||
|
||||
try {
|
||||
@@ -361,6 +365,7 @@ foreach ($project in $projectList) {
|
||||
$projectBudget = [System.Diagnostics.Stopwatch]::StartNew()
|
||||
|
||||
$build = Invoke-Dotnet $buildArguments $projectBudget $PerProjectTimeoutMinutes $logPath
|
||||
$record.buildSeconds = [int]$projectBudget.Elapsed.TotalSeconds
|
||||
if ($build.TimedOut) {
|
||||
throw ("The build exhausted the $PerProjectTimeoutMinutes-minute per-project ceiling, " +
|
||||
'which the build and the test share.')
|
||||
@@ -401,6 +406,7 @@ foreach ($project in $projectList) {
|
||||
}
|
||||
|
||||
$test = Invoke-Dotnet $testArguments $projectBudget $PerProjectTimeoutMinutes $logPath
|
||||
$record.testSeconds = [int]$projectBudget.Elapsed.TotalSeconds - $record.buildSeconds
|
||||
|
||||
$results = Measure-TestResults $projectResults
|
||||
$record.total = $results.Total
|
||||
@@ -457,11 +463,11 @@ $summaryPath = Join-Path $resultsRoot 'batch-summary.json'
|
||||
$lines = @(
|
||||
"### $CustomTestTarget / $Framework / $Configuration",
|
||||
'',
|
||||
'| Project | Outcome | Passed | Total | Detail |',
|
||||
'| --- | --- | ---: | ---: | --- |')
|
||||
'| Project | Outcome | Passed | Total | Build (s) | Test (s) | Detail |',
|
||||
'| --- | --- | ---: | ---: | ---: | ---: | --- |')
|
||||
foreach ($record in $records) {
|
||||
$detail = $record.reason -replace '\r?\n', ' '
|
||||
$lines += "| $([System.IO.Path]::GetFileNameWithoutExtension($record.project)) | $($record.outcome) | $($record.passed) | $($record.total) | $detail |"
|
||||
$lines += "| $([System.IO.Path]::GetFileNameWithoutExtension($record.project)) | $($record.outcome) | $($record.passed) | $($record.total) | $($record.buildSeconds) | $($record.testSeconds) | $detail |"
|
||||
}
|
||||
$lines += ''
|
||||
if (-not [string]::IsNullOrWhiteSpace($env:GITHUB_STEP_SUMMARY)) {
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Refreshes .github/ci-test-durations.json from the results of a CI run.
|
||||
|
||||
.DESCRIPTION
|
||||
get-ci-matrix.ps1 packs test projects into batches by the weights in
|
||||
.github/ci-test-durations.json. The weights only steer balance - a stale or
|
||||
missing weight never changes what runs - but batches drift out of balance as
|
||||
suites grow, so refresh them from a recent run now and then:
|
||||
|
||||
gh run download <run-id> --pattern 'dotnet-results-*' --dir ./ci-results
|
||||
./.github/scripts/update-test-durations.ps1 -ResultsPath ./ci-results -Source 'Run <run-id>'
|
||||
|
||||
Each project's weight is its slowest profile's test minutes, rounded up,
|
||||
plus one minute for its incremental build in a batch that already built the
|
||||
shared dependencies. The cold build of those dependencies is a fixed cost of
|
||||
the batch, not of any project, so first-in-batch build time is ignored.
|
||||
Projects the run did not execute keep their current weight.
|
||||
|
||||
.PARAMETER ResultsPath
|
||||
Directory holding the downloaded 'dotnet-results-*' artifacts. Every
|
||||
batch-summary.json below it is read.
|
||||
|
||||
.PARAMETER Source
|
||||
Short description of the run the weights came from, recorded in the table.
|
||||
|
||||
.PARAMETER DurationsPath
|
||||
The table to update. Defaults to .github/ci-test-durations.json.
|
||||
#>
|
||||
|
||||
Param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string] $ResultsPath,
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string] $Source,
|
||||
[string] $DurationsPath = ''
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($DurationsPath)) {
|
||||
$DurationsPath = Join-Path $PSScriptRoot '..' 'ci-test-durations.json'
|
||||
}
|
||||
|
||||
$table = Get-Content -LiteralPath $DurationsPath -Raw | ConvertFrom-Json -AsHashtable
|
||||
$weights = @{}
|
||||
foreach ($entry in $table.projects.GetEnumerator()) {
|
||||
$weights[$entry.Key] = [int]$entry.Value
|
||||
}
|
||||
|
||||
$summaries = @(Get-ChildItem -LiteralPath $ResultsPath -Recurse -File -Filter 'batch-summary.json')
|
||||
if ($summaries.Count -eq 0) {
|
||||
throw "No batch-summary.json below '$ResultsPath'. Download the run's 'dotnet-results-*' artifacts first."
|
||||
}
|
||||
|
||||
$slowest = @{}
|
||||
foreach ($summary in $summaries) {
|
||||
$document = Get-Content -LiteralPath $summary.FullName -Raw | ConvertFrom-Json
|
||||
foreach ($record in $document.projects) {
|
||||
# A failed or timed-out project's duration says nothing about a healthy
|
||||
# run, and a not-applicable one did not run at all.
|
||||
if ($record.outcome -ne 'passed' -or $null -eq $record.testSeconds) {
|
||||
continue
|
||||
}
|
||||
$stem = [System.IO.Path]::GetFileNameWithoutExtension($record.project)
|
||||
if (-not $slowest.ContainsKey($stem) -or $record.testSeconds -gt $slowest[$stem]) {
|
||||
$slowest[$stem] = [int]$record.testSeconds
|
||||
}
|
||||
}
|
||||
}
|
||||
if ($slowest.Count -eq 0) {
|
||||
throw "The summaries below '$ResultsPath' record no passed project with a duration."
|
||||
}
|
||||
|
||||
foreach ($entry in $slowest.GetEnumerator()) {
|
||||
$weights[$entry.Key] = [int][System.Math]::Ceiling($entry.Value / 60.0) + 1
|
||||
}
|
||||
|
||||
$keys = [System.Collections.Generic.List[string]]::new()
|
||||
foreach ($key in $weights.Keys) {
|
||||
$keys.Add($key)
|
||||
}
|
||||
$keys.Sort([System.StringComparer]::Ordinal)
|
||||
$projects = [ordered]@{}
|
||||
foreach ($key in $keys) {
|
||||
$projects[$key] = $weights[$key]
|
||||
}
|
||||
|
||||
[ordered]@{
|
||||
source = $Source
|
||||
defaultMinutes = [int]$table.defaultMinutes
|
||||
projects = $projects
|
||||
} | ConvertTo-Json -Depth 4 | Set-Content -LiteralPath $DurationsPath -Encoding utf8NoBOM
|
||||
|
||||
Write-Host "Updated $($slowest.Count) of $($projects.Count) weights in '$DurationsPath' from $($summaries.Count) batch summaries."
|
||||
Reference in New Issue
Block a user