Files
UA-.NETStandard/.github/workflows/stress-test.yml
romanett b8f07843b2 Secure channel: never answer with an expired token; accept a just-expired one (+ labelled soak/stress CI) (#4698)
## Summary

**Secure channel: a client could drop the channel when the server
answered with a just-expired token.** Running the long-dormant
ConnectionStability soak locally, the subscription channel shut down
with `BadSecureChannelClosed`. The session then timed out and the
subscription was abandoned. Root cause, against OPC 10000-4 §5.6.2.1:

1. **Server (rule 2: use the old token only "until the SecurityToken
expires").** After a renew, the server keeps securing messages with the
old token until the client sends something secured with the new one.
That activation only happened on *receive*. A client that sends nothing,
for example one whose Publish requests are all outstanding, kept getting
responses secured with the old token after it expired.
2. **Client (rule 3: "accept Messages secured by an expired
SecurityToken for up to 25 % of the token lifetime").** The client
rejected any such response the moment the token expired, throwing
`BadTcpSecureChannelUnknown`, and shut the channel down.

From the failing run (token lifetime 10 s): the client renewed to token
#11 at 09:59:01.686 and sent nothing on that channel afterwards. The
server, still on #10, sent a Publish response at 09:59:04.389, 380 ms
after #10 expired. The client rejected it and closed the channel.
Nothing reconnected, the 10 s session timeout expired, and the server
abandoned the subscription.

Both halves date back to the 2016 initial commit; this is not a recent
regression. It hits whenever the client sends nothing on the channel
between renewing and the old token's expiry, so it is timing-dependent.

### Fix
- `UaSCBinaryChannel.ActivateRenewedTokenIfDue()` activates a pending
renewed token once the current one passes `TokenActivationPeriod` (95
%). This is the rule the receive path already applied. It is now shared,
and `TcpServerChannel.SendResponse` calls it before securing a response.
- `ReadSymmetricMessage` accepts responses (not requests) secured with
an expired token for `TcpMessageLimits.TokenExpiryGracePeriod` (25 %) of
its lifetime.

### Tests
`TcpReconnectKeyAgreementTests` now runs the server channel and the peer
on one shared `FakeTimeProvider`:
- `ResponseSentAfterTheCurrentTokenExpiresUsesTheRenewedTokenAsync`: a
response sent at 96 % of the old token's lifetime is secured with the
renewed token.
-
`ResponseSecuredWithTheExpiredTokenIsAcceptedWithinTheGracePeriodAsync`:
accepted at 120 % of the lifetime, rejected at 130 %.

Both fail without the fix. All 40 tests in the fixture pass with it.

## CI: soak and stress runs on labelled PRs (and the soak that tested
nothing)

The bug stayed hidden because `stability-test.yml` still filtered
`Opc.Ua.Client.Tests` after #3810 moved the test to
`Opc.Ua.Sessions.Tests` (2026-05-28). Since then the nightly has run
**zero tests** while reporting green: runs dropped from ~100 min to ~15
min.

- `stability-test.yml` now targets `Opc.Ua.Sessions.Tests` and writes a
TRX file.
- New `.github/scripts/assert-tests-ran.ps1`, used by all three opt-in
workflows, fails a run in which no test executed, or none passed.
Contract-tested in `CiAssertTestsRanTests`.
- Like `interop`, the opt-in workflows can now run on a labelled PR.
They are not required checks. A later push cancels the run in progress,
and adding an unrelated label neither starts nor cancels a run.

| Label | Workflows | Run on a PR |
| --- | --- | --- |
| `stress` | Stress Tests (ChaosTCP) | Random seed, saved as
`chaos-seed.txt` in the results artifact |
| `soak` | Connection Stability Test, Sample HA Long-Haul Test | 30 min
stability; 15 min per HA scenario |

## Test plan

- [x] `TcpReconnectKeyAgreementTests`: 40/40 pass; the 2 new tests fail
without the fix
- [x] All `Opc.Ua.Core.Tests.Stack` tests (net10.0): 3548 passed, 0
failed
- [x] `CiAssertTestsRanTests`: 4/4
- [x] `soak` + `stress` labelled run on this PR (before the fix): both
stability tests, both HA long-haul scenarios and ChaosTCP passed on
Linux (the label triggers and push-cancellation work)
- [x] Local short-haul soak with the fix (Windows, 10 s tokens), 2 runs:
both pass with full notification counts; run 2 hit the window once
(`Token activated forced` on send) and kept the channel
- [x] `soak` + `stress` labelled run with the fix: Connection Stability
(2 tests), Sample HA Long-Haul and ChaosTCP all pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 19:14:25 +02:00

140 lines
4.8 KiB
YAML

# Stress test umbrella workflow.
#
# Each job is one stress scenario, filtered by NUnit Category. Add new
# scenarios by adding a sibling job below with the same seed-generation
# pattern and the same 'if' gate.
#
# Opt-in: nightly, manual, and pull requests labelled 'stress'. It is not a
# required check. Adding the label starts a run; every later push to the
# labelled pull request starts a new one and cancels the run in progress.
name: Stress Tests
on:
schedule:
# Run nightly at 3:00 AM UTC.
- cron: '0 3 * * *'
workflow_dispatch:
inputs:
seed:
description: 'Optional ChaosTCP seed. Leave empty to generate a random seed.'
required: false
default: ''
type: string
pull_request:
types: [labeled, synchronize, opened, reopened]
permissions:
contents: read
# A pull request run is superseded by the next push. A 'labeled' event for any
# other label gets a group of its own so it cannot cancel a run in progress.
concurrency:
group: ${{ github.workflow }}-${{ github.event.action == 'labeled' && github.event.label.name != 'stress' && github.run_id || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
chaos-tcp:
name: Chaos TCP
# On a pull request only when it carries the 'stress' label, and not again
# when some other label is added to it.
if: >-
github.event_name != 'pull_request' ||
(contains(github.event.pull_request.labels.*.name, 'stress') &&
(github.event.action != 'labeled' || github.event.label.name == 'stress'))
runs-on: ubuntu-latest
timeout-minutes: 120
env:
DOTNET_VERSION: '10.0.401'
TARGET_FRAMEWORK: 'net10.0'
CONFIGURATION: 'Release'
CUSTOM_TEST_TARGET: 'net10.0'
STRESS_PROJECT: './tests/Opc.Ua.Stress.Tests/Opc.Ua.Stress.Tests.csproj'
TESTRESULTS: './TestResults/chaos-tcp'
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Setup .NET ${{ env.DOTNET_VERSION }}
uses: actions/setup-dotnet@v6
with:
dotnet-version: ${{ env.DOTNET_VERSION }}
- name: Set Cloud Version
shell: pwsh
run: ./.azurepipelines/set-version.ps1
- name: Restore dependencies
run: dotnet restore 'UA.slnx'
- name: Build Stress Tests
run: >
dotnet build ${{ env.STRESS_PROJECT }}
--framework ${{ env.TARGET_FRAMEWORK }}
--configuration ${{ env.CONFIGURATION }}
--no-restore
/p:CustomTestTarget=${{ env.CUSTOM_TEST_TARGET }}
- name: Generate ChaosTCP seed
id: seed
shell: pwsh
env:
INPUT_SEED: ${{ github.event.inputs.seed }}
run: |
$configuredSeed = $env:INPUT_SEED
if ([string]::IsNullOrWhiteSpace($configuredSeed))
{
$seed = [System.Security.Cryptography.RandomNumberGenerator]::GetInt32([int]::MaxValue)
}
else
{
$seed = [int]::Parse($configuredSeed, [System.Globalization.CultureInfo]::InvariantCulture)
}
New-Item -ItemType Directory -Force -Path $env:TESTRESULTS | Out-Null
"OPCUA_CHAOS_SEED=$seed" | Out-File -FilePath $env:GITHUB_ENV -Append
"OPCUA_TEST_SEED=$seed" | Out-File -FilePath $env:GITHUB_ENV -Append
"seed=$seed" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
$repro = @(
"ChaosTCP seed: $seed",
'',
'Reproduce locally from the repository root:',
'PowerShell:',
" `$env:OPCUA_CHAOS_SEED = '$seed'",
" `$env:OPCUA_TEST_SEED = '$seed'",
' dotnet test .\tests\Opc.Ua.Stress.Tests\Opc.Ua.Stress.Tests.csproj --framework net10.0 --configuration Release --filter "Category=ChaosTCP"',
'',
'Bash:',
" OPCUA_CHAOS_SEED=$seed OPCUA_TEST_SEED=$seed dotnet test ./tests/Opc.Ua.Stress.Tests/Opc.Ua.Stress.Tests.csproj --framework net10.0 --configuration Release --filter 'Category=ChaosTCP'")
$repro | Set-Content -Path (Join-Path $env:TESTRESULTS 'chaos-seed.txt')
Write-Host "ChaosTCP seed: $seed"
- name: Run ChaosTCP tests
shell: pwsh
run: |
dotnet test $env:STRESS_PROJECT `
--configuration $env:CONFIGURATION `
--no-build `
--framework $env:TARGET_FRAMEWORK `
--filter "Category=ChaosTCP" `
--logger "trx;LogFileName=chaos-tcp.trx" `
--logger "console;verbosity=detailed" `
--results-directory $env:TESTRESULTS `
/p:CustomTestTarget=$env:CUSTOM_TEST_TARGET
- name: Check that the ChaosTCP tests ran
shell: pwsh
run: ./.github/scripts/assert-tests-ran.ps1 -ResultsDirectory $env:TESTRESULTS
- name: Upload ChaosTCP results
uses: actions/upload-artifact@v7
with:
name: chaos-tcp-${{ steps.seed.outputs.seed }}
path: ${{ env.TESTRESULTS }}
if: always()