Files
romanett b8f07843b2 Secure channel: never answer with an expired token; accept a just-expired one (+ labelled soak/stress CI) (#4698)
## Summary

**Secure channel: a client could drop the channel when the server
answered with a just-expired token.** Running the long-dormant
ConnectionStability soak locally, the subscription channel shut down
with `BadSecureChannelClosed`. The session then timed out and the
subscription was abandoned. Root cause, against OPC 10000-4 §5.6.2.1:

1. **Server (rule 2: use the old token only "until the SecurityToken
expires").** After a renew, the server keeps securing messages with the
old token until the client sends something secured with the new one.
That activation only happened on *receive*. A client that sends nothing,
for example one whose Publish requests are all outstanding, kept getting
responses secured with the old token after it expired.
2. **Client (rule 3: "accept Messages secured by an expired
SecurityToken for up to 25 % of the token lifetime").** The client
rejected any such response the moment the token expired, throwing
`BadTcpSecureChannelUnknown`, and shut the channel down.

From the failing run (token lifetime 10 s): the client renewed to token
#11 at 09:59:01.686 and sent nothing on that channel afterwards. The
server, still on #10, sent a Publish response at 09:59:04.389, 380 ms
after #10 expired. The client rejected it and closed the channel.
Nothing reconnected, the 10 s session timeout expired, and the server
abandoned the subscription.

Both halves date back to the 2016 initial commit; this is not a recent
regression. It hits whenever the client sends nothing on the channel
between renewing and the old token's expiry, so it is timing-dependent.

### Fix
- `UaSCBinaryChannel.ActivateRenewedTokenIfDue()` activates a pending
renewed token once the current one passes `TokenActivationPeriod` (95
%). This is the rule the receive path already applied. It is now shared,
and `TcpServerChannel.SendResponse` calls it before securing a response.
- `ReadSymmetricMessage` accepts responses (not requests) secured with
an expired token for `TcpMessageLimits.TokenExpiryGracePeriod` (25 %) of
its lifetime.

### Tests
`TcpReconnectKeyAgreementTests` now runs the server channel and the peer
on one shared `FakeTimeProvider`:
- `ResponseSentAfterTheCurrentTokenExpiresUsesTheRenewedTokenAsync`: a
response sent at 96 % of the old token's lifetime is secured with the
renewed token.
-
`ResponseSecuredWithTheExpiredTokenIsAcceptedWithinTheGracePeriodAsync`:
accepted at 120 % of the lifetime, rejected at 130 %.

Both fail without the fix. All 40 tests in the fixture pass with it.

## CI: soak and stress runs on labelled PRs (and the soak that tested
nothing)

The bug stayed hidden because `stability-test.yml` still filtered
`Opc.Ua.Client.Tests` after #3810 moved the test to
`Opc.Ua.Sessions.Tests` (2026-05-28). Since then the nightly has run
**zero tests** while reporting green: runs dropped from ~100 min to ~15
min.

- `stability-test.yml` now targets `Opc.Ua.Sessions.Tests` and writes a
TRX file.
- New `.github/scripts/assert-tests-ran.ps1`, used by all three opt-in
workflows, fails a run in which no test executed, or none passed.
Contract-tested in `CiAssertTestsRanTests`.
- Like `interop`, the opt-in workflows can now run on a labelled PR.
They are not required checks. A later push cancels the run in progress,
and adding an unrelated label neither starts nor cancels a run.

| Label | Workflows | Run on a PR |
| --- | --- | --- |
| `stress` | Stress Tests (ChaosTCP) | Random seed, saved as
`chaos-seed.txt` in the results artifact |
| `soak` | Connection Stability Test, Sample HA Long-Haul Test | 30 min
stability; 15 min per HA scenario |

## Test plan

- [x] `TcpReconnectKeyAgreementTests`: 40/40 pass; the 2 new tests fail
without the fix
- [x] All `Opc.Ua.Core.Tests.Stack` tests (net10.0): 3548 passed, 0
failed
- [x] `CiAssertTestsRanTests`: 4/4
- [x] `soak` + `stress` labelled run on this PR (before the fix): both
stability tests, both HA long-haul scenarios and ChaosTCP passed on
Linux (the label triggers and push-cancellation work)
- [x] Local short-haul soak with the fix (Windows, 10 s tokens), 2 runs:
both pass with full notification counts; run 2 hit the window once
(`Token activated forced` on send) and kept the channel
- [x] `soak` + `stress` labelled run with the fix: Connection Stability
(2 tests), Sample HA Long-Haul and ChaosTCP all pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 19:14:25 +02:00

106 lines
3.8 KiB
YAML

name: Connection Stability Test
# Long-haul connection stability soak (Category=ConnectionStability in
# Opc.Ua.Sessions.Tests; the tests are [Explicit] and only run here).
#
# Opt-in: nightly (90 minutes), manual (choose the duration), and pull requests
# labelled 'soak' (30 minutes). It is not a required check. Adding the label
# starts a run; every later push to the labelled pull request starts a new one
# and cancels the run in progress.
on:
schedule:
# Run nightly at 2:00 AM UTC
- cron: '0 2 * * *'
workflow_dispatch:
inputs:
duration:
description: 'Test duration in minutes'
required: false
default: '90'
type: string
pull_request:
types: [labeled, synchronize, opened, reopened]
permissions:
contents: read
# A pull request run is superseded by the next push. A 'labeled' event for any
# other label gets a group of its own so it cannot cancel a run in progress.
concurrency:
group: ${{ github.workflow }}-${{ github.event.action == 'labeled' && github.event.label.name != 'soak' && github.run_id || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
stability-test:
name: Connection Stability Test
# On a pull request only when it carries the 'soak' label, and not again
# when some other label is added to it.
if: >-
github.event_name != 'pull_request' ||
(contains(github.event.pull_request.labels.*.name, 'soak') &&
(github.event.action != 'labeled' || github.event.label.name == 'soak'))
runs-on: ubuntu-latest
timeout-minutes: 120 # Allow extra time beyond test duration for setup/teardown
env:
DOTNET_VERSION: '10.0.401'
TARGET_FRAMEWORK: 'net10.0'
CONFIGURATION: 'Release'
TESTPROJECT: './tests/Opc.Ua.Sessions.Tests/Opc.Ua.Sessions.Tests.csproj'
TEST_DURATION_MINUTES: ${{ github.event_name == 'pull_request' && '30' || github.event.inputs.duration || '90' }}
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Setup .NET ${{ env.DOTNET_VERSION }}
uses: actions/setup-dotnet@v6
with:
dotnet-version: ${{ env.DOTNET_VERSION }}
- name: Set Cloud Version
shell: pwsh
run: ./.azurepipelines/set-version.ps1
- name: Restore dependencies
run: dotnet restore 'UA.slnx'
- name: Build Sessions Tests
run: >
dotnet build ${{ env.TESTPROJECT }}
--framework ${{ env.TARGET_FRAMEWORK }}
--configuration ${{ env.CONFIGURATION }}
--no-restore
/p:CustomTestTarget=${{ env.TARGET_FRAMEWORK }}
- name: Run Connection Stability Test
run: |
echo "Starting connection stability test for $TEST_DURATION_MINUTES minutes"
dotnet test ${{ env.TESTPROJECT }} \
--configuration ${{ env.CONFIGURATION }} \
--no-build \
--framework ${{ env.TARGET_FRAMEWORK }} \
/p:CustomTestTarget=${{ env.TARGET_FRAMEWORK }} \
--filter "Category=ConnectionStability" \
--logger "trx;LogFileName=connection-stability.trx" \
--logger "console;verbosity=detailed" \
--results-directory ./TestResults
timeout-minutes: 110 # Slightly longer than expected test duration
# 'dotnet test' exits 0 when the filter matches nothing; until this check
# existed the nightly run stayed green while running no test at all, from the
# move of the test to Opc.Ua.Sessions.Tests (#3810, May 2026) until the workflow
# was pointed at that project.
- name: Check that the stability tests ran
shell: pwsh
run: ./.github/scripts/assert-tests-ran.ps1 -ResultsDirectory ./TestResults
- name: Upload test results
uses: actions/upload-artifact@v7
with:
name: stability-test-results
path: ./TestResults
if: always()