Files
romanett b03575b4e2 Run the interop tests against six foreign OPC UA stacks and extend feature coverage (#4644)
## Proposed changes

Runs the interop fixtures added in #4630 against peers built on six
other open source OPC UA stacks, so the 2.0 client and server are tested
against independent implementations, not only the released 1.5.378
stack. It also extends the fixtures beyond read/write/browse to events,
subscriptions, identity and more services.

### Peers (`tests/Opc.Ua.Interop.Peers`)

| Peer | Stack | Toolchain | Roles |
| --- | --- | --- | --- |
| `node-opcua` | node-opcua 2.186.15 (MIT) | Node.js 24 | server, client
|
| `milo` | Eclipse Milo 1.1.7 (EPL-2.0) | Java 17+, Maven | server,
client |
| `asyncua` | opcua-asyncio 2.0.1 (LGPL-3.0) | Python 3.12 | server,
client |
| `open62541` | open62541 1.5.8 (MPL-2.0), OpenSSL backend | C, CMake |
server, client, ECC |
| `async-opcua` | async-opcua 0.19.0 (MPL-2.0) | Rust | server, client |
| `gopcua` | gopcua 0.9.1 (MIT) | Go | client only |

Every peer speaks the command line and stdout contract of
`Opc.Ua.Interop.LegacyPeer` (`server` / `client --checks`, `PEER-INFO
{json}`, `RESULT {json}`). The stacks are fetched at build time and run
as separate processes; nothing is linked into or shipped with any
package.

### Harness

- **Peer selection:** `OPCUA_INTEROP_PEER_HOST` +
`OPCUA_INTEROP_PEER_HOST_ARGUMENTS` choose the host (`node`, `java
-jar`, `python`) for the peer named by `OPCUA_INTEROP_LEGACY_PEER`.
Native peers (open62541, async-opcua, gopcua) run without a host. Unset,
the 1.5.378 peer runs exactly as before, so the main CI is unchanged.
- **`PEER-INFO {json}`:** stack, version, application URI, software
version and, optionally, the security policies the peer implements.
`LegacyServerInteropTests` compares the ApplicationUri and
SoftwareVersion against it. `EccInteropTests` runs policies beyond the
four 1.05 ECC policies only against a peer that declares them.
- **`expected-differences.json` per peer:** confirmed defects of a
stack, keyed by check name, `Fixture.Test` or a `*` prefix, each with
the source file or spec section that shows it. A listed failure is
reported as inconclusive with its reason; a listed test that passes
produces a warning. The 2.0 client tests apply it through the
fixture-level `[PeerDifferences]` action.

### New coverage (all peers, including the 1.5.378 peer)

- **16 client checks**, run by each peer's client against the 2.0 server
(`LegacyClientFeatureCheckAsync`, Basic256Sha256 SignAndEncrypt):
EventSubscription, ConditionRefresh, AlarmAcknowledge, DeadbandFilter,
QueueOverflow, Triggering, Republish, TransferSubscription,
WrongPasswordRejected, X509UserToken, RegisterNodes, HistoryReadRaw,
NodeManagement, IndexRange, FindServers, SessionReconnect.
- **`ServerFeatureInteropTests`**: the 2.0 client against each peer
server. It covers RaiseEvent, ConditionRefresh, absolute deadband, queue
overflow bit, triggering, republish errors, transfer to a new session,
X509 user token, RegisterNodes, IndexRange, AddNodes/DeleteNodes,
FindServers and session reconnect. Every peer server gained a
`RaiseEvent` method and accepts X509 user tokens.
- **ECC:** with open62541's OpenSSL backend, `EccInteropTests` adds
`ECC_nistP256_AesGcm` and `ECC_nistP256_ChaChaPoly` (both directions;
SignAndEncrypt anonymous and username, Sign, and token renewal across
two renewals). It also adds `ECC_curve25519`/`ECC_curve448` cases, which
are ignored while 2.0 marks those policies unsupported.

### CI and docs

- **Workflow `interop-foreign.yml`:** opt-in (nightly,
`workflow_dispatch`, PRs labelled `interop`), not a required check. It
builds all six peers on Windows, Linux and macOS (open62541 from source
on Linux only, against the distro OpenSSL), with dependency caching.
- **`tests/Opc.Ua.Interop.Peers/README.md`:** how to install the
toolchains, build each peer (including Windows without Visual Studio),
run the fixtures against it, maintain expected differences and add a new
stack. `DeveloperGuide.md` links to it.

## Results

Run locally, sequentially, net10.0, Windows. Not passed = expected
differences, inconclusive by design (e.g. AddNodes on a server without
NodeManagement), or ECC cases a peer or the platform does not support.

| Peer | Fixtures | Total | Passed | Failed |
| --- | --- | --- | --- | --- |
| 1.5.378 | whole interop project | 220 | 218 | 0 |
| node-opcua | server + client + feature | 145 | 143 | 0 |
| Eclipse Milo | server + client + feature | 145 | 141 | 0 |
| asyncua | server + client + feature | 145 | 133 | 0 |
| open62541 (OpenSSL) | server + client + feature + ECC | 199 | 175 | 0
|
| async-opcua | server + client + feature | 145 | 137 | 0 |
| gopcua | client | 104 | 81 | 0 |

After the AEAD cases were added, `EccInteropTests` against the 1.5.378
peer ran separately: 54 total, 26 passed, 0 failed. The new cases are
ignored there because the peer does not declare them.

## Findings

No 2.0 defects beyond the two now fixed in #4654:
- Browse with a null ViewId and a Timestamp now returns
`Bad_ViewTimestampInvalid`.
- Subscription transfer no longer fails when the optional
`Server.GetMonitoredItems` is unavailable.

After #4654 merges, `TransferSubscriptionToNewSessionAsync` can drop its
expected differences for asyncua.

Upstream defects found, all listed in the peers'
`expected-differences.json` with evidence; issues are still to be filed:

- **All five non-.NET stacks:** cannot handle the TestData
`ScalarStructureDataType` (fields of the abstract
Enumeration/Structure/BaseDataType). Milo encodes the abstract
`Enumeration` field as a Variant (Part 6 5.2.4 says Int32).
- **open62541:**
- **Sign-only ECC keys:** derives encryption key and IV into the HKDF
length for Sign-only ECC (Part 6 6.8.1), so Sign mode fails for the four
1.05 curves. The AEAD policies are not affected.
  - **HMAC mismatch:** reported as `Bad_InternalError`.
  - **ConditionRefresh:** needs the experimental A&C option.
- **Response too large:** closes the connection instead of answering
`Bad_ResponseTooLarge`.
  - **Array limit:** no MaxArrayLength.
- **gopcua:**
- **RsaPss:** `RSAOAEPMinPaddingSHA256` is 130 instead of 66, which
breaks `Aes256_Sha256_RsaPss`.
  - **Nil LocalizedText:** encoding a nil `*LocalizedText` panics.
- **asyncua server:**
  - **Queue overflow:** no Overflow bit.
  - **SetTriggering:** not implemented.
- **Republish:** answers Good for an unknown sequence number or
subscription.
  - **GetMonitoredItems:** not implemented.
  - **IndexRange:** ignored on Read and Write.
- **Reconnect:** closes a session without subscriptions on transport
loss.
- **Limits:** no BrowseNext; operation limits not enforced; no
`Bad_ResponseTooLarge`.
- **async-opcua:**
- **X509 user token (client):** treats an empty token SecurityPolicyUri
as None.
- **Reconnect (client):** overwrites the session nonce with the channel
nonce on reconnect.
  - **ConditionRefresh (server):** not implemented.
  - **Overflow bit (server):** set without the DataValue InfoType bit.
  - **ECC:** no ECC policies.
- **Milo:**
- **Wrong password:** `BadIdentityTokenInvalid` instead of
`BadUserAccessDenied` / `BadIdentityTokenRejected`.
  - **Array limit:** accepts arrays above its advertised MaxArrayLength.
  - **AddNodes:** answers `BadNotSupported`.

## Related Issues

- Follows #4630. Related fixes: #4654.

## Types of changes

- [x] Test / CI only

## Checklist

- [x] Interop fixtures pass locally against all seven peers (net10.0,
Windows).
- [x] `dotnet format whitespace --verify-no-changes` on the changed C#
files.
- [ ] `interop-foreign.yml` has not run on GitHub yet; it needs a manual
dispatch or the `interop` label.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 12:03:05 +02:00

217 lines
9.3 KiB
YAML

name: Foreign stack interop
# Runs the interop fixtures of tests/Opc.Ua.Interop.Tests against peers built
# on other OPC UA stacks (tests/Opc.Ua.Interop.Peers/<stack>). The fixtures are
# the same ones the main CI runs against the 1.5.378 peer; the peer is swapped
# through OPCUA_INTEROP_PEER_HOST, OPCUA_INTEROP_PEER_HOST_ARGUMENTS and
# OPCUA_INTEROP_LEGACY_PEER (native peers run without a host). Known defects of
# a stack are listed in its expected-differences.json and reported as
# inconclusive.
#
# Opt-in: nightly, manual, and pull requests labelled 'interop'. It is not a
# required check, so a regression of a foreign stack never blocks a merge.
on:
schedule:
- cron: '17 3 * * *'
workflow_dispatch:
pull_request:
types: [labeled, synchronize, opened, reopened]
permissions:
contents: read
concurrency:
group: interop-foreign-${{ github.ref }}
cancel-in-progress: true
env:
DOTNET_NOLOGO: true
DOTNET_CLI_TELEMETRY_OPTOUT: true
OPEN62541_VERSION: v1.5.8
jobs:
interop:
if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'interop')
name: ${{ matrix.peer }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
peer: [node-opcua, milo, asyncua, open62541, async-opcua, gopcua]
exclude:
# Built from source; Linux covers the stack (incl. the ECC policies,
# with OpenSSL also ECC_nistP256_AesGcm/_ChaChaPoly).
- os: windows-latest
peer: open62541
- os: macos-latest
peer: open62541
env:
PEERS: ${{ github.workspace }}/tests/Opc.Ua.Interop.Peers
steps:
- uses: actions/checkout@v7
with:
# Nerdbank.GitVersioning computes the version height from history and
# fails the build of a shallow clone.
fetch-depth: 0
- name: Setup .NET
uses: actions/setup-dotnet@v6
with:
dotnet-version: '10.0.401'
# ---------------------------------------------------------------- node-opcua
- name: Setup Node.js
if: matrix.peer == 'node-opcua'
uses: actions/setup-node@v7
with:
node-version: '24'
cache: npm
cache-dependency-path: tests/Opc.Ua.Interop.Peers/node-opcua/package-lock.json
- name: Install the node-opcua peer
if: matrix.peer == 'node-opcua'
working-directory: tests/Opc.Ua.Interop.Peers/node-opcua
run: npm ci --no-audit --no-fund
# ---------------------------------------------------------------- Milo
- name: Setup Java
if: matrix.peer == 'milo'
uses: actions/setup-java@v6
with:
distribution: temurin
java-version: '21'
cache: maven
cache-dependency-path: tests/Opc.Ua.Interop.Peers/milo/pom.xml
- name: Build the Milo peer
if: matrix.peer == 'milo'
run: mvn -B -q -f tests/Opc.Ua.Interop.Peers/milo/pom.xml package
# ---------------------------------------------------------------- asyncua
- name: Setup Python
if: matrix.peer == 'asyncua'
uses: actions/setup-python@v7
with:
python-version: '3.12'
cache: pip
cache-dependency-path: tests/Opc.Ua.Interop.Peers/asyncua/requirements.txt
- name: Install the asyncua peer
if: matrix.peer == 'asyncua'
run: pip install -r tests/Opc.Ua.Interop.Peers/asyncua/requirements.txt
# ---------------------------------------------------------------- open62541
- name: Cache open62541
if: matrix.peer == 'open62541'
id: open62541-cache
uses: actions/cache@v6
with:
path: ${{ runner.temp }}/open62541-install
key: open62541-${{ env.OPEN62541_VERSION }}-openssl3-xml-${{ runner.os }}
# OpenSSL rather than mbedTLS: open62541 implements the AEAD
# (ECC_nistP256_AesGcm/_ChaChaPoly) and Edwards curve ECC policies only
# with OpenSSL. The installed library links the distro libssl (3.x).
- name: Install OpenSSL
if: matrix.peer == 'open62541'
run: dpkg -s libssl-dev > /dev/null 2>&1 || (sudo apt-get update -q && sudo apt-get install -y --no-install-recommends libssl-dev)
- name: Build open62541 with OpenSSL
if: matrix.peer == 'open62541' && steps.open62541-cache.outputs.cache-hit != 'true'
working-directory: ${{ runner.temp }}
run: |
git clone -q --depth 1 --branch "$OPEN62541_VERSION" https://github.com/open62541/open62541.git
git -C open62541 submodule update --init --depth 1 deps/ua-nodeset
# The client sources are only built with discovery enabled. The
# InputArguments/OutputArguments values of the namespace 0 methods
# (e.g. Server.GetMonitoredItems, which the .NET client calls after
# TransferSubscriptions) are only generated with XML encoding enabled.
cmake -S open62541 -B build-open62541 -DCMAKE_BUILD_TYPE=Release -DUA_ENABLE_ENCRYPTION=OPENSSL \
-DUA_NAMESPACE_ZERO=FULL -DUA_ENABLE_DISCOVERY=ON -DUA_ENABLE_XML_ENCODING=ON \
-DUA_BUILD_EXAMPLES=OFF -DUA_BUILD_UNIT_TESTS=OFF -DUA_FORCE_WERROR=OFF \
-DCMAKE_INSTALL_PREFIX="$RUNNER_TEMP/open62541-install"
cmake --build build-open62541 --parallel && cmake --install build-open62541
- name: Build the open62541 peer
if: matrix.peer == 'open62541'
run: |
cmake -S tests/Opc.Ua.Interop.Peers/open62541 -B tests/Opc.Ua.Interop.Peers/open62541/build -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH="$RUNNER_TEMP/open62541-install"
cmake --build tests/Opc.Ua.Interop.Peers/open62541/build
# ---------------------------------------------------------------- async-opcua
- name: Cache cargo
if: matrix.peer == 'async-opcua'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
tests/Opc.Ua.Interop.Peers/async-opcua/target
key: async-opcua-${{ runner.os }}-${{ hashFiles('tests/Opc.Ua.Interop.Peers/async-opcua/Cargo.toml') }}
- name: Build the async-opcua peer
if: matrix.peer == 'async-opcua'
run: cargo build --release --manifest-path tests/Opc.Ua.Interop.Peers/async-opcua/Cargo.toml
# ---------------------------------------------------------------- gopcua
- name: Setup Go
if: matrix.peer == 'gopcua'
uses: actions/setup-go@v7
with:
go-version-file: tests/Opc.Ua.Interop.Peers/gopcua/go.mod
cache-dependency-path: tests/Opc.Ua.Interop.Peers/gopcua/go.sum
- name: Build the gopcua peer
if: matrix.peer == 'gopcua'
working-directory: tests/Opc.Ua.Interop.Peers/gopcua
run: go build -o build/gopcua-peer${{ runner.os == 'Windows' && '.exe' || '' }} .
# ---------------------------------------------------------------- run
- name: Select the peer
shell: pwsh
run: |
$ext = $IsWindows ? '.exe' : ''
$main = 'FullyQualifiedName~LegacyServerInteropTests|FullyQualifiedName~LegacyClientInteropTests|FullyQualifiedName~ServerFeatureInteropTests'
switch ('${{ matrix.peer }}') {
'node-opcua' {
"OPCUA_INTEROP_PEER_HOST=$((Get-Command node).Source)" >> $env:GITHUB_ENV
"OPCUA_INTEROP_LEGACY_PEER=$env:PEERS/node-opcua/peer.mjs" >> $env:GITHUB_ENV
$filter = $main
}
'milo' {
"OPCUA_INTEROP_PEER_HOST=$((Get-Command java).Source)" >> $env:GITHUB_ENV
"OPCUA_INTEROP_PEER_HOST_ARGUMENTS=-jar" >> $env:GITHUB_ENV
"OPCUA_INTEROP_LEGACY_PEER=$env:PEERS/milo/target/milo-peer.jar" >> $env:GITHUB_ENV
$filter = $main
}
'asyncua' {
"OPCUA_INTEROP_PEER_HOST=$((Get-Command python).Source)" >> $env:GITHUB_ENV
"OPCUA_INTEROP_LEGACY_PEER=$env:PEERS/asyncua/peer.py" >> $env:GITHUB_ENV
$filter = $main
}
'open62541' {
"OPCUA_INTEROP_LEGACY_PEER=$env:PEERS/open62541/build/open62541-peer$ext" >> $env:GITHUB_ENV
$filter = "$main|FullyQualifiedName~EccInteropTests"
}
'async-opcua' {
"OPCUA_INTEROP_LEGACY_PEER=$env:PEERS/async-opcua/target/release/async-opcua-peer$ext" >> $env:GITHUB_ENV
$filter = $main
}
'gopcua' {
# Client only: the gopcua server cannot run the server contract.
"OPCUA_INTEROP_LEGACY_PEER=$env:PEERS/gopcua/build/gopcua-peer$ext" >> $env:GITHUB_ENV
$filter = 'FullyQualifiedName~LegacyClientInteropTests'
}
}
"INTEROP_FILTER=$filter" >> $env:GITHUB_ENV
- name: Test
run: >
dotnet test tests/Opc.Ua.Interop.Tests/Opc.Ua.Interop.Tests.csproj
-c Release -f net10.0 -p:CustomTestTarget=net10.0
--filter "${{ env.INTEROP_FILTER }}"
--logger "trx;LogFileName=interop-${{ matrix.peer }}.trx"
--results-directory ./TestResults
- name: Upload results
if: always()
uses: actions/upload-artifact@v7
with:
name: interop-${{ matrix.peer }}-${{ matrix.os }}
path: ./TestResults