Files
Marc Schier ea0f266b66 Move build and test to the Azure DevOps managed pool and replace Codecov with an in-pipeline coverage gate (#4122)
# Description

Moves the all-target-framework build, the cross-platform test matrix and
the Native AoT run from GitHub Actions onto the Azure DevOps
`netstandard` Managed DevOps Pool, and replaces Codecov with a coverage
gate that runs inside the pipeline.

## 1. CI backend switch

One conceptual switch decides who owns that work. It is checked into
source in **two places that must be flipped together**:

| File | Setting | Default |
| --- | --- | --- |
| `azure-pipelines.yml` | `parameters.ciBuildBackend` | `ado` |
| `.github/workflows/buildandtest.yml` | `env.CI_BUILD_BACKEND` | `ado`
|

Setting both to `actions` restores the previous behaviour exactly.

Azure Pipelines gains what it was missing to take the work over:

- A **Linux all-TFM `sln.yml` leg**, replacing `build-all-tfm-linux`.
`sln.yml` gets a `configurations` parameter so this leg builds `Release`
only instead of doubling the managed-pool job count.
- A **Linux agent in the `net10.0` `testfastpr` matrix**, replacing the
ubuntu `build-and-test` job.
- **Native AoT and Code Coverage now gate pull requests** instead of
running only on Schedule/Manual. The coverage stage drops its
`testdebug` dependency, which is Schedule/Manual-only and would
otherwise have skipped coverage on every PR.
- On a PR, Windows builds the .NET Framework TFMs and Linux the modern
ones — mirroring the split the GitHub Actions jobs used — so all seven
TFMs stay covered without fanning every TFM across both pools.

Two things are deliberately **not** covered by the switch:

- **macOS** stays on GitHub-hosted runners, because Managed DevOps Pools
provide no macOS image.
- **`master378` and `develop/*`** keep running the GitHub Actions jobs,
since Azure Pipelines only builds `master`/`main` from this file.

GitHub Actions does not expose the `env` context to job-level `if:`
conditions, so the switch is resolved once in the existing `discover`
job and re-exported as job outputs.

A new **`build-and-test summary`** job gives branch protection one
stable check to require, instead of the generated matrix job names. It
treats an intentionally skipped job as success but still fails on a real
failure or cancellation.

The `paths:` allow-list was moved off the `pull_request` trigger and
into `discover`. A workflow filtered out by `paths` never reports its
checks, and a required check that never reports leaves a PR permanently
*"Expected — waiting for status to be reported"*. That would have made
the routine "ratchet the coverage floor" PR (which touches only
`coverage-thresholds.json` and docs) unmergeable.

## 2. Coverage gate replacing Codecov

Codecov is removed entirely — `codecov.yml`, both uploads, the badge,
and the token usage. `.azurepipelines/check-coverage.ps1` runs at the
end of the `Code Coverage` stage against the Cobertura report
ReportGenerator already merges, driven by `coverage-thresholds.json`:

| Check | Behaviour |
| --- | --- |
| Project floor | **Blocking** — total line and branch rates against an
absolute floor. |
| Patch coverage | **Blocking on PRs** — changed lines must reach
`patch.target`, tolerating `patch.threshold` percentage points (the same
semantics `codecov/patch` had). |
| Baseline delta | **Advisory** — reports the change against the
recorded baseline, never fails. |

The project rate is recomputed from the per-line data rather than read
off the Cobertura root element, so the `ignore` globs carried over from
`codecov.yml` apply to the project floor as well as to the changed-lines
gate — otherwise samples and generated code would count towards one gate
but not the other.

`testcc.yml` gains an explicit `checkout: self` with `fetchDepth: 0`,
because the default shallow clone cannot diff against the base branch.

## Follow-up actions for maintainers

Two items cannot be automated from source:

1. **Enable the `/azp run` gate.** In the Azure DevOps portal: pipeline
→ **More actions** → **Triggers** → **Pull request validation** →
**Require a team member's comment before building a pull request**,
scoped to *Only on pull requests from non-team members*. This makes
Copilot-authored and outside-contributor PRs wait for a maintainer's
`/azp run`, mirroring the approval GitHub Actions already requires. The
documentation added here describes the workflow and the
private-org-membership gotcha; **the setting itself still has to be
switched on.**
2. **Update branch protection and secrets.** Drop the `codecov/project`
and `codecov/patch` required checks, require `build-and-test summary`
and the ADO coverage check instead, and remove `CODECOV_TOKEN` from the
GitHub repository secrets and the ADO variable group.

Also note the thresholds in `coverage-thresholds.json` are seeded from
the last Codecov reading (73.61 %) with headroom, because the
in-pipeline number had not been observed yet. The first full `Code
Coverage` run on master should confirm it and the floors should be
tightened then — the gate prints the exact figure it computes.

## Related Issues

_No tracking issue; this is CI infrastructure work requested directly._

## Checklist

- [x] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- No product code changed, so there are no unit tests to add. The new
logic was verified with purpose-built harnesses instead:
- All 8 branch/event/backend combinations of the `discover` switch,
confirming `master378` and `develop/*` are untouched and that `actions`
restores the old behaviour.
- 12 scenarios for `check-coverage.ps1`: the project floor, the patch
gate, ignore globs applying to *both* gates, three path-normalization
shapes, and the advisory delta.
- 14 path-relevance cases confirming the in-job filter matches the
retired `paths:` allow-list exactly.
- End-to-end against a **real** ReportGenerator report: correctly
resolved `src/Opc.Ua.PubSub/DependencyInjection/PubSubBuilder.cs`, found
exactly the four modified tracked lines and reported 50 % patch coverage
(2 covered, 2 uncovered).
- [x] I have added all necessary documentation.
- New **Continuous integration** section in `docs/DeveloperGuide.md`, a
CI note in `CONTRIBUTING.md`, and links from `docs/README.md`. Also
corrected the CI matrix description, which claimed the PR gate ran net48
+ **net8.0** — it runs net48 + **net10.0**.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- `dotnet build src/Opc.Ua.Server` → 0 warnings, 0 errors. No product
code changed.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- Not applicable — no product code changed. `Opc.Ua.Server.Tests` was
run on net10.0 during development (3480 passed; the single
`ServerFluentApiHostingTests` failure is pre-existing and reproduces on
master).
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [x] I have addressed **all** PR feedback received.
- A review pass over this branch flagged the `paths:`-filter deadlock
described above; it is fixed in `1926e240e`.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 181ba7ea-c72f-42d7-a737-00a3faadaa17
Copilot-Session: 26aa55a2-5731-4a4c-b61c-df2f7a785eca
2026-07-31 21:06:25 +00:00

15 KiB
Raw Permalink Blame History

Fuzz testing for UA.NET Standard

This project provides integration of SharpFuzz with the UA.NET Standard libraries, with support for both afl-fuzz and libFuzzer. Each fuzz area lives directly under fuzzing/ as four sibling folders — host (*.Fuzz), corpus (*.Fuzz.Corpus), test fixture (*.Fuzz.Tests), and tools (*.Fuzz.Tools) — and shares a generic NUnit replay harness plus the SharpFuzz host under Common/.

Areas

Area Surface Status
Opc.Ua.Encoders.Fuzz BinaryDecoder / JsonDecoder / XmlDecoder + idempotent round-trip, individual built-in type readers (NodeId, ExpandedNodeId, Variant, ExtensionObject, DataValue, DiagnosticInfo, QualifiedName, LocalizedText) and parser entry points (NodeId.Parse, ExpandedNodeId.Parse, RelativePathFormatter.Parse, QualifiedName.Parse, NumericRange.Parse, Uuid round-trip) shipped
Opc.Ua.Certificates.Fuzz X509CRL decode + extensions (X509SubjectAltNameExtension, X509AuthorityKeyIdentifierExtension, X509CrlNumberExtension), PEMReader cert/key import, Pkcs10CertificationRequest, low-level AsnUtils helpers shipped
Opc.Ua.Network.Fuzz OPC UA UA-SC / TCP framing via Opc.Ua.Core.Diagnostics (OpcUaFrameParser, TcpStreamReassembler, OfflineSecureChannel.ReadChunk, ServiceCallReassembler, MockServerReplay/MockClientReplay stateful drivers) and the Opc.Ua.Core UA-SC parse seam (TcpMessageParsers.TryParseChunkHeader / ReadHelloMessage / ReadAcknowledgeMessage / ReadErrorMessage / ReadReverseHelloMessage / ReadAsymmetricMessageHeader) shipped

Encoders and Certificates build for the repo's standard TestsTargetFrameworks matrix (net48, net8.0, net9.0, net10.0). Network is net8.0;net9.0;net10.0 only because Opc.Ua.Core.Diagnostics does not target .NET Framework.

Directory layout

fuzzing/
  Fuzzing.md                                                # this file

  Common/                                                   # shared source (no csproj)
    Fuzz/
      Program.cs, FuzzMethods.cs                            # SharpFuzz host
    Fuzz.Tests/
      FuzzTargetTestsBase.cs, FuzzTargetFunction.cs
      TestcaseAsset.cs, TestAssetUtils.cs                   # generic NUnit harness
    Fuzz.Tools/
      Program.cs, Playback.cs, Logging.cs, Testcases.cs     # Tools host

  Dictionaries/                                             # libFuzzer / afl dictionaries
    asn1.dict  binary.dict  json.dict  nodeid.dict
    tcp.dict   uasc.dict    xml.dict

  Scripts/                                                  # area-agnostic runners
    fuzz-afl.ps1  fuzz-libfuzzer.ps1  fuzz-menu.ps1
    install.sh  readme.txt

  Opc.Ua.Encoders.Fuzz/                                     # csproj + FuzzableCode partials
  Opc.Ua.Encoders.Fuzz.Corpus/                              # seed corpus (Testcases.*/)
  Opc.Ua.Encoders.Fuzz.Tests/                               # deterministic NUnit replay
  Opc.Ua.Encoders.Fuzz.Tools/                               # corpus generator + playback

  Opc.Ua.Certificates.Fuzz/   Opc.Ua.Certificates.Fuzz.Corpus/
  Opc.Ua.Certificates.Fuzz.Tests/   Opc.Ua.Certificates.Fuzz.Tools/

  Opc.Ua.Network.Fuzz/   Opc.Ua.Network.Fuzz.Corpus/
  Opc.Ua.Network.Fuzz.Tests/   Opc.Ua.Network.Fuzz.Tools/

The flat layout matches the rest of the repository (tests/, src/, src/ also list projects directly under the root with no domain grouping). Every csproj folder name equals the csproj filename (without the .csproj extension), and every assembly is prefixed with Opc.Ua. in line with the repo's AssemblyPrefix convention.

Each area's *.Fuzz project hosts FuzzableCode.*.cs partials whose public static methods with a single parameter of type Stream (afl), string (afl), or ReadOnlySpan<byte> (libFuzzer) are auto-discovered by:

  • the SharpFuzz host (Common/Fuzz/Program.cs) at fuzz-run time, and
  • the generic NUnit harness (Common/Fuzz.Tests/FuzzTargetTestsBase.cs) at test time — every target is replayed against the area's *.Fuzz.Corpus/Testcases.*/ corpus plus crash/timeout assets, with no per-area test code required.

Each area's *.Fuzz.Tests is therefore a single thin subclass:

[TestFixture]
[Category("Fuzzing")]
public class EncoderTests : FuzzTargetTestsBase
{
    [DatapointSource]
    public static readonly FuzzTargetFunction[] FuzzableFunctions =
        CreateFuzzTargetFunctions(typeof(FuzzableCode));

    protected override Type FuzzableCodeType => typeof(FuzzableCode);
}

How to add a new fuzz area

  1. Pick a surface. Anything that takes untrusted input (bytes, string, or stream) is a candidate. Prefer surfaces with deterministic output and bounded resource usage.
  2. Copy the four Opc.Ua.Encoders.Fuzz* folders to Opc.Ua.<NewArea>.Fuzz*. Rename the three csproj files and update <AssemblyName> (keep the Opc.Ua. prefix) and <RootNamespace>.
  3. Replace FuzzableCode.*.cs with your area's targets. Naming convention: Aflfuzz<Target>(Stream|string) for afl-fuzz and Libfuzz<Target>(ReadOnlySpan<byte>) for libFuzzer. Each target should swallow expected exceptions (ServiceResultException with BadDecodingError / BadEncodingLimitsExceeded, CryptographicException, FormatException, ArgumentException) and let unexpected ones bubble — that's what the fuzzer finds.
  4. Subclass the harness. Opc.Ua.<NewArea>.Fuzz.Tests/<NewArea>Tests.cs inherits from FuzzTargetTestsBase, sets FuzzableCodeType => typeof(FuzzableCode), and exposes the [DatapointSource] FuzzableFunctions field with CreateFuzzTargetFunctions(typeof(FuzzableCode)).
  5. Generate seeds. Add <NewArea>.Testcases.cs to Opc.Ua.<NewArea>.Fuzz.Tools/ that builds valid sample inputs through the real producer code and writes them under Opc.Ua.<NewArea>.Fuzz.Corpus/Testcases.<Bucket>/. Each Testcases.*/ subdirectory becomes a logical seed bucket and is auto-discovered by TestAssetUtils.DiscoverTestcaseEncoderSuffixes.
  6. Add a dictionary. Put <NewArea>.dict under fuzzing/Dictionaries/ with common tokens / magic bytes / length sentinels — this dramatically improves libFuzzer coverage progression.
  7. Wire into UA.slnx. Add the three new projects to the /fuzzing/ folder, the *.dict file under /fuzzing/Dictionaries/, and the seed loose files you want visible in the IDE.
  8. Update this Fuzzing.md. Add a row to the area table.

The Azure pipeline test template (.azurepipelines/test.yml) automatically runs every *.Fuzz.Tests project under the [Category("Fuzzing")] filter.

Areas in detail

Network / Transport area — Opc.Ua.Core.Diagnostics + Core UA-SC seam

The Network area is unusual because it's split across two complementary entry points:

  • Phase 4a (no Core changes). Fuzzes the public surfaces of src/Opc.Ua.Core.Diagnostics: OpcUaFrameParser.Process (TCP → UA-SC chunk splitter), TcpStreamReassembler.Process (raw TCP), OfflineSecureChannel.ReadChunk (UA-SC symmetric decrypt + verify using the stack's own UaSCUaBinaryChannel.ReadSymmetricMessage, so every security profile is covered for free), and ServiceCallReassembler.Push (chunk → service-call assembly with mixed sequence numbers, request ids, oversize bodies). Two stateful replay drivers (MockServerReplay, MockClientReplay) are also wired as libFuzzer-only targets but kept in a separate target list so the cheap stateless targets dominate throughput.

  • Phase 4b (internal Core seam). Adds internal static class TcpMessageParsers in src/Opc.Ua.Core/Stack/Tcp/TcpMessageParsers.cs covering the pre-crypto, pre-auth chunk surface the pcap binding does not expose: TryParseChunkHeader, ReadHelloMessage, ReadAcknowledgeMessage, ReadErrorMessage, ReadReverseHelloMessage, ReadAsymmetricMessageHeader. Surfaced to the fuzz area via <InternalsVisibleTo Include="Opc.Ua.Network.Fuzz" /> (and matching *.Tools / *.Tests assemblies because the linked partials compile from every host).

Seed-corpus and key-material discipline

Network.Testcases.cs and Transport.Testcases.cs generate seeds from a deterministic in-process handshake (Hello → OPN → MSG (Read/Browse) → CLO) using test certificates and the pcap binding's LoopbackFrameBuilder / CapturingMessageSocketFactory. Outputs:

  • raw TCP segments → Opc.Ua.Network.Fuzz.Corpus/Testcases.Tcp/
  • UA-SC chunks → Opc.Ua.Network.Fuzz.Corpus/Testcases.Chunks/, Testcases.Tcp.Hello/, Testcases.Tcp.Ack/, Testcases.Tcp.Err/, Testcases.Tcp.Rhe/, Testcases.Tcp.AsymHdr/
  • paired ChannelKeyMaterial JSON → Opc.Ua.Network.Fuzz.Corpus/Testcases.Keys/

Never commit real keylog material. Seed corpora use only key material generated from the existing fixture test certificates via the binding's own multi-TFM replay helpers.

Dependency hygiene

The Network fuzz host references Opc.Ua.Core.Diagnostics (which transitively pulls PacketDotNet + SharpPcap) but does not instantiate NicCaptureSource so the AFL/libFuzzer process never opens raw sockets.

Installation

Linux (afl-fuzz + libFuzzer)

Both fuzzers are supported on Linux. afl-fuzz can be compiled on any Linux system; for libFuzzer prebuilt binaries are available for Debian / Ubuntu / Windows from the libfuzzer-dotnet releases.

cd <repo>/fuzzing
sudo apt-get update
sudo apt-get install -y build-essential cmake git dotnet-sdk-10.0
# Powershell on Linux (required by the helper scripts):
# https://learn.microsoft.com/powershell/scripting/install/install-ubuntu
./Scripts/install.sh                    # builds afl-fuzz + installs SharpFuzz.CommandLine

install.sh downloads afl-2.52b, runs make install, then dotnet tool install --global SharpFuzz.CommandLine. Validate with:

afl-fuzz --help
sharpfuzz

Windows (libFuzzer via WSL or native)

Install the latest .NET 10 SDK / runtime, then:

dotnet tool install --global SharpFuzz.CommandLine

For afl-fuzz, use WSL with the Linux instructions above.

Running a fuzzer

The dynamic menu script lists every FuzzableCode static target in a built area assembly without hardcoding target names:

powershell -File fuzzing/Scripts/fuzz-menu.ps1 `
    -AssemblyPath fuzzing/Opc.Ua.Network.Fuzz/bin/Debug/net10.0/Opc.Ua.Network.Fuzz.dll
# -Filter <regex> narrows the list; -Index <n> selects a target without prompting.

Scripts/fuzz-libfuzzer.ps1 and Scripts/fuzz-afl.ps1 accept a -fuzztarget parameter matching one of the listed names:

cd fuzzing
powershell -File Scripts/fuzz-libfuzzer.ps1 `
    -libFuzzer ./libfuzzer-dotnet-windows.exe `
    -project ./Opc.Ua.Encoders.Fuzz/Opc.Ua.Encoders.Fuzz.csproj `
    -fuzztarget LibfuzzBinaryDecoder `
    -corpus ./Opc.Ua.Encoders.Fuzz.Corpus/Testcases.Binary/

The fuzzer runs until a crash / timeout / Ctrl-C. libFuzzer writes findings to the current directory with crash- / timeout- prefixes; afl-fuzz writes them to its findings/ directory.

Replay of crashes and timeouts

Run the area's *.Fuzz.Tools project with -p / -s for playback with stack traces:

dotnet run --project fuzzing/Opc.Ua.Network.Fuzz.Tools -- --playback --stacktrace

The playback tool finds all crashes / timeouts in the default folders and replays them against every libFuzzer target. Equivalent afl-fuzz seeds are skipped because they are duplicates of the libFuzzer findings.

Recreate or improve seeds

Run the area's *.Fuzz.Tools project with -t to (re)generate the area's Testcases.*/ seed corpus:

dotnet run --project fuzzing/Opc.Ua.Network.Fuzz.Tools -- --testcases

This re-runs the deterministic seed-generation pipeline (handshake recorder for Network, encoder round-trip for Encoders, cert/CRL/CSR builder for Certificates) and emits byte-stable artifacts the harness's Testcases.*/ discovery picks up automatically. The default output path for each area's Tools project is its own sibling Opc.Ua.<Area>.Fuzz.Corpus/Testcases folder, derived from the running Tools assembly name in Common/Fuzz.Tools/Program.cs.

Automation — fuzz-tester custom agent

.github/agents/fuzz-tester.agent.md defines a GitHub Copilot custom agent that drives this entire toolchain autonomously. It detects the host OS, picks the available engines (libFuzzer everywhere, afl-fuzz on Linux when installed), publishes + SharpFuzz-instruments each area's host project, enumerates FuzzableCode targets via Scripts/fuzz-menu.ps1, launches one detached fuzz process per (area × engine × target) tuple, and polls the per-instance work dirs under fuzzing/.runs/ for new findings.

When the agent picks up a novel crash-* / timeout-* / slow-unit-* (or afl crashes/ / hangs/) file it:

  1. SHA-1 dedups against every Opc.Ua.<area>.Fuzz.Tests/Assets/<prefix>-<sha1> already in the repo so it never re-investigates a known regression seed.
  2. Reproduces the crash locally via dotnet run --project fuzzing/Opc.Ua.<area>.Fuzz.Tools -- --playback --stacktrace.
  3. Designs a minimal fix following every repo guideline (no SYNC-over-ASYNC, Span<byte> / ReadOnlySpan<byte> / ByteString in public API, Allman + 4-space + CRLF + MIT header on new files, no #region, no [Obsolete] usage, NativeAOT-safe, TreatWarningsAsErrors clean, no exposed locks).
  4. Runs a rubber-duck agent review (max 2 rounds) on the proposed diff.
  5. Copies the failing input to fuzzing/Opc.Ua.<area>.Fuzz.Tests/Assets/<prefix>-<sha1> so the existing harness (FuzzCrashAssets / FuzzTimeoutAssets / FuzzSlowAssets) replays it through every target on every future dotnet test run.
  6. Rebuilds and runs the three *.Fuzz.Tests projects on net10.0; the run must end at or above the baseline 4681-test count, 0 failed.
  7. Briefly re-runs the originating fuzz instance with the new asset in its corpus to confirm the same crash is no longer reachable.
  8. Commits one fix per commit (Fuzz fix [<area>]: <root cause>), pushes to fuzzing, and resumes the still-running fuzz instances.

If a fix introduces a regression, requires a public-API change, or breaks compatibility with the 1.5.378 baseline, the agent pauses and prompts the user via ask_user with four options (revert / accept the regression and migrate the affected seed / provide an alternative fix / accept the API break).

Trigger phrases: "run the fuzz tests", "start fuzzing", "fuzz the encoders", "fuzz the network", "fuzz the certificates", "fuzz until I say stop", "react to fuzz findings", "find fuzz crashes and fix them", "run libfuzzer and fix what it finds", "autonomous fuzz loop". Stop the agent at any time with "stop", "halt", "stop fuzzing", "that's enough".

The agent's own toolchain setup mirrors the manual Installation section above: SharpFuzz.CommandLine global tool, the libfuzzer-dotnet driver under fuzzing/.tools/ (gitignored), and afl-fuzz on PATH for the AFL engine on Linux. If any are missing AND cannot be auto-installed (air-gapped box, sudo not available, …), the agent surfaces the gap and asks the user how to proceed.