# Description Moves the all-target-framework build, the cross-platform test matrix and the Native AoT run from GitHub Actions onto the Azure DevOps `netstandard` Managed DevOps Pool, and replaces Codecov with a coverage gate that runs inside the pipeline. ## 1. CI backend switch One conceptual switch decides who owns that work. It is checked into source in **two places that must be flipped together**: | File | Setting | Default | | --- | --- | --- | | `azure-pipelines.yml` | `parameters.ciBuildBackend` | `ado` | | `.github/workflows/buildandtest.yml` | `env.CI_BUILD_BACKEND` | `ado` | Setting both to `actions` restores the previous behaviour exactly. Azure Pipelines gains what it was missing to take the work over: - A **Linux all-TFM `sln.yml` leg**, replacing `build-all-tfm-linux`. `sln.yml` gets a `configurations` parameter so this leg builds `Release` only instead of doubling the managed-pool job count. - A **Linux agent in the `net10.0` `testfastpr` matrix**, replacing the ubuntu `build-and-test` job. - **Native AoT and Code Coverage now gate pull requests** instead of running only on Schedule/Manual. The coverage stage drops its `testdebug` dependency, which is Schedule/Manual-only and would otherwise have skipped coverage on every PR. - On a PR, Windows builds the .NET Framework TFMs and Linux the modern ones — mirroring the split the GitHub Actions jobs used — so all seven TFMs stay covered without fanning every TFM across both pools. Two things are deliberately **not** covered by the switch: - **macOS** stays on GitHub-hosted runners, because Managed DevOps Pools provide no macOS image. - **`master378` and `develop/*`** keep running the GitHub Actions jobs, since Azure Pipelines only builds `master`/`main` from this file. GitHub Actions does not expose the `env` context to job-level `if:` conditions, so the switch is resolved once in the existing `discover` job and re-exported as job outputs. A new **`build-and-test summary`** job gives branch protection one stable check to require, instead of the generated matrix job names. It treats an intentionally skipped job as success but still fails on a real failure or cancellation. The `paths:` allow-list was moved off the `pull_request` trigger and into `discover`. A workflow filtered out by `paths` never reports its checks, and a required check that never reports leaves a PR permanently *"Expected — waiting for status to be reported"*. That would have made the routine "ratchet the coverage floor" PR (which touches only `coverage-thresholds.json` and docs) unmergeable. ## 2. Coverage gate replacing Codecov Codecov is removed entirely — `codecov.yml`, both uploads, the badge, and the token usage. `.azurepipelines/check-coverage.ps1` runs at the end of the `Code Coverage` stage against the Cobertura report ReportGenerator already merges, driven by `coverage-thresholds.json`: | Check | Behaviour | | --- | --- | | Project floor | **Blocking** — total line and branch rates against an absolute floor. | | Patch coverage | **Blocking on PRs** — changed lines must reach `patch.target`, tolerating `patch.threshold` percentage points (the same semantics `codecov/patch` had). | | Baseline delta | **Advisory** — reports the change against the recorded baseline, never fails. | The project rate is recomputed from the per-line data rather than read off the Cobertura root element, so the `ignore` globs carried over from `codecov.yml` apply to the project floor as well as to the changed-lines gate — otherwise samples and generated code would count towards one gate but not the other. `testcc.yml` gains an explicit `checkout: self` with `fetchDepth: 0`, because the default shallow clone cannot diff against the base branch. ## Follow-up actions for maintainers Two items cannot be automated from source: 1. **Enable the `/azp run` gate.** In the Azure DevOps portal: pipeline → **More actions** → **Triggers** → **Pull request validation** → **Require a team member's comment before building a pull request**, scoped to *Only on pull requests from non-team members*. This makes Copilot-authored and outside-contributor PRs wait for a maintainer's `/azp run`, mirroring the approval GitHub Actions already requires. The documentation added here describes the workflow and the private-org-membership gotcha; **the setting itself still has to be switched on.** 2. **Update branch protection and secrets.** Drop the `codecov/project` and `codecov/patch` required checks, require `build-and-test summary` and the ADO coverage check instead, and remove `CODECOV_TOKEN` from the GitHub repository secrets and the ADO variable group. Also note the thresholds in `coverage-thresholds.json` are seeded from the last Codecov reading (73.61 %) with headroom, because the in-pipeline number had not been observed yet. The first full `Code Coverage` run on master should confirm it and the floors should be tightened then — the gate prints the exact figure it computes. ## Related Issues _No tracking issue; this is CI infrastructure work requested directly._ ## Checklist - [x] I have signed the [CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf) and read the [CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md) doc. - [x] I have added tests that prove my fix is effective or that my feature works and increased code coverage. - No product code changed, so there are no unit tests to add. The new logic was verified with purpose-built harnesses instead: - All 8 branch/event/backend combinations of the `discover` switch, confirming `master378` and `develop/*` are untouched and that `actions` restores the old behaviour. - 12 scenarios for `check-coverage.ps1`: the project floor, the patch gate, ignore globs applying to *both* gates, three path-normalization shapes, and the advisory delta. - 14 path-relevance cases confirming the in-job filter matches the retired `paths:` allow-list exactly. - End-to-end against a **real** ReportGenerator report: correctly resolved `src/Opc.Ua.PubSub/DependencyInjection/PubSubBuilder.cs`, found exactly the four modified tracked lines and reported 50 % patch coverage (2 covered, 2 uncovered). - [x] I have added all necessary documentation. - New **Continuous integration** section in `docs/DeveloperGuide.md`, a CI note in `CONTRIBUTING.md`, and links from `docs/README.md`. Also corrected the CI matrix description, which claimed the PR gate ran net48 + **net8.0** — it runs net48 + **net10.0**. - [x] I have verified that my changes do not introduce (new) build or analyzer warnings. - `dotnet build src/Opc.Ua.Server` → 0 warnings, 0 errors. No product code changed. - [ ] I ran **all** tests locally using the **UA.slnx** solution against at least .net **framework** and .net **10**, and all passed. - Not applicable — no product code changed. `Opc.Ua.Server.Tests` was run on net10.0 during development (3480 passed; the single `ServerFluentApiHostingTests` failure is pre-existing and reproduces on master). - [ ] I fixed **all** failing and flaky tests in the CI pipelines and **all** CodeQL warnings. - [x] I have addressed **all** PR feedback received. - A review pass over this branch flagged the `paths:`-filter deadlock described above; it is fixed in `1926e240e`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 181ba7ea-c72f-42d7-a737-00a3faadaa17 Copilot-Session: 26aa55a2-5731-4a4c-b61c-df2f7a785eca
15 KiB
Fuzz testing for UA.NET Standard
This project provides integration of SharpFuzz with the
UA.NET Standard libraries, with support for both afl-fuzz and
libFuzzer. Each fuzz area lives directly under fuzzing/
as four sibling folders — host (*.Fuzz), corpus (*.Fuzz.Corpus), test fixture (*.Fuzz.Tests),
and tools (*.Fuzz.Tools) — and shares a generic NUnit replay harness plus the SharpFuzz host
under Common/.
Areas
| Area | Surface | Status |
|---|---|---|
Opc.Ua.Encoders.Fuzz |
BinaryDecoder / JsonDecoder / XmlDecoder + idempotent round-trip, individual built-in type readers (NodeId, ExpandedNodeId, Variant, ExtensionObject, DataValue, DiagnosticInfo, QualifiedName, LocalizedText) and parser entry points (NodeId.Parse, ExpandedNodeId.Parse, RelativePathFormatter.Parse, QualifiedName.Parse, NumericRange.Parse, Uuid round-trip) |
shipped |
Opc.Ua.Certificates.Fuzz |
X509CRL decode + extensions (X509SubjectAltNameExtension, X509AuthorityKeyIdentifierExtension, X509CrlNumberExtension), PEMReader cert/key import, Pkcs10CertificationRequest, low-level AsnUtils helpers |
shipped |
Opc.Ua.Network.Fuzz |
OPC UA UA-SC / TCP framing via Opc.Ua.Core.Diagnostics (OpcUaFrameParser, TcpStreamReassembler, OfflineSecureChannel.ReadChunk, ServiceCallReassembler, MockServerReplay/MockClientReplay stateful drivers) and the Opc.Ua.Core UA-SC parse seam (TcpMessageParsers.TryParseChunkHeader / ReadHelloMessage / ReadAcknowledgeMessage / ReadErrorMessage / ReadReverseHelloMessage / ReadAsymmetricMessageHeader) |
shipped |
Encoders and Certificates build for the repo's standard TestsTargetFrameworks matrix
(net48, net8.0, net9.0, net10.0). Network is net8.0;net9.0;net10.0 only because
Opc.Ua.Core.Diagnostics does not target .NET Framework.
Directory layout
fuzzing/
Fuzzing.md # this file
Common/ # shared source (no csproj)
Fuzz/
Program.cs, FuzzMethods.cs # SharpFuzz host
Fuzz.Tests/
FuzzTargetTestsBase.cs, FuzzTargetFunction.cs
TestcaseAsset.cs, TestAssetUtils.cs # generic NUnit harness
Fuzz.Tools/
Program.cs, Playback.cs, Logging.cs, Testcases.cs # Tools host
Dictionaries/ # libFuzzer / afl dictionaries
asn1.dict binary.dict json.dict nodeid.dict
tcp.dict uasc.dict xml.dict
Scripts/ # area-agnostic runners
fuzz-afl.ps1 fuzz-libfuzzer.ps1 fuzz-menu.ps1
install.sh readme.txt
Opc.Ua.Encoders.Fuzz/ # csproj + FuzzableCode partials
Opc.Ua.Encoders.Fuzz.Corpus/ # seed corpus (Testcases.*/)
Opc.Ua.Encoders.Fuzz.Tests/ # deterministic NUnit replay
Opc.Ua.Encoders.Fuzz.Tools/ # corpus generator + playback
Opc.Ua.Certificates.Fuzz/ Opc.Ua.Certificates.Fuzz.Corpus/
Opc.Ua.Certificates.Fuzz.Tests/ Opc.Ua.Certificates.Fuzz.Tools/
Opc.Ua.Network.Fuzz/ Opc.Ua.Network.Fuzz.Corpus/
Opc.Ua.Network.Fuzz.Tests/ Opc.Ua.Network.Fuzz.Tools/
The flat layout matches the rest of the repository (tests/, src/, src/ also list
projects directly under the root with no domain grouping). Every csproj folder name equals the
csproj filename (without the .csproj extension), and every assembly is prefixed with Opc.Ua.
in line with the repo's AssemblyPrefix convention.
Each area's *.Fuzz project hosts FuzzableCode.*.cs partials whose public static methods
with a single parameter of type Stream (afl), string (afl), or ReadOnlySpan<byte>
(libFuzzer) are auto-discovered by:
- the SharpFuzz host (
Common/Fuzz/Program.cs) at fuzz-run time, and - the generic NUnit harness (
Common/Fuzz.Tests/FuzzTargetTestsBase.cs) at test time — every target is replayed against the area's*.Fuzz.Corpus/Testcases.*/corpus plus crash/timeout assets, with no per-area test code required.
Each area's *.Fuzz.Tests is therefore a single thin subclass:
[TestFixture]
[Category("Fuzzing")]
public class EncoderTests : FuzzTargetTestsBase
{
[DatapointSource]
public static readonly FuzzTargetFunction[] FuzzableFunctions =
CreateFuzzTargetFunctions(typeof(FuzzableCode));
protected override Type FuzzableCodeType => typeof(FuzzableCode);
}
How to add a new fuzz area
- Pick a surface. Anything that takes untrusted input (bytes, string, or stream) is a candidate. Prefer surfaces with deterministic output and bounded resource usage.
- Copy the four
Opc.Ua.Encoders.Fuzz*folders toOpc.Ua.<NewArea>.Fuzz*. Rename the three csproj files and update<AssemblyName>(keep theOpc.Ua.prefix) and<RootNamespace>. - Replace
FuzzableCode.*.cswith your area's targets. Naming convention:Aflfuzz<Target>(Stream|string)for afl-fuzz andLibfuzz<Target>(ReadOnlySpan<byte>)for libFuzzer. Each target should swallow expected exceptions (ServiceResultExceptionwithBadDecodingError/BadEncodingLimitsExceeded,CryptographicException,FormatException,ArgumentException) and let unexpected ones bubble — that's what the fuzzer finds. - Subclass the harness.
Opc.Ua.<NewArea>.Fuzz.Tests/<NewArea>Tests.csinherits fromFuzzTargetTestsBase, setsFuzzableCodeType => typeof(FuzzableCode), and exposes the[DatapointSource]FuzzableFunctionsfield withCreateFuzzTargetFunctions(typeof(FuzzableCode)). - Generate seeds. Add
<NewArea>.Testcases.cstoOpc.Ua.<NewArea>.Fuzz.Tools/that builds valid sample inputs through the real producer code and writes them underOpc.Ua.<NewArea>.Fuzz.Corpus/Testcases.<Bucket>/. EachTestcases.*/subdirectory becomes a logical seed bucket and is auto-discovered byTestAssetUtils.DiscoverTestcaseEncoderSuffixes. - Add a dictionary. Put
<NewArea>.dictunderfuzzing/Dictionaries/with common tokens / magic bytes / length sentinels — this dramatically improves libFuzzer coverage progression. - Wire into
UA.slnx. Add the three new projects to the/fuzzing/folder, the*.dictfile under/fuzzing/Dictionaries/, and the seed loose files you want visible in the IDE. - Update this
Fuzzing.md. Add a row to the area table.
The Azure pipeline test template (.azurepipelines/test.yml) automatically runs every
*.Fuzz.Tests project under the [Category("Fuzzing")] filter.
Areas in detail
Network / Transport area — Opc.Ua.Core.Diagnostics + Core UA-SC seam
The Network area is unusual because it's split across two complementary entry points:
-
Phase 4a (no Core changes). Fuzzes the public surfaces of
src/Opc.Ua.Core.Diagnostics:OpcUaFrameParser.Process(TCP → UA-SC chunk splitter),TcpStreamReassembler.Process(raw TCP),OfflineSecureChannel.ReadChunk(UA-SC symmetric decrypt + verify using the stack's ownUaSCUaBinaryChannel.ReadSymmetricMessage, so every security profile is covered for free), andServiceCallReassembler.Push(chunk → service-call assembly with mixed sequence numbers, request ids, oversize bodies). Two stateful replay drivers (MockServerReplay,MockClientReplay) are also wired as libFuzzer-only targets but kept in a separate target list so the cheap stateless targets dominate throughput. -
Phase 4b (internal Core seam). Adds
internal static class TcpMessageParsersinsrc/Opc.Ua.Core/Stack/Tcp/TcpMessageParsers.cscovering the pre-crypto, pre-auth chunk surface the pcap binding does not expose:TryParseChunkHeader,ReadHelloMessage,ReadAcknowledgeMessage,ReadErrorMessage,ReadReverseHelloMessage,ReadAsymmetricMessageHeader. Surfaced to the fuzz area via<InternalsVisibleTo Include="Opc.Ua.Network.Fuzz" />(and matching*.Tools/*.Testsassemblies because the linked partials compile from every host).
Seed-corpus and key-material discipline
Network.Testcases.cs and Transport.Testcases.cs generate seeds from a deterministic
in-process handshake (Hello → OPN → MSG (Read/Browse) → CLO) using test certificates and
the pcap binding's LoopbackFrameBuilder / CapturingMessageSocketFactory. Outputs:
- raw TCP segments →
Opc.Ua.Network.Fuzz.Corpus/Testcases.Tcp/ - UA-SC chunks →
Opc.Ua.Network.Fuzz.Corpus/Testcases.Chunks/,Testcases.Tcp.Hello/,Testcases.Tcp.Ack/,Testcases.Tcp.Err/,Testcases.Tcp.Rhe/,Testcases.Tcp.AsymHdr/ - paired
ChannelKeyMaterialJSON →Opc.Ua.Network.Fuzz.Corpus/Testcases.Keys/
Never commit real keylog material. Seed corpora use only key material generated from the existing fixture test certificates via the binding's own multi-TFM replay helpers.
Dependency hygiene
The Network fuzz host references Opc.Ua.Core.Diagnostics (which transitively pulls
PacketDotNet + SharpPcap) but does not instantiate NicCaptureSource so the
AFL/libFuzzer process never opens raw sockets.
Installation
Linux (afl-fuzz + libFuzzer)
Both fuzzers are supported on Linux. afl-fuzz can be compiled on any Linux system; for libFuzzer prebuilt binaries are available for Debian / Ubuntu / Windows from the libfuzzer-dotnet releases.
cd <repo>/fuzzing
sudo apt-get update
sudo apt-get install -y build-essential cmake git dotnet-sdk-10.0
# Powershell on Linux (required by the helper scripts):
# https://learn.microsoft.com/powershell/scripting/install/install-ubuntu
./Scripts/install.sh # builds afl-fuzz + installs SharpFuzz.CommandLine
install.sh downloads afl-2.52b, runs make install, then
dotnet tool install --global SharpFuzz.CommandLine. Validate with:
afl-fuzz --help
sharpfuzz
Windows (libFuzzer via WSL or native)
Install the latest .NET 10 SDK / runtime, then:
dotnet tool install --global SharpFuzz.CommandLine
For afl-fuzz, use WSL with the Linux instructions above.
Running a fuzzer
The dynamic menu script lists every FuzzableCode static target in a built area assembly
without hardcoding target names:
powershell -File fuzzing/Scripts/fuzz-menu.ps1 `
-AssemblyPath fuzzing/Opc.Ua.Network.Fuzz/bin/Debug/net10.0/Opc.Ua.Network.Fuzz.dll
# -Filter <regex> narrows the list; -Index <n> selects a target without prompting.
Scripts/fuzz-libfuzzer.ps1 and Scripts/fuzz-afl.ps1 accept a -fuzztarget parameter
matching one of the listed names:
cd fuzzing
powershell -File Scripts/fuzz-libfuzzer.ps1 `
-libFuzzer ./libfuzzer-dotnet-windows.exe `
-project ./Opc.Ua.Encoders.Fuzz/Opc.Ua.Encoders.Fuzz.csproj `
-fuzztarget LibfuzzBinaryDecoder `
-corpus ./Opc.Ua.Encoders.Fuzz.Corpus/Testcases.Binary/
The fuzzer runs until a crash / timeout / Ctrl-C. libFuzzer writes findings to the current
directory with crash- / timeout- prefixes; afl-fuzz writes them to its findings/
directory.
Replay of crashes and timeouts
Run the area's *.Fuzz.Tools project with -p / -s for playback with stack traces:
dotnet run --project fuzzing/Opc.Ua.Network.Fuzz.Tools -- --playback --stacktrace
The playback tool finds all crashes / timeouts in the default folders and replays them against every libFuzzer target. Equivalent afl-fuzz seeds are skipped because they are duplicates of the libFuzzer findings.
Recreate or improve seeds
Run the area's *.Fuzz.Tools project with -t to (re)generate the area's Testcases.*/
seed corpus:
dotnet run --project fuzzing/Opc.Ua.Network.Fuzz.Tools -- --testcases
This re-runs the deterministic seed-generation pipeline (handshake recorder for Network,
encoder round-trip for Encoders, cert/CRL/CSR builder for Certificates) and emits
byte-stable artifacts the harness's Testcases.*/ discovery picks up automatically. The
default output path for each area's Tools project is its own sibling
Opc.Ua.<Area>.Fuzz.Corpus/Testcases folder, derived from the running Tools assembly name
in Common/Fuzz.Tools/Program.cs.
Automation — fuzz-tester custom agent
.github/agents/fuzz-tester.agent.md defines a GitHub Copilot custom agent that drives
this entire toolchain autonomously. It detects the host OS, picks the available engines
(libFuzzer everywhere, afl-fuzz on Linux when installed), publishes + SharpFuzz-instruments
each area's host project, enumerates FuzzableCode targets via
Scripts/fuzz-menu.ps1, launches one detached fuzz process per (area × engine × target)
tuple, and polls the per-instance work dirs under fuzzing/.runs/ for new findings.
When the agent picks up a novel crash-* / timeout-* / slow-unit-* (or afl
crashes/ / hangs/) file it:
- SHA-1 dedups against every
Opc.Ua.<area>.Fuzz.Tests/Assets/<prefix>-<sha1>already in the repo so it never re-investigates a known regression seed. - Reproduces the crash locally via
dotnet run --project fuzzing/Opc.Ua.<area>.Fuzz.Tools -- --playback --stacktrace. - Designs a minimal fix following every repo guideline (no SYNC-over-ASYNC,
Span<byte>/ReadOnlySpan<byte>/ByteStringin public API, Allman + 4-space + CRLF + MIT header on new files, no#region, no[Obsolete]usage, NativeAOT-safe,TreatWarningsAsErrorsclean, no exposed locks). - Runs a rubber-duck agent review (max 2 rounds) on the proposed diff.
- Copies the failing input to
fuzzing/Opc.Ua.<area>.Fuzz.Tests/Assets/<prefix>-<sha1>so the existing harness (FuzzCrashAssets/FuzzTimeoutAssets/FuzzSlowAssets) replays it through every target on every futuredotnet testrun. - Rebuilds and runs the three
*.Fuzz.Testsprojects onnet10.0; the run must end at or above the baseline 4681-test count, 0 failed. - Briefly re-runs the originating fuzz instance with the new asset in its corpus to confirm the same crash is no longer reachable.
- Commits one fix per commit (
Fuzz fix [<area>]: <root cause>), pushes tofuzzing, and resumes the still-running fuzz instances.
If a fix introduces a regression, requires a public-API change, or breaks compatibility
with the 1.5.378 baseline, the agent pauses and prompts the user via ask_user
with four options (revert / accept the regression and migrate the affected seed /
provide an alternative fix / accept the API break).
Trigger phrases: "run the fuzz tests", "start fuzzing", "fuzz the encoders", "fuzz the network", "fuzz the certificates", "fuzz until I say stop", "react to fuzz findings", "find fuzz crashes and fix them", "run libfuzzer and fix what it finds", "autonomous fuzz loop". Stop the agent at any time with "stop", "halt", "stop fuzzing", "that's enough".
The agent's own toolchain setup mirrors the manual Installation section above:
SharpFuzz.CommandLine global tool, the libfuzzer-dotnet driver under
fuzzing/.tools/ (gitignored), and afl-fuzz on PATH for the AFL engine on Linux.
If any are missing AND cannot be auto-installed (air-gapped box, sudo not available,
…), the agent surfaces the gap and asks the user how to proceed.