Files
Marc Schier d764161a53 Extract core/types fixes for schemas, replay, and source generation (#4801)
# Description

Extract the reusable core/types layer from #4462 without including
UaLens or the other planned extraction layers.

- Preserve null versus empty in `ByteString.Copy`, and provide
older-framework path, random-byte, and SHA-256 convenience APIs.
- Encode abstract numeric structure fields (`Number`, `Integer`, and
`UInteger`) as Variants in JSON, XSD, and BSD schemas, with regression
coverage and schema documentation.
- Add opt-in complete-file validation for finalized PCAP replay while
retaining the existing tolerant reader default.
- Keep placeholder references on generated type templates, not runtime
instances, while preserving ordinary external references.

**Initial extraction:** commit
`8881e096fb5e4eaa36f3c13f8caa57733373bd19` preserves the assigned
changes from source `390c4a7a51b71da40caec2d22ef74518bbe670ce`, based on
pinned `master` commit `5b03e9afa492cdb36049ee52ae844f630f89d8db`: 20
exact full-file extractions and only the generic replay paragraph in
`docs/Diagnostics.md`. The UaLens capture-window paragraph remains with
the source PR. No UaLens projects, UI, native-rendering paths,
dependencies, or workflows are included.

**Review follow-up:** `814c0f2dc6056ea5eb898611565b3e96cfa65104` fixes
direct-placeholder references that bypassed filtering through
`TargetPath` and emitted an undefined `forInstance` guard in
instance-only factories. Filtering now follows target resolution.
Compilation/reference-shape regressions cover ObjectType/VariableType,
both placeholder rules, and direct-child/external targets;
`docs/NodeManagers.md` documents the behavior. The review thread was
resolved after publication and verification.

**CI follow-up:** `93ec3928bd5a9cb2ea86110085e79131db17d443` addresses
both causes found in [CI run
38700770996](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/38700770996):

- `MODELGEN012` treated an identical payload-less transitive dependency
declaration as a competing model producer after the SDK Programs model
appeared in newer `master`. Matching closure rows no longer emit that
diagnostic. Producer ordering is unchanged; actual producers, differing
declarations, and malformed payload handling retain their existing
diagnostics. New regression cases exercise both reference orders, and
`docs/ModelDependencies.md` records the distinction.
- The queue-fairness test sampled OS scheduling and failed on macOS.
This reuses the exact one-file correction from
`84d2c34d3902bfe700a87f6c30708b435500878f` in #4765: hold the existing
heavy-owner publication hook, prove concurrent work-conserving progress,
then assert exact recovered 1:3 service, all 960 unique requests, and
complete queue/token/owner/lease cleanup. No production scheduler or
fairness bound was changed.

The PR now changes 26 paths. Its pinned extraction ancestry is
preserved: no rebase, force push, or merge of newer `master` was
performed. This remains the bottom of native stack 4807, ordered 4801 ->
4802 -> 4803 -> 4804 -> 4806.

### Current CI-repair validation

Windows, existing .NET SDK **10.0.401**, **Release**, with analyzers
enabled. CI tested merge `9a2d252793da12aeb99ed379dfc9b03e89c912b7`
(master `d40843c23a5fe8116ffe380cf937d88759f5218b` plus PR head
`814c0f2dc6056ea5eb898611565b3e96cfa65104`), which includes newer tests
absent from the pinned extraction branch. That exact merge was
reproduced in a disposable detached worktree, then the proposed patch
was applied. Git patch-id verified that the tested patch and committed
branch patch were identical
(`b7ed3ba2b61d1862825dd33d93c50c64d3a759ea`).

Before the fix, all four failing client-accessor cases reproduced
`MODELGEN012`, and both new matching-reexport cases failed. After the
fix:

| Context / scope | Framework | Passed | Failed | Skipped |
| --- | --- | ---: | ---: | ---: |
| Exact CI merge + patch: client-accessor and dependency fixtures |
net10.0 | 32 | 0 | 0 |
| Same fixtures with CI coverage instrumentation | net10.0 | 32 | 0 | 0
|
| Exact CI merge + patch: entire SourceGeneration.Tests project |
net10.0 | 295 | 0 | 0 |
| Exact CI merge + patch: entire SourceGeneration.Tests project | net48
| 295 | 0 | 0 |
| Preserved PR branch: dependency and placeholder fixtures | net10.0 |
30 | 0 | 0 |
| Preserved PR branch: dependency and placeholder fixtures | net48 | 30
| 0 | 0 |
| Preserved PR branch: entire FairRequestQueueTests fixture, with
coverage | net10.0 | 93 | 0 | 0 |
| Preserved PR branch: entire FairRequestQueueTests fixture | net48 | 93
| 0 | 0 |
| Exact CI merge + patch: entire FairRequestQueueTests fixture, with
coverage | net10.0 | 93 | 0 | 0 |

All nine commands exited **0**. The repository TRX verdict helpers
verified **993 passing test executions, 0 failures, and 0 fixture
setup/teardown failures** across these overlapping runs. No
build/analyzer warnings were reported. Normal restore resolved missing
assets/target entries in the new checkout and when switching TFMs; no
dependency declaration or SDK pin was changed.

Exact successful test argument sets follow. Variables abbreviate the
executable, project paths, and results directory only; output was also
captured to separate local logs.

```powershell
$env:DOTNET_ROOT = 'D:\git\UA-.NETStandard_master\TestResults\ualens-528-deep-repro\sdk'
$env:PATH = "$env:DOTNET_ROOT;$env:PATH"
$dotnet = "$env:DOTNET_ROOT\dotnet.exe"
$generator = 'tests\Opc.Ua.SourceGeneration.Tests\Opc.Ua.SourceGeneration.Tests.csproj'
$coreTests = 'tests\Opc.Ua.Core.Tests\Opc.Ua.Core.Tests.csproj'
$results = 'C:\Users\mschier\.copilot\session-state\8f497e89-7fc2-4f48-880b-4803d97c1c12\files\ci-validation'

# In the disposable exact-CI-merge checkout, with the identical proposed patch applied:
Set-Location 'D:\Temp\ua-ci4801-8f497e89'

# Passed: 32, Failed: 0, Skipped: 0
& $dotnet test $generator --no-restore --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~ClientAccessorGeneratorTests|FullyQualifiedName~ModelDependencyScannerTests' --logger 'trx;LogFileName=ci-metadata-green-net10.trx' --results-directory $results --verbosity minimal

# Passed: 295, Failed: 0, Skipped: 0; reuses the analyzer-enabled build above
& $dotnet test $generator --no-build --no-restore --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 --logger 'trx;LogFileName=ci-sourcegeneration-merge-net10.trx' --results-directory $results --verbosity quiet

# Passed: 295, Failed: 0, Skipped: 0
& $dotnet test $generator --configuration Release --framework net48 -p:CustomTestTarget=net48 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --logger 'trx;LogFileName=ci-sourcegeneration-merge-net48.trx' --results-directory $results --verbosity minimal

# Passed: 93, Failed: 0, Skipped: 0
& $dotnet test $coreTests --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true -p:CollectCoverage=true --collect 'XPlat Code Coverage' --settings 'tests\coverlet.runsettings.xml' --filter 'FullyQualifiedName~FairRequestQueueTests' --logger 'trx;LogFileName=ci-fairqueue-merge-net10.trx' --results-directory $results --verbosity minimal

# Passed: 32, Failed: 0, Skipped: 0
& $dotnet test $generator --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true -p:CollectCoverage=true --collect 'XPlat Code Coverage' --settings 'tests\coverlet.runsettings.xml' --filter 'FullyQualifiedName~ClientAccessorGeneratorTests|FullyQualifiedName~ModelDependencyScannerTests' --logger 'trx;LogFileName=ci-metadata-coverage-net10.trx' --results-directory $results --verbosity minimal

# In the preserved PR branch worktree:
Set-Location 'D:\git\copilot-worktrees\UA-wot-d3-final\marcschier-congenial-umbrella'

# Passed: 30, Failed: 0, Skipped: 0
& $dotnet test $generator --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~ModelDependencyScannerTests|FullyQualifiedName~PlaceholderReferenceGeneratorTests' --logger 'trx;LogFileName=ci-sourcegeneration-branch-net10.trx' --results-directory $results --verbosity minimal

# Passed: 93, Failed: 0, Skipped: 0
& $dotnet test $coreTests --no-restore --configuration Release --framework net10.0 -p:CustomTestTarget=net10.0 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true -p:CollectCoverage=true --collect 'XPlat Code Coverage' --settings 'tests\coverlet.runsettings.xml' --filter 'FullyQualifiedName~FairRequestQueueTests' --logger 'trx;LogFileName=ci-fairqueue-branch-net10.trx' --results-directory $results --verbosity minimal

# Passed: 93, Failed: 0, Skipped: 0
& $dotnet test $coreTests --configuration Release --framework net48 -p:CustomTestTarget=net48 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~FairRequestQueueTests' --logger 'trx;LogFileName=ci-fairqueue-branch-net48.trx' --results-directory $results --verbosity minimal

# Passed: 30, Failed: 0, Skipped: 0
& $dotnet test $generator --configuration Release --framework net48 -p:CustomTestTarget=net48 -p:RunAnalyzers=true -p:RunAnalyzersDuringBuild=true --filter 'FullyQualifiedName~ModelDependencyScannerTests|FullyQualifiedName~PlaceholderReferenceGeneratorTests' --logger 'trx;LogFileName=ci-sourcegeneration-branch-net48.trx' --results-directory $results --verbosity minimal
```

`git diff --cached --check` passed. The queue-test blob was verified
byte-for-byte against the existing published correction. The upstream
branch was verified as `93ec3928bd5a9cb2ea86110085e79131db17d443` after
push.

**Remote CI is pending, not claimed green:** [new CI run
38712127138](https://github.com/OPCFoundation/UA-.NETStandard/actions/runs/38712127138).
Cross-platform confirmation, CodeQL, coverage gates, and the complete
`UA.slnx` suites on this new head are not established by these local
checks. The source PR's checks are not claimed as evidence for this
branch.

### Earlier validation

The original extraction had 1,407 passing scoped test executions across
net10.0/net48. The placeholder-review follow-up had six passing focused
cases and 269 passing tests in the full source-generator project on each
framework, with analyzer-enabled builds. Those are historical results
for their respective commits, not fresh full-suite evidence for the
CI-repair head. Earlier command details remain in this PR's edit
history.

## Related Issues

- Related to #4461 (existing tracking issue; not auto-closing).
- Extracted from #4462 (source PR; remains open and is not
auto-closing).
- Reuses the bounded queue-test correction from #4765; does not
incorporate that PR's other changes.

_If this is a relatively large or complex change, a design must have
been discussed in the related tracking issue and signed off (which
becomes the Architectural Decision Record (ADR))._

This began as an exact extraction of existing behavior. The subsequent
commits address the reported placeholder-generation defect and observed
CI failures; they are not a new design or a claim of design sign-off.

## Checklist

_Put an `x` in the boxes that apply. You can complete these step by step
after opening the PR._

- [ ] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [ ] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [x] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [x] I have addressed **all** PR feedback received.
2026-10-15 12:21:46 +02:00

237 lines
12 KiB
C#

/* ========================================================================
* Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved.
*
* OPC Foundation MIT License 1.00
*
* Permission is hereby granted, free of charge, to any person
* obtaining a copy of this software and associated documentation
* files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following
* conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*
* The complete license agreement can be found here:
* http://opcfoundation.org/License/MIT/1.00/
* ======================================================================*/
using System.Linq;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Xml.Linq;
using Json.Schema;
using NUnit.Framework;
using Opc.Ua.Schema.Bsd;
using Opc.Ua.Schema.Json;
using Opc.Ua.Schema.Xsd;
namespace Opc.Ua.Schema.Tests
{
/// <summary>
/// Verifies the Variant wire encoding of abstract numeric structure fields.
/// </summary>
[TestFixture]
public sealed class AbstractNumericSchemaTests
{
[Test]
public void AbstractNumericScalarAndArrayFieldsUseVariantEncoding(
[Values(BuiltInType.Number, BuiltInType.Integer, BuiltInType.UInteger)] BuiltInType builtInType,
[Values(UaSchemaFormat.JsonCompact, UaSchemaFormat.JsonVerbose, UaSchemaFormat.Xsd, UaSchemaFormat.Bsd)]
UaSchemaFormat format)
{
UaTypeDescription type = NumericType(builtInType);
IUaSchema schema = CreateProvider(type).CreateSchema(type, format);
if (format is UaSchemaFormat.JsonCompact or UaSchemaFormat.JsonVerbose)
{
JsonNode document = JsonNode.Parse(schema.ToSchemaString())!;
JsonNode properties = document["$defs"]!["NumericFields"]!["properties"]!;
JsonObject scalarSchema = WithoutNullBranch(properties["Scalar"]!);
JsonObject arraySchema = WithoutNullBranch(properties["Array"]!);
JsonObject elementSchema = WithoutNullBranch(arraySchema["items"]!);
Assert.That(scalarSchema.ContainsKey("$ref"), Is.True,
"Abstract numeric fields must not become unconstrained schemas.");
Assert.That(scalarSchema["$ref"]!.GetValue<string>(), Is.EqualTo("#/$defs/Ua_Variant"));
Assert.That(elementSchema["$ref"]!.GetValue<string>(),
Is.EqualTo("#/$defs/Ua_Variant"));
Assert.That(document["$defs"]!["Ua_Variant"], Is.Not.Null);
}
else
{
var document = XDocument.Parse(schema.ToSchemaString());
string fieldTag = format == UaSchemaFormat.Xsd ? "element" : "Field";
string nameAttribute = format == UaSchemaFormat.Xsd ? "name" : "Name";
string typeAttribute = format == UaSchemaFormat.Xsd ? "type" : "TypeName";
XElement scalar = document.Descendants().Single(element =>
element.Name.LocalName == fieldTag && (string?)element.Attribute(nameAttribute) == "Scalar");
XElement array = document.Descendants().Single(element =>
element.Name.LocalName == fieldTag && (string?)element.Attribute(nameAttribute) == "Array");
if (format == UaSchemaFormat.Xsd)
{
array = array.Descendants().Single(element => element.Name.LocalName == "element");
}
Assert.That((string?)scalar.Attribute(typeAttribute), Is.EqualTo("ua:Variant"));
Assert.That((string?)array.Attribute(typeAttribute), Is.EqualTo("ua:Variant"));
}
}
[Test]
public void AbstractNumericJsonSchemaValidatesEncodedVariantsAndRejectsBareValues(
[Values(BuiltInType.Number, BuiltInType.Integer, BuiltInType.UInteger)] BuiltInType builtInType,
[Values(false, true)] bool verbose)
{
UaTypeDescription type = NumericType(builtInType);
IUaSchema schema = CreateProvider(type).GetJsonSchema(type, verbose);
Variant value = builtInType switch
{
BuiltInType.Integer => Variant.From(-42L),
BuiltInType.UInteger => Variant.From(ulong.MaxValue),
_ => Variant.From(12.5d)
};
JsonNode encoded = Encode(value, verbose);
EvaluationResults valid = Evaluate(schema, encoded);
Assert.That(valid.IsValid, Is.True, valid.ToString());
Assert.That(encoded["Scalar"]!["UaType"], Is.Not.Null);
Assert.That(encoded["Scalar"]!["Value"], Is.Not.Null);
JsonNode bareScalar = encoded.DeepClone();
bareScalar["Scalar"] = 12.5;
Assert.That(Evaluate(schema, bareScalar).IsValid, Is.False);
JsonNode bareArrayElement = encoded.DeepClone();
bareArrayElement["Array"]![0] = 12.5;
Assert.That(Evaluate(schema, bareArrayElement).IsValid, Is.False);
}
[TestCase(false)]
[TestCase(true)]
public void NumericVariantDefaultPayloadUsesTheDeclaredWireType(bool verbose)
{
UaTypeDescription type = NumericType(BuiltInType.Number);
IUaSchema schema = CreateProvider(type).GetJsonSchema(type, verbose);
JsonNode encoded = Encode(Variant.From(0d), verbose);
EvaluationResults result = Evaluate(schema, encoded);
Assert.That(result.IsValid, Is.True, result.ToString());
Assert.That(encoded["Scalar"]!["UaType"]!.GetValue<int>(), Is.EqualTo(11));
Assert.That(encoded["Scalar"]!["Value"]!.GetValue<double>(), Is.Zero);
Assert.That(encoded["Array"]![0]!["Value"]!.GetValue<double>(), Is.Zero);
}
[Test]
public void AbstractNumericJsonSchemaRequiresTheCurrentVariantEnvelope()
{
UaTypeDescription type = NumericType(BuiltInType.Number);
IUaSchema schema = CreateProvider(type).GetJsonSchema(type);
JsonNode missingType = Encode(Variant.From(12.5d), verbose: false);
missingType["Scalar"]!.AsObject().Remove("UaType");
JsonNode legacyEnvelope = Encode(Variant.From(12.5d), verbose: false);
legacyEnvelope["Scalar"] = new JsonObject { ["Type"] = 11, ["Body"] = 12.5 };
JsonNode abstractTypeTag = Encode(Variant.From(12.5d), verbose: false);
abstractTypeTag["Scalar"]!["UaType"] = 26;
Assert.That(Evaluate(schema, missingType).IsValid, Is.False);
Assert.That(Evaluate(schema, legacyEnvelope).IsValid, Is.False);
Assert.That(Evaluate(schema, abstractTypeTag).IsValid, Is.False);
}
[Test]
public void FieldEncodingClassificationPreservesPrimitiveAliasesAndCustomIdentity()
{
Assert.That(SchemaTypeInfo.GetFieldEncodingType(DataTypeIds.Number), Is.EqualTo(BuiltInType.Variant));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(DataTypeIds.Integer), Is.EqualTo(BuiltInType.Variant));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(DataTypeIds.UInteger), Is.EqualTo(BuiltInType.Variant));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(DataTypeIds.Duration), Is.EqualTo(BuiltInType.Double));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(DataTypeIds.DiagnosticInfo),
Is.EqualTo(BuiltInType.DiagnosticInfo));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(DataTypeIds.Enumeration),
Is.EqualTo(BuiltInType.Enumeration));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(new NodeId(26u, 1)), Is.EqualTo(BuiltInType.Null));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(new NodeId("26", 0)), Is.EqualTo(BuiltInType.Null));
Assert.That(SchemaTypeInfo.GetFieldEncodingType(NodeId.Null), Is.EqualTo(BuiltInType.Null));
Assert.That(TypeInfo.GetBuiltInType(DataTypeIds.Number), Is.EqualTo(BuiltInType.Null));
}
[Test]
public void NumericIdentifiersInAnotherNamespaceResolveAsCustomStructures(
[Values(26u, 27u, 28u)] uint identifier)
{
UaTypeDescription custom = SchemaTestData.Structure(
identifier, "ApplicationNumber",
SchemaTestData.Field("Value", DataTypeIds.Int32));
UaTypeDescription root = SchemaTestData.Structure(
3951, "CustomRoot",
SchemaTestData.Field("Value", new NodeId(identifier, SchemaTestData.TestNamespaceIndex)));
var registry = new DataTypeDefinitionRegistry();
registry.Add(custom);
registry.Add(root);
var provider = new DefaultSchemaProvider(registry, [new JsonSchemaGenerator()]);
JsonNode schema = JsonNode.Parse(provider.GetJsonSchema(root).ToSchemaString())!;
Assert.That(schema["$defs"]!["CustomRoot"]!["properties"]!["Value"]!["$ref"]!.GetValue<string>(),
Is.EqualTo("#/$defs/ApplicationNumber"));
}
private static JsonObject WithoutNullBranch(JsonNode schema)
{
if (schema["anyOf"] is JsonArray alternatives)
{
Assert.That(alternatives, Has.Count.EqualTo(2));
Assert.That(alternatives[1]!["type"]!.GetValue<string>(), Is.EqualTo("null"));
return alternatives[0]!.AsObject();
}
return schema.AsObject();
}
private static UaTypeDescription NumericType(BuiltInType builtInType)
{
NodeId dataType = SchemaTestData.BuiltIn(builtInType);
return SchemaTestData.Structure(
3950, "NumericFields",
SchemaTestData.Field("Scalar", dataType),
SchemaTestData.Field("Array", dataType, ValueRanks.OneDimension));
}
private static DefaultSchemaProvider CreateProvider(UaTypeDescription type)
{
var registry = new DataTypeDefinitionRegistry();
registry.Add(type);
return new DefaultSchemaProvider(
registry, [new JsonSchemaGenerator(), new XsdSchemaGenerator(), new BsdSchemaGenerator()]);
}
private static JsonNode Encode(Variant value, bool verbose)
{
using var encoder = new JsonEncoder(
ServiceMessageContext.Create(null), verbose ? JsonEncoderOptions.Verbose : JsonEncoderOptions.Compact);
encoder.WriteVariant("Scalar", value);
encoder.WriteVariantArray("Array", [value, Variant.Null]);
return JsonNode.Parse(encoder.CloseAndReturnText())!;
}
private static EvaluationResults Evaluate(IUaSchema schema, JsonNode instance)
{
JsonSchema compiled = JsonSchema.FromText(
schema.ToSchemaString(), new BuildOptions { SchemaRegistry = new SchemaRegistry() });
return compiled.Evaluate(
JsonSerializer.SerializeToElement(instance),
new EvaluationOptions { OutputFormat = OutputFormat.List });
}
}
}