- Add a solution to provide fuzzer support for afl-fuzz and libfuzzer.
- Add fuzz targets for Binary, Json and Xml encoder and decoder fuzzing. For afl-fuzz and libfuzzer.
- Fix issues in encoders and decoders when unexpected exceptions are returned. The expectation is to return only ServiceResultException with BadDecodingError, BadEncodingError or BadEncodingLimitsExceeded.
- Add a tool to create Testcases.
- Add a tool to playback crashes and timeouts.
- Solution is extensible with common files to discover new fuzz targets by reflection. New fuzz projects can share many files with the first project.
- Breaking change: Fix how BinaryDecoder is used: Previously, calling close on the BinaryDecoder implicitly disposed the stream. Change behavior so that `using` pattern can be applied across the board.
* Added MaxChannelCount and detect and cleanup "stale" channels based on ChannelLifetime
* Add Environment.TickCount as time reference
* Set a socket backlog value to avoid socket open overflow
* Added a ChannelKeepAlive to signal when a session is still active
* Add options to console client to leak channels and run connect loop forever
Co-authored-by: Martin Regen <mregen@microsoft.com>
- Implement `int Read(Span<byte> buffer)` and `void Write(ReadOnlySpan<byte> buffer)` in ArraySegmentStream
- for ArraySegmentStream streaming lower the CPU load for .NET8 OPC UA binary encoding/decoding by 50%
- Implement dispose to return buffers that were not transferred.
- Implement a GetSequence in ArraySegmentStream
Added redaction (disabled by default)
Added redaction strategies for username, password, endpoint, and exception message.
Updated logging to use redaction where it was critical
In #2510 a message type check was added to catch non UA service connections earlier.
But the Abort message type was missing in the check, so an Abort caused a disconnect.
However, in such a case the transport can just continue with the next message.
Fix allowed Roles of Method Unregister Application allow Application Self Admin Privilge:
https://reference.opcfoundation.org/GDS/v105/docs/6.6.8
Add Parameter NoPrivateKeys to OpenStoreMethod of CertificateStoreIdentifier to allow GDS client to update the private key of a store which is not its own store.
NodeState classes are not thread safe. In many places event handlers are handled in a manner that can cause NullReferenceException in a case of a bad timing. In addition, in some places non-thread-safe collections are iterated which could cause issues if the collections are changed at the same time.
This PR addresses the aforementioned issues by using local variables for event handlers and then making null check and calling them. When applicable null-conditional operators are used. Non-thread-safe collection iteration is guarded by locking. The issues were reproduces with unit tests part of this PR and also the fix was verified by running the same unit tests.
During high load there were a few portions in the send/receive pipeline identified which may cause lock contention and thread starvation due to locks:
- Remove lock for the codepath when a incoming message is processed.
- Add a concurrent dictionary to allow for the processing of OperationCompleted and to add new requests without locking
- Allow to process normal service requests without locking
Due to the change it is possible to send KeepAlive even when a incoming message is received.
-Implement the support for ApplicationSelfAdmin privilege
-implement the workflow in a test to pull the trust list after sucessfully receiving a gds signed certificate.
-this authorization is described here in the OPC Spec: https://reference.opcfoundation.org/GDS/v105/docs/7.6
- only republish available sequence numbers
- store republish result to handle unsuccessful message republish in main loop
- republish in parallel instead of sequential to reduce response times / blocking in the publish response processing loop
EscapeString is a function which parses every string for special characters, but often times it is already known that there are no escape characters to process. Improve WriteSimpleField to only call JsonEncoder.EscapeString if necessary.
-- note: fixed a few csproj files
Co-authored-by: Bhupendra Naphade <bhnaphad@microsoft.com>
Update the function to improve performance.
Fix test code warnings (CA1305) related to ToString() method
Based on the results of the benchmark runs, `EscapeStringLegacyPlus` was picked for all platforms that do not support `StreamWriter.Write(ReadOnlySpan<char>)`, e.g. .NET Framework. There is a slight improvement for the cases which require a 4 digit hex encoding.
For platforms that support Span end to end `EscapeStringSpanCharsInline` was picked. The functions tries to optimize how often the underlying StreamWriter is called by combining calls to Write into to slices. It was also noted that up to three character strings, it performs better to do single character writes than calling with a small string.
Reduce lock contention for encoders/decoders.
The IServiceMessageContext locks all properties without really a need, because assigning/reading int and objects is transaction safe.
- fix NormalizeHostName when host ip addresses are not returned on macOS, try the native hostname
- Ensure endpoints are returned even if the client url port do not match (similar to client)
- Improve endpoint tests
Fix warnings regarding localization, CLS compliance, performance. No functional changes.
Many but not all adressed:
CA1859
CA1868
CA1305
CA1310
CA1861
- based on the discussion, use IFormatProvider where appropriate in ToString
- Reduce socket log output and demote to Trace
- Check for invalid chunk header and reduce log output
- Close socket immediately if a bad chunk header is detected (e.g. if socket is opened by another protocol)
- Expose Remote endpoint for application
- The default implementation using `ToString("yyyy-MM-dd'T'HH:mm:ss.FFFFFFFK"` was identified in a CPU hotpath analysis when decoded DataValues are JSON encoded.
- The "o" time conversion implementations is up to 10 times faster (see .NET6.0), but as a caveat it does not omit trailing zeroes.
- Even with handling the special case to omit the trailing zeroes, the `o` version used with `ConvertToUniversalTime` with .NET8 runs still 3 to 7 times faster than the default implementation.
- Simplify the BeginConnect implementation, which is an artifact to support IPv6 on .NET Standard 1.3 and UWP.
- Dual stack is supported by .NET, and the DNS resolution in the socket layer can be delegated to the platform.
- Fixes a race condition under high load, when BeginConnect throws a `Socket already connected` error.
- return serviceResult from client Session.Republish API (Breaking API change)
- refactor #2447. move SessionCreated virtual callback to end of Session.Create.
- fix bug in timeouthint calculation (always 0)
- do not print error if BadSequenceNumberUnknown is returned (server may have cleaned up himself)
- allow for a acktosend threshold when publish response requests are processed out of order
- revert EndAsync call to not throw timeout exception
- use static Randomgenerator instance in X509PfxUtils
- Print suppressible Certificate validation failed errors only if not auto accepted or accepted in callback.
Fix: When the port of AlternateBaseAddress is different from the Endpoint port the discovery returns no valid endpoints.
Co-authored-by: Martin Regen <mregen@microsoft.com>
Issues:
- no ack were added to the list of ack to send after PR #2427.
- Sometimes during network interruptions publish responses are never acked from the republish queue, unless cleaned up by the server.
Fixes:
- removed some debug code from the publish response processing and added acks to send back to the right list.
- Use a dedicated lock for the list of sequence numbers to send
- Detect still available sleeping sequence numbers and mark for deletion after a reasonable age (sequence number diff > 100).
- Breaking change: A `TransportChannel` which causes a `NullReferenceException` in a service call is now returned as `ServiceResultException` with status `BadSecureChannelClosed`. Use `NullableTransportChannel` to avoid the exception and get a null instead.
revert fix for regression introduced by fix for #2343, apply fix in nodemanagers which do not invert the inverse property, so bidirectional references were not removed.
Issue:
- On high load, the keep alive timers keep firing and may get blocked in locks, leading to thread starvation.
- The `AsyncAutoResetEvent` for subscriptions is triggered from the reader loop inside a lock and the following execution on the subscription workload blocks the reader thread.
- A reconnect is never triggered on a 'BadSessionIdInvalid`
Fixes:
- Use periodic timer for platforms supporting it
- Make the `PublishingStopped` and lastkeepalive non blocking by using interlocked variables.
- To decouple the reader loop from the subscription processing, create the TaskCompletionSource with `TaskCreationOptions.RunContinuationsAsynchronously` option.
- Trigger a reconnect if the KeepAlive read request returns with a bad service result.
- Remove unnecessary locks around events, changed calls to be thread safe
- Improve the `Utils.IsEqual` helper for some special cases which compare byte[].
- In the actual implementation the compilers can not optimize for special cases, e.g. comparing a typical certificate as byte[] with 1k size on .NET8 took 167µs, the customized version needs only 69ns and is even faster than a P/Invoke to memc mp (80ns).
- see benchmark result for improvements
- Due to the little difference between a custom byte[] and the T[] compare, the byte[] was moved back to the tests
- .NET Framework falls back on memcmp
- In order to reduce leaking private key files on windows, `EphemeralKeySet` is used on supporting platforms.
- Keys from the Ephemeral Keyset can not be used with `CopyWithPrivateKey`, add support to copy these keys, e.g. for GDS push support.
- MacOS throws PlatformNotSupported when ephemeral keyset is used.
- Kestrel needs a dedicated key without the ephemeral keyset flag.
- Dispose private keys where possible to reduce number of leaked files in tests.
- For bouncy castle legacy support, do not use the `PersistKey` flag by default on netstandard2.0 and net462. However keys may still leak.
Fix cases where a cancellable WaitAsync(ct) on a SemaphoreSlim within a try/finally could end up in a deadlock.
Fix: The WaitAsync must be outside the try/finally block. if the WaitAsync is cancelled, it should not release the semaphore
The goal is to reduce the overall number of builds, stop using end of life platforms in the build pipeline and to support
build of future/deprecated build targets in a custom build step, which is also triggered weekly in the ci builds.
In addition:
- added a doc which describes the release and versioning strategy of this project
- produce the Nuget packges with .NET 8 assemblies.
- produce a Docker build based on a .NET 8 based container.
- add custom test targets to test outdated platforms. Need a manual step to enable for VS or command line.
- bug fixes: the latest move to an alternate crypto library for Bouncy Castle, which is only used for the GDS support and .NET Framework 4.6.2./netstandard2.0, revealed already a few issues which were previously not found.
- only .NET8/.NET6/.NET Framework 4.8 are tested in Pull requests. Remaining builds in weekly scheduled builds.
Since some distributions stop to trust SHA1 signatures by default, replace some internal algorithm checks with SHA256 versions.
The test case to force signature check also reenables ECC tests for .NET 4.8 which had been previously disabled.
- Implement the reconnect and republish with async service calls.
bug fixes:
- subscription Ids should start from a random number
- the keep alive interval calculation is too short, adding +1 for the timeout
- on BadSessionIdInvalid recreate session immediately
- on a restarted server with new certificate, also reload endpoints when BadCertificateInvalid is returned
Currently, due to some internal cloning of sessions, subscriptions and monitored items it is not possible to use subclassed variants which are properly recreated after a session reconnect.
- Add a `ISessionInstantiator` interface to allow to subclass a `Session` and override virtual function.
- Add a `Subscription.CloneSubscription` and `MonitoredItem.CloneMonitoredItem` virtual method to recreate those with event handlers and client handles.
- A sample implementation with `TestableXXX` classes is provided to test the BadInvalidTimestamp fix by injecting modified timestamps in the session class.
- Remove duplicate `IDisposable` with `Session`. `IClientBase` is disposable already.
- adds `knownTypes` parameter to `Session.Load` and `Session.Save` to support save and restore of custom Subscriptions and Monitored items.
bug fixes:
- IOP: an empty DataTypeDictionary throws an exception when loaded by the complex type system, server are allowed to leave it empty to save memory.
- BadInvalidTimestamp should not be returned when the timestamp in the requestheader is out of sync. #2364
Martin Regen