* Bump the dependencies group
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Robert Hague <rh@johnstreetcapital.com>
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
Currently an array is allocated to read each packet from the socket, followed by
decryption which allocates another array for the plaintext payload. We can save one
of these two allocations by adding a persistent buffer for socket receives, and
allowing the cipher implementations to decrypt into the given payload array.
We can save the other allocation similarly, but in a separate change.
Several commonly used Write methods on SshDataStream end up calling Write(ReadOnlySpan)
on the base MemoryStream. But since SshDataStream is a derived type, that method just
rents a buffer and hands it to Write(byte[], int, int), which defeats any stackalloc'ing
or renting that SshDataStream does itself. Instead, with a bit extra accounting we can
just write directly into the underlying buffer.
An SFTP download performs several reads from the server in parallel, allocating an array
to store each result until it's ready to be consumed. Since these buffers are short-lived
and normally of the same large-ish size (32KB), it seems like a good candidate for pooling.
ShellStream does not currently override the Read/Write async variants. They fall back to
the base class implementations which run the sync variants on a thread pool thread, only
allowing one call of either at a time in order to protect implementations that would
break if Read/Write were called simultaneously. In ShellStream, reads and writes are
independent so mutually excluding their use is unnecessary and can lead to effective
deadlocks.
We therefore override WriteAsync to get around this restriction. We do not override
ReadAsync because the sync implementation does not lend itself well to async given the
use of Monitor.Wait/Pulse. Note that while reading and writing simultaneously is allowed,
it is not intended that ShellStream is used with multiple simultaneous reads or multiple
simultaneous writes, so it is fine to keep the base one-at-a-time implementation on
ReadAsync.
Another note is that the new WriteAsync will be simple (synchronous) buffer copying in
most cases, with a call to FlushAsync in others. We also do not override FlushAsync, so
that will go onto a thread pool thread and potentially acquire some locks. But given that
the current base implementation of WriteAsync does that unconditionally, it makes the new
WriteAsync slightly better and certainly no worse than the current version.
This adds an SftpException which sits between the existing SftpPathNotFoundException/
SftpPermissionDeniedException and SshException, and which contains the response code
from the SSH_FXP_STATUS packet, along with a default message if one was not provided.
SftpPathNotFoundException also gains a Path property which is populated in cases where
it makes sense.
* Build the read-ahead mechanism into SftpFileStream
This change unifies the SFTP download implementations that exist via DownloadFile and
via SftpFileStream, by rewriting SftpFileStream to perform the same "read-aheads" as
DownloadFile. This brings the performance of downloads via SftpFileStream in line with
DownloadFile, such that the latter is now effectively SftpFileStream.CopyTo. It also
brings the recently added DownloadFileAsync up to speed since that was implemented via
SftpFileStream.CopyToAsync.
The methodology is a mix of the previous one and that within OpenSSH: the first call to
SftpFileStream.Read sends one read request to the server. The second sends two and when
not interrupted by Write or similar, the number of in-flight read requests continues to
scale up in this fashion.
I have measured CopyTo to be 3-20x faster than before, depending on file size and server
round-trip time.
* Check CanSeek in ReadAllBytes
* Squeeze out some performance
* Avoid rounding issues when checking Timeout values (#1700)
AsTimeout is called from the SshCommand constructor with
Timeout.InfiniteTimeSpan. In this scenario the range check should never
fail, but unfortunately it does in certain scenarios, due to a runtime
or compiler bug (as soon as optimizations are turned off the issue
miraculously disappears).
Closes#1700
* fix tests
---------
Co-authored-by: Robert Hague <rh@johnstreetcapital.com>
* Fix SftpFileAttributes file type detection
To get the file type, S_IFMT should be used as the mask. Instead it was using each file
type as the mask. It meant that e.g. a symbolic link would also show as a regular file
and a character device.
Also allow setting and retrieving the setuid/setgid/sticky bits
* fix build
The message loop currently sits in a call to Poll until the socket has data to read or
it is closed. This is unnecessary - it can equally just sit in the call to Receive.
The call to Poll in Session.IsConnected is also unnecessary - we can instead just call
Socket.Connected. This only returns the connection state as of the last operation, but
we are always performing operations in the message loop (or else we are not connected),
so it should work equally well while being cheaper.
Lastly, when shutting down the socket, shut down both sides rather than just the sending
side (SocketShutdown.Both rather than SocketShutdown.Send) - at this point we do not care
about reading anything else. This makes it (more) certain that we will break out of the
Receive call in the message loop, as has been noted in #355 for whatever remaining issues
still exist there.
* Use BCL Curve25519 when possible
* Update KeyExchangeMLKem768X25519Sha256 and KeyExchangeSNtruP761X25519Sha512
* Split Start and Finish methods for inheritance
* Some refactor
* Update src/Renci.SshNet/Security/KeyExchangeEC.BclImpl.cs
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
* revert
* Create dedicated KeyExchangeECCurve25519 BclImpl
* cleanup
* minor code refactor
* integration test
* Revert "integration test"
This reverts commit 326962664c.
---------
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
When the key exchange produces less key material than is needed for the cipher or hmac
algorithms, there is an iterative procedure to produce more.
For example, a SHA-1 key exchange algorithm produces 20 bytes of key material.
A SHA-256 hmac uses a 32 byte key, so one iteration of the procedure produces another
20 bytes of key material for a total of 40 which is sufficient for the hmac key.
The library works correctly in such cases of one iteration, but the logic is wrong if
more than one iteration is needed. In #1660, the connection uses a SHA-1 kex algorithm
with a SHA-512 hmac (64 byte key), requiring 3 iterations of the extension procedure and
resulting in an error upon connection.
This change fixes the logic to use the output of the previous iteration per the spec.
closes#1660
SftpClient.CreateText does not truncate the file if it exists, contrary to
System.IO.File.CreateText which does. It is documented, but seems pretty
unintuitive and more like a mistake than a concious decision.
I considered leaving this 14 year old behaviour as it is, but turns out other
people have hit it as well (it also affects WriteAll{Bytes/Text/Lines}).
The recently added UploadFileAsync effectively calls stream.CopyToAsync(SftpFileStream).
This is slower than the sync UploadFile (by about 4x in a local test) because the sync
version sends multiple write requests concurrently, without waiting for each response
in turn like the stream-based version does.
This change adapts the sync code for async and uses it to bring the performance of
UploadFileAsync in line with that of UploadFile.
* Refactor logging to allow a loggerfactory per session specified in the ConnectionInfo.
This commit introduces an `ILoggerFactory` to various classes, replacing the static logger factory with an instance-based approach for more flexible and session-specific logging. These changes improve the logging framework's flexibility and maintainability and allow unit testing of logging.
* Improvements bases on feedback. Fixed tests. Added documentation.
* Update src/Renci.SshNet/ConnectionInfo.cs
---------
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
* Bump the dependencies group with 5 updates
Signed-off-by: dependabot[bot] <support@github.com>
* seal types not publicly instantiable (ma0053)
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
* Fixes spelling errors in comments, private members, and public members in test projects
* Adds en-GB as spell checker option; reverts notable cases of American English
* convert file UTF-16 LE BOM -> UTF-8
---------
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
The library currently allocates 4 bytes (and some) for every 1 byte of file
downloaded(*). It could be 0. This takes it to 3.
(*)
1. Array allocated for read of encrypted packet from socket
2. Array for decrypted packet
3. Array for channel data (removed in this change)
4. Array for sftp data packet
The sftp packet stream runs within but independently of the channel data stream,
meaning a channel data packet can contain multiple sftp packets, or an sftp packet
can be split across multiple channel data packets.
Normally the packets are sized such there is a 1-to-1 relationship for efficiency.
When this doesn't happen the library falls back to buffering via a List<byte>,
which is not so efficient. This change uses an array-based buffer instead.
In a sample download which hit this fallback I see about a 20% reduction in memory
allocated.
The library currently does not check for matching host key algorithms until needed
at the end of the key exchange, in contrast to other algorithm types which are checked
beforehand. This leads to confusing or uninformative errors, normally from the server
(correctly) closing the connection.
This change moves that check alongside the rest of them, and also improves the error
messages that arise from no matching algorithms.
* Added GetAttributesAsync to SftpClient
* Adding integration tests + unit test
* Address warnings in test classes.
---------
Co-authored-by: William Decker <william.decker@syndigo.com>
Where beneficial, add additional overrides from the base Stream class. Namely the Span
variants and for PipeStream, the WriteAsync variants (see comments).
The change also adds an internal type borrowed from the runtime repo for easier buffer
management, which could also be used elsewhere.
Wireshark can already helpfully dissect the initial SSH handshake. When given the
session keys, it can also dissect the encrypted traffic for inspection/debugging.
This adds a helper in Debug mode to write out that information in the format
Wireshark requires.
Usage is to set `SshNetLoggingConfiguration.WiresharkKeyLogFilePath` before connecting, and supply the same value to Wireshark in Edit -> Preferences -> Protocols
-> SSH -> "Key log filename".
The description of the format is at https://wiki.wireshark.org/SSH#key-log-format
* Bump the dependencies group with 5 updates
* use MEL 8.0.3
* use MSTest meta package
* revert Meziantou due to NRE
* add a more useful global.json and pin third party action
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
* Add an OrderedDictionary implementation for algorithm priorities
During the key exchange, the algorithms to be used are chosen based on the order that
the client sends: first algorithm is most desirable. Currently, the algorithm
collections in ConnectionInfo are defined as IDictionary<,> and backed by
Dictionary<,>, which does not have any guarantees on the order of enumeration
(in practice, when only adding and not removing items it does enumerate in the order
that items were added as an implementation detail, but it's not great to rely on it).
This change adds IOrderedDictionary<,> and uses it in ConnectionInfo. On .NET 9,
this is backed by System.Collections.Generic.OrderedDictionary<,> and on lower
targets, it uses a relatively simple implementation backed by a List and a
Dictionary.
* use ThrowIfNegative
* Drop net6.0 target
* Update src/Renci.SshNet/Common/TaskToAsyncResult.cs
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
* remove redundant #if
for some reason this made the compiler suddenly
realize that the plain text variables are unused.
* use TargetFrameworkIdentifier
this doesn't work in Directory.Build.props, moved it to Directory.Build.targets.
* fix null reference warnings in Benchmarks
seems like the warnings were (somehow) disabled here
before and were fixed by the previous TargetFrameworkIdentifier
change.
* fix unused plainTextOffset in AesGcmCipher.BclImpl
* CI retry
* more cosmetics
* more
* update README
* Revert "use TargetFrameworkIdentifier"
This reverts commit 076ede161d.
---------
Co-authored-by: Rob Hague <rob.hague00@gmail.com>
Co-authored-by: Robert Hague <rh@johnstreetcapital.com>
dependabot[bot]