Frends.SFTP.UploadFiles / 4.4.0
Summary
Transfers local files to an SFTP server over SSH, deciding per file what happens to the source afterwards and how to handle a name already in use.
Typical use — scheduled delivery to a partner, where renaming after transfer tells the receiving side the file is complete
Remarks
Transfer overview
The file transfer progress has the following steps:
Initialize
Initializes the transfer and opens the source connection.
ListFiles
Get a list of files from the source endpoint according to the filename/mask. If there are no files to transfer, the source connections are closed, and the transfer finishes. The result of the task will then depend on what the option
NoSourceActionwas set to. If it was set to Error, either the #result.Success property will be set to false, or if the `ThrowErrorTransfer files
If there are files to transfer, they are then transferred individually. For every file in the list returned from the source endpoint, the following process is repeated:GetFile
Get a file from the source endpoint to the local work directory. If the parameter
RenameSourceFileBeforeTransferis set totrue, the file is first renamed with a temporary filename before transfer.Rename or move the source file.
This is done before transferring the file to the destination, this means that possible errors in the renaming or moving that would cause the transfer to fail will happen as early as possible - before we actually try to transfer files onward.
Transfer the file.
If destination file already exists, depending on the parameter
DestinationFileExistsActioneither an exception is thrown, the destination file is overwritten or the source file is appended to the destination file.
If the parameterRenameDestinationFileDuringTransferistrue, the file is first transferred with a temporary file name and afterwards renamed to intended filename, otherwise the file is transferred with the intended filename. The intended filename has its possible file masks expanded.Do the source file operation.
Perform the operation defined by the
SourceOperation.
Finish
Close the source and destination endpoint connections.
If the transfer is cancelled (e.g. by calling Terminate on the process instance), the files that are currently being transferred will be processed until finished, but no new files will be transferred. The cancelled transfer end result will be Failed.
Macro reference
Macros can be used to dynamically configure source directory, destination directory or destination file name for a file transfer.
Generally the following rules apply for macros:
- Macros are case insensitive.
- You can use any number of macros in all of the cases.
- Dates and times are formatted with leading zeros.
The following macros can be used with all of dynamically configurable locations for file transfer:
- %Ticks% = will be replace with the current time as Ticks.
- %DateTime% = will be replaced with date and time in format: "yyyy-MM-dd-HH-mm-ss".
- %DateTimeMs% = will be replace with date and time in format: "yyyy-MM-dd-HH-mm-ss-fff".
- %Date% = will be replaced with date in format: "yyyy-MM-dd".
- %Time% = will be replaced with time in format: "HH-mm-ss".
- %Year% = will be replaced with current year.
- %Month% = will be replaced with current month.
- %Day% = will be replaced with current day.
- %Hour% = will be replaced with current hour.
- %Minute% = will be replaced with current minute.
- %Second% = will be replaced with current second.
- %Millisecond% = will be replaced with current millisecond.
- %WeekDay% = will be replaced with a number of weekday, ranging from 1 (monday) to 7 (sunday).
- %Guid% = will be replaced with a new unique identifier.
- %TransferId% = will be replaced with the transfer id.
- %TransferName% = will be replaced with TransferName parameter specified in Connection point schema.
- %TransferGroupName% = will be replaced with TransferGroupName parameter specified in routine's task arguments.
Destination directory has one additional macro:
- %SourceRelativeDirectory% = will be replaced with the relative path from source directory to file's directory during recursive uploads when IncludeSubdirectories is enabled.
Empty for files in root source directory. Example: for source 'C:\upload' and file 'C:\upload\sub1\sub2\file.txt',
macro expands to 'sub1/sub2/'. Use in Destination.Directory like '/upload/%SourceRelativeDirectory%'
to preserve source folder structure on the SFTP server.
Destination file name has two additional macros that can be used for dynamically creating destination file name.
- %SourceFileName% = will be replaced with source file name without extension.
- %SourceFileExtension% = will be replaced with source file's extension, with the dot '.' included, i.e. if the source file is named 'foo.txt', the %SourceFileExtension% will be expanded as '.txt'. If the source file name does not have an extension, the macro result will be empty, i.e. for original file name "foo", "bar%SourceFileExtension%" will result in "bar"
- An empty or unknown encoding name now gives a clear error message ("Unknown encoding: '...'") instead of a NullReferenceException.
- Fixed iso-8859-1 and other built-in .NET encodings not being found when entered as a custom encoding
- Fixed thread pool exhaustion in FileOperations
- Fixed NullReferenceException when using FilePaths parameter with null or empty Directory value.
- Added new parameter Source.IncludeSubdirectories which enables recursive file search from subdirectories. Default value is false to maintain backward compatibility.
- Added support for %SourceRelativeDirectory% macro in Destination.Directory. This macro preserves the source folder structure during upload when used with IncludeSubdirectories=true. For files in the root source directory, the macro expands to an empty string. Example: for source 'C:\upload' and file 'C:\upload\sub1\sub2\file.txt', the macro expands to 'sub1/sub2/'.
- Drop support for .NET Framework 4.7.1 and .NET Standard 2.0.
- Updated dependency SSH.NET to the newest version 2025.1.0.
- Drop DSS support
- Operation timeout is now using the Connection.ConnectionTimeout parameter.
- The default Source.FileName is "". "" means that it matches all files (same behavior as for "*")
- Improved asynchronous logic and cancellation token timeout handling
- Improved directory creation logic for SFTP uploads, enhancing reliability when creating nested directories.
- Fixed memory leak with Cancellation token source by disposing the token source at finally block.
- Fixed issue with temp files leaving inside the source directory by adding check for them in clean up method.
- Updated Renci.SshNet library to version 2024.1.0.
- Breaking Changed Result property TransferredDestinationFilePaths to string array.
- Changed SftpClient operations to use async methods with CancellationToken.
- Fixed issue where single failure stops other file transfers when using FilePaths.
- Fixed issue where Task can't be cancelled from UI. Change enables users to terminate the transfer even if file upload is ongoing.
- Updated Renci.SshNet library to the newest version 2023.0.0.
- Added FileOperations class which handles file operations as async with CancellationToken e.g. Append, Copy, Move.
- Added Timeout Option which will stop the Task when the operation takes enough time.
- Breaking New feature: Added parameter AssumeFileExistence which will when enabled, skip checking if destination file esists and overwrites any existing files automatically.
- Fixed issue with empty destination directory by adding forward slash to the parameter if it's empty.
- Added SFTPClient.OperationTimeout which should cause timeout during operations if the Task becomes stuck.
- Breaking Added new parameter for keyboard-interactive authentication where users can add prompts and responses.
- Modified operations log to list current system and sftp server information.
- Modified private key passphrase to be visible when all private key authentication options were enabled.
- Fixed operations log to show case exceptions more precisely.
- Fixed issue with TransferredFileNames was incorrect when FilePaths parameter was used.
- Added new result attribute TransferredDestinationFilePaths list consisting of the file paths of destination files.
- Fixed Permission denied error when using mounted CIFS share by canonizing the destination path.
- Fixed bug with task trying to rename the source file after exception without it being renamed in the first place.
- Fixed bug with FilePaths when using list of objects in parameter.
- Documentational fixes to the FilePaths parameter.
- Re-enabled key exchange algorithms 'curve25519-sha256' and 'curve25519-sha256@libssh.org'.
- Added cancellationToken to method ListSourceFiles so Task's execution can be terminated via Frends.
- Breaking Fixed issue where keepaliveinterval and operationtimeout is set as the connectiontimeout by creating them for the keepaliveinterval its own parameter and removed operationtimeout.
- Updated dependencies System.Text.Encoding.CodePages and Microsoft.Extensions.DependencyInjection to the newest version.
- Modified test to run against net471 instead of net6.
- Breaking Added parameters for the file extension of temporary source and destination files when rename options are enabled.
- Fixed operations log to show correct state when source files are not found with filePaths.
- Fixed operations log to use temp work path when getting source files to temp directory.
- Added tests for the filePaths.
- Added boolean parameter for adding a new line when appending to an existing file.
- Changed the appending to use AppendAllText instead of AppendAllLines.
- Fixed OperationTimeout and KeepAliveInterval attributes to use directly user input.
- Added OperationTimeout and KeepAliveInterval attributes to SftpClient and set them to same value as the ConnectionTimeout parameter.
- Added possibility to give different directory to the task when using SourceOperation.Rename.
- Updated documentation.
- Breaking Added option to enable keyboard-interactive authentication method. This change will break the automatic updates of the task.
- Fixed issue with connecting to server which uses keyboard-interactive authentication method. Fixed by adding UseKeyboardInteractiveAuthentication parameter and handled the method inside tha task.
- Fixed issue with the error message by changing how the error message is build. Error message had 'SFTP://' in both endpoints.
- Fixed error handler by adding connection check to FileTransporter and SingleFileTransfer classes. If the client is not connected the task tries to connect again before handling errors.
- Added some tests and separated Appending tests to their own class.
- Updated depricated library Microsoft.Extensions.DependencyInjection from 5.0.1 to 6.0.0
- [Beaking] Removed UTF-16 and Unicode FileEncoding because their implementation didn't work. These were used as a parameter so autoupdate won't work.
- Fixed how the Encoding on windows-1252 is handled. Added NuGet System.Text.Encoding.CodePages which can handle that encoding.
- Fixed error handling by adding catch to FileTransporter to catch SftpPathNotFoundException and general Exception.
- Added HostKeyAlgorithm parameter which enables users to change the host key algorithm used in the task. Before task defaults to ED25519.
- Added tests to test the file name and content encoding.
- Updated the document to state that Ssh.Net only supports private keys in OpenSSH and ssh.com formats.
- Added documentation on the private key formatingm from putty.ppk to OpenSSH.
- Moved all the SourceOperation tests to their own test class.
- Fixed issue with server fingerprint given by user in SHA256 hex format was not accepted: Added conversion to the fingerprint given by user.
- Fixed issue when using invalid server fingerprint in MD5 string format throws wrong error message: Added more specific error messages.
- Changed how MD5 string is handled. MD5 can now be given without ':' or '-' characters.
- Fixed issue that Sha256 was only accepted in Base64 format: Added support for Sha256 in hex format.
- Added selector for host key algorithm which when enabled will force the task to use specific algorithm.
- Added more tests for using server fingerprints.
- Fixed issue where when using SourceOperation.Move the source file cannot be restored when exception occurs.
- Breaking Changed the implementation to work similar to Cobalt by moving the source file to local Temp folder before transferring to destination.
- Fixed issue where PreserveModified caused exceptions because the method used wrong file path.
- Fixed bug where source files were deleted if RenameSourceFileBeforeTransfer was enabled and SourceOperation.Move had directory that didn't exist.
- Added logger usage in places where it was needed to make the operations log and error info more informative.
- Modified the logger usage that the logger.NotifyInformation is done after the action so it's easier to see where errors has occurred.
- Fixed Source Operations removed source file when RenameSourceFileBeforeTransfer was enabled and transfer failed.
- Fixed Destination file being removed when RenameDestinationFileDuringTransfer was enabled and transfer failed.
- Removed permission changes to destination file.
- Fixed documentation.
- Cleaned up test files, added more test for macros, fixed source file rename and move after transfer
- Initial implementation