SFTP

Frends.SFTP.DownloadFiles / 4.2.0

Summary

Transfers files off an SFTP server over SSH, deciding per file what happens to the source and how to handle a name collision at the target, and reporting counts and errors per file. Server fingerprint checking is available, which is what makes the host itself verifiable.

Typical use — scheduled collection from a partner; moving or renaming the source after transfer is what prevents a second run refetching it

Remarks

Transfer overview

The file transfer progress has the following steps:

  1. Initialize

    Initializes the transfer and opens the source connection.

  2. ListFiles

    Get a list of files from the source endpoint according to the filename/mask. If there are no files to transfer, the source connections are closed, and the transfer finishes. The result of the task will then depend on what the option NoSourceAction was set to. If it was set to Error, either the #result.Success property will be set to false, or if the `ThrowError

  3. Transfer files
    If there are files to transfer, they are then transferred individually. For every file in the list returned from the source endpoint, the following process is repeated:

    1. GetFile

      Get a file from the source endpoint to the local work directory. If the parameter RenameSourceFileBeforeTransfer is set to true, the file is first renamed with a temporary filename before transfer.

    2. Rename or move the source file.

      This is done before transferring the file to the destination, this means that possible errors in the renaming or moving that would cause the transfer to fail will happen as early as possible - before we actually try to transfer files onward.

    3. Transfer the file.

      If destination file already exists, depending on the parameter DestinationFileExistsAction either an exception is thrown, the destination file is overwritten or the source file is appended to the destination file.
      If the parameter RenameDestinationFileDuringTransfer is true, the file is first transferred with a temporary file name and afterwards renamed to intended filename, otherwise the file is transferred with the intended filename. The intended filename has its possible file masks expanded.

    4. Do the source file operation.

      Perform the operation defined by the SourceOperation.

  4. Finish

    Close the source and destination endpoint connections.
    If the transfer is cancelled (e.g. by calling Terminate on the process instance), the files that are currently being transferred will be processed until finished, but no new files will be transferred. The cancelled transfer end result will be Failed.

Macro reference

Macros can be used to dynamically configure source directory, destination directory or destination file name for a file transfer.

Generally the following rules apply for macros:

  • Macros are case insensitive.
  • You can use any number of macros in all of the cases.
  • Dates and times are formatted with leading zeros.

The following macros can be used with all of dynamically configurable locations for file transfer:

  • %Ticks% = will be replace with the current time as Ticks.
  • %DateTime% = will be replaced with date and time in format: "yyyy-MM-dd-HH-mm-ss".
  • %DateTimeMs% = will be replace with date and time in format: "yyyy-MM-dd-HH-mm-ss-fff".
  • %Date% = will be replaced with date in format: "yyyy-MM-dd".
  • %Time% = will be replaced with time in format: "HH-mm-ss".
  • %Year% = will be replaced with current year.
  • %Month% = will be replaced with current month.
  • %Day% = will be replaced with current day.
  • %Hour% = will be replaced with current hour.
  • %Minute% = will be replaced with current minute.
  • %Second% = will be replaced with current second.
  • %Millisecond% = will be replaced with current millisecond.
  • %WeekDay% = will be replaced with a number of weekday, ranging from 1 (monday) to 7 (sunday).
  • %Guid% = will be replaced with a new unique identifier.
  • %TransferId% = will be replaced with the transfer id.
  • %TransferName% = will be replaced with TransferName parameter specified in Connection point schema.
  • %TransferGroupName% = will be replaced with TransferGroupName parameter specified in routine's task arguments.

Destination directory has one additional macro:

  • %SourceRelativeDirectory% = will be replaced with the relative path from source directory to file's directory during recursive downloads.
    Empty for files in root source directory. Example: for source '/upload/' and file '/upload/sub1/sub2/file.txt',
    macro expands to 'sub1\sub2'. Use in Destination.Directory like 'C:\download%SourceRelativeDirectory%'
    to preserve source folder structure.

Destination file name has two additional macros that can be used for dynamically creating destination file name.

  • %SourceFileName% = will be replaced with source file name without extension.
  • %SourceFileExtension% = will be replaced with source file's extension, with the dot '.' included, i.e. if the source file is named 'foo.txt', the %SourceFileExtension% will be expanded as '.txt'. If the source file name does not have an extension, the macro result will be empty, i.e. for original file name "foo", "bar%SourceFileExtension%" will result in "bar"

Details

Frends version
5.5 and newer
Agent
Crossplatform
Target framework
net6.0
4.2.02026-10-21
Fixed
  • An empty or unknown encoding name now gives a clear error message ("Unknown encoding: '...'") instead of a NullReferenceException.
  • Fixed iso-8859-1 and other built-in .NET encodings not being found when entered as a custom encoding.
4.1.02026-05-22
Added
  • Added support for %SourceRelativeDirectory% macro in Destination.Directory. This macro preserves the source folder structure during download when used with Source.IncludeSubdirectories=true. For files in the root source directory, the macro expands to an empty string. Example: for source '/upload/' and file '/upload/sub1/sub2/file.txt', the macro expands to 'sub1\sub2'.
4.0.02026-04-26
Changed
  • Drop support for .NET Framework 4.7.1 and .NET Standard 2.0.
3.1.02026-02-17
Fixed
  • Fixed the issue with not deleting all temporary files.
3.0.02026-01-27
Changed
Breaking
  • Updated dependency SSH.NET to the newest version 2025.1.0.
  • Drop DSS support
2.18.02026-01-23
Fixed
  • Improve error handling to secure against data loss within temporary files.
2.17.02026-01-22
Fixed
  • Use safe GUID temp filename to handle source files with Windows-invalid characters
2.16.02026-01-13
Fixed
  • Operation timeout is now using the Connection.ConnectionTimeout parameter.
2.15.02025-10-17
Fixed
  • Fixed restoring destination file in any case.
2.14.02024-09-30
Fixed
  • Fixed small memory leak with CancellationTokenSource by adding dispose to the end of the Task.
2.13.02024-09-05
Fixed
  • Fixed issue with certain SFTP servers which did not use IsRegularFile property on files by modifying the logic to check that the file is not anything else than a regular file.
2.12.02024-08-16
Updated
  • Updated Renci.SshNet library to version 2024.1.0.
2.11.02024-04-30
Fixed
  • Fixed issue were Task threw exception when using macros in Source Directory.
2.10.02024-03-06
Added
  • Breaking Added new parameter Source.IncludeSubdirectories which enables source file search from subdirectories.
2.9.12024-02-22
Changed
  • Added StyleCop.Analyzers to analyze the Task code.
  • Made code changes suggested by the StyleCop.Analyzers.
  • Fixed Readme.
2.9.02023-12-15
Added
  • Breaking Added new parameter Options.Timeout which will cause the Task to terminate at assigned period of time if enabled.
Fixed
  • Fixed FileOperations.MoveAsync to remove the moved file.
2.8.02023-12-05
Changed
  • Breaking Changed TransferredDestinationFilePaths Result property type from List to string[].
  • Changed SftpClient operations to use async methods with CancellationToken.
Added
  • Added FileOperations class which handles file operations as async with CancellationToken e.g. Append, Copy, Move.
Fixed
  • Fixed issue where the Task gets stuck and can't be terminated.
2.7.12023-09-08
Added
  • Added SFTPClient.OperationTimeout which should cause timeout during operations if the Task becomes stuck.
2.7.02023-06-07
Added
  • Breaking Added new parameter for keyboard-interactive authentication where users can add prompts and responses.
  • Modified operations log to list current system and sftp server information.
Fixed
  • Modified private key passphrase to be visible when all private key authentication options were enabled.
  • Fixed operations log to show case exceptions more precisely.
2.6.12023-05-17
Fixed
  • Fixed issue with TransferredFileNames was incorrect when FilePaths parameter was used.
2.6.02023-05-16
Added
  • Added new result attribute TransferredDestinationFilePaths list consisting of the destination file paths.
2.5.42023-02-14
Added
  • Re-enabled key exchange algorithms 'curve25519-sha256' and 'curve25519-sha256@libssh.org'.
2.5.32023-02-08
Fixed
  • Fixed issue with file not found when downloading from server when server has automatic delete action enabled.
2.5.22023-01-31
Added
  • Added cancellation token to the method listing the source files so Task won't freeze in that execution.
2.5.12022-12-30
Fixed
  • Fixed issue with empty source dir throws an exception by setting empty source directory as '/'.
  • Fixed issue with when no source files userResultMessage still said no. files transferred and not no files transferred.
2.5.02022-12-16
Fixed
  • Breaking Fixed issue where keepaliveinterval and operationtimeout is set as the connectiontimeout by creating them for the keepaliveinterval its own parameter and removed operationtimeout.
2.4.22022-12-01
Updated
  • Updated dependencies System.Text.Encoding.CodePages and Microsoft.Extensions.DependencyInjection to the newest version.
  • Modified test to run against net471 instead of net6.
2.4.12022-11-24
Fixed
  • Fixed issue with files not found when the name consist of special characters by adding a check for the files exact name.
2.4.02022-11-03
Added
  • Breaking Added parameters for the file extension of temporary source and destination files when rename options are enabled.
  • Fixed task to enable transfers with file paths.
  • Fixed operations log to show correct state when source files are not found with filePaths.
  • Fixed operations log to use temp work path when getting source files to temp directory.
  • Added tests for the filePaths.
  • Modified tests to not use physical files but to create them during test run.
2.3.02022-10-12
Added
  • Added boolean parameter for adding a new line when appending to an existing file.
  • Changed the appending to use AppendAllText instead of AppendAllLines.
2.2.32022-10-12
Fixed
  • Fixed OperationTimeout and KeepAliveInterval attributes to use directly user input.
2.2.22022-10-03
Added
  • Added OperationTimeout and KeepAliveInterval attributes to SftpClient and set them to same value as the ConnectionTimeout parameter.
2.2.12022-09-30
Fixed
  • Added possibility to give different directory to the task when using SourceOperation.Rename.
  • Updated documentation.
2.2.02022-09-26
Fixed
  • Breaking Added option to enable keyboard-interactive authentication method. This change will break the automatic updates of the task.
  • Fixed issue with connecting to server which uses keyboard-interactive authentication method. Fixed by adding UseKeyboardInteractiveAuthentication parameter and handled the method inside tha task.
2.1.22022-09-23
Fixed
  • Fixed issue with the error message by changing how the error message is build. Error message had 'SFTP://' in both endpoints.
2.1.12022-09-16
Fixed
  • Fixed error handler by adding connection check to FileTransporter and SingleFileTransfer classes. If the client is not connected the task tries to connect again before handling errors.
2.1.02022-09-09
Fixed
  • [Beaking] Removed UTF-16 and Unicode FileEncoding because their implementation didn't work. These were used as a parameter so autoupdate won't work.
  • Fixed how the Encoding on windows-1252 is handled. Added NuGet System.Text.Encoding.CodePages which can handle that encoding.
  • Fixed error handling by adding catch to FileTransporter to catch SftpPathNotFoundException and general Exception.
  • Added tests to test the file name and content encoding.
  • Updated the document to state that Ssh.Net only supports private keys in OpenSSH and ssh.com formats.
  • Added documentation on the private key formatingm from putty.ppk to OpenSSH.
  • Fixed HostKeyAlgorithm by removing the forcing of the ssh-rsa.
  • Added HostKeyAlgorithm parameter which enables users to change the host key algorithm used in the task. Before task defaults to ED25519.
  • Added enum HostKeyAlgorithms with the supported algorithms.
  • Modified tests to create testfiles instead of using files from project directory.
2.0.32022-08-19
Fixed
  • Fixed issue with server fingerprint given by user in SHA256 hex format was not accepted: Added conversion to the fingerprint given by user.
  • Fixed issue when using invalid server fingerprint in MD5 string format throws wrong error message: Added more specific error messages.
  • Changed how MD5 string is handled. MD5 can now be given without ':' or '-' characters.
  • Fixed issue that Sha256 was only accepted in Base64 format: Added support for Sha256 in hex format.
  • Changed the used HostKeyAlgorithm by forcing to use ssh-rsa default was ed25519.
  • Added more tests for using server fingerprints.
2.0.22022-08-10
Fixed
  • Fixed issue where when using Rename options and appending / overwrite the task would throw an exception because the work directory given by user was used with renaming destination file.
  • Added error message if a file with the same name as source file exists in DirectoryToMoveAfterTransfer.
  • Fixed issue where when using SourceOperation.Move the source file cannot be restored when exception occurs.
  • Fixed issue that if SourceOperation is ignore and no source files were found no operations log is created.
  • Removed FileEncoding UTF-16 which was not implemented and threw exception if selected.
2.0.12022-07-18
Fixed
  • Fixed issue with task throwing exception when no source files found and source action info or ignore selected.
2.0.02022-07-15
Fixed
  • Breaking Changed the implementation to work similar to Cobalt by moving the source file to local Temp folder before transfering to destination.
  • Fixed issue where PreserveModified caused exceptions because the method used wrong file path.
  • Fixed bug where source files were deleted if RenameSourceFileBeforeTransfer was enabled and SourceOperation.Move had directory that didn't exist.'
  • Added logger usage in places where it was needed to make the operations log and error info more informative.
  • Modified the logger usage that the logger.NotifyInformation is done after the action so it's easier to see where errors has occurred.
1.0.42022-06-30
Fixed
  • Fixed issue where '.' and '..' directories were also fetched when using '*' character as source file mask.
  • Added check for GetSourceFiles so that only files are fetched and not directories.
  • Updated Microsoft.Extension.DependencyInjection library.
1.0.32022-06-29
Fixed
  • Fixed issue with forward slash being added to the source directory.
1.0.22022-06-15
Fixed
  • Fixed issue with download failes when RenameDestinationFileDuringTransfer was enabled and destination file existed.
  • Added tests to test the issue.
1.0.12022-06-13
Fixed
  • Fixed Source Operations removed source file when RenameSourceFileBeforeTransfer was enabled and transfer failed.
  • Fixed Destination file being removed when RenameDestinationFileDuringTransfer was enabled and transfer failed.
  • Removed permission changes to destination file.
  • Fixed documentation.
1.0.02022-06-03
Added
  • Initial implementation
An unhandled error has occurred. Reload 🗙