SFTP

Frends.SFTP.DownloadFiles / 4.1.0

Summary

Transfers files off an SFTP server over SSH, deciding per file what happens to the source and how to handle a name collision at the target, and reporting counts and errors per file. Server fingerprint checking is available, which is what makes the host itself verifiable.

Typical use — scheduled collection from a partner; moving or renaming the source after transfer is what prevents a second run refetching it

Remarks

Transfer overview

The file transfer progress has the following steps:

  1. Initialize

    Initializes the transfer and opens the source connection.

  2. ListFiles

    Get a list of files from the source endpoint according to the filename/mask. If there are no files to transfer, the source connections are closed, and the transfer finishes. The result of the task will then depend on what the option NoSourceAction was set to. If it was set to Error, either the #result.Success property will be set to false, or if the `ThrowError

  3. Transfer files
    If there are files to transfer, they are then transferred individually. For every file in the list returned from the source endpoint, the following process is repeated:

    1. GetFile

      Get a file from the source endpoint to the local work directory. If the parameter RenameSourceFileBeforeTransfer is set to true, the file is first renamed with a temporary filename before transfer.

    2. Rename or move the source file.

      This is done before transferring the file to the destination, this means that possible errors in the renaming or moving that would cause the transfer to fail will happen as early as possible - before we actually try to transfer files onward.

    3. Transfer the file.

      If destination file already exists, depending on the parameter DestinationFileExistsAction either an exception is thrown, the destination file is overwritten or the source file is appended to the destination file.
      If the parameter RenameDestinationFileDuringTransfer is true, the file is first transferred with a temporary file name and afterwards renamed to intended filename, otherwise the file is transferred with the intended filename. The intended filename has its possible file masks expanded.

    4. Do the source file operation.

      Perform the operation defined by the SourceOperation.

  4. Finish

    Close the source and destination endpoint connections.
    If the transfer is cancelled (e.g. by calling Terminate on the process instance), the files that are currently being transferred will be processed until finished, but no new files will be transferred. The cancelled transfer end result will be Failed.

Macro reference

Macros can be used to dynamically configure source directory, destination directory or destination file name for a file transfer.

Generally the following rules apply for macros:

  • Macros are case insensitive.
  • You can use any number of macros in all of the cases.
  • Dates and times are formatted with leading zeros.

The following macros can be used with all of dynamically configurable locations for file transfer:

  • %Ticks% = will be replace with the current time as Ticks.
  • %DateTime% = will be replaced with date and time in format: "yyyy-MM-dd-HH-mm-ss".
  • %DateTimeMs% = will be replace with date and time in format: "yyyy-MM-dd-HH-mm-ss-fff".
  • %Date% = will be replaced with date in format: "yyyy-MM-dd".
  • %Time% = will be replaced with time in format: "HH-mm-ss".
  • %Year% = will be replaced with current year.
  • %Month% = will be replaced with current month.
  • %Day% = will be replaced with current day.
  • %Hour% = will be replaced with current hour.
  • %Minute% = will be replaced with current minute.
  • %Second% = will be replaced with current second.
  • %Millisecond% = will be replaced with current millisecond.
  • %WeekDay% = will be replaced with a number of weekday, ranging from 1 (monday) to 7 (sunday).
  • %Guid% = will be replaced with a new unique identifier.
  • %TransferId% = will be replaced with the transfer id.
  • %TransferName% = will be replaced with TransferName parameter specified in Connection point schema.
  • %TransferGroupName% = will be replaced with TransferGroupName parameter specified in routine's task arguments.

Destination directory has one additional macro:

  • %SourceRelativeDirectory% = will be replaced with the relative path from source directory to file's directory during recursive downloads.
    Empty for files in root source directory. Example: for source '/upload/' and file '/upload/sub1/sub2/file.txt',
    macro expands to 'sub1\sub2'. Use in Destination.Directory like 'C:\download%SourceRelativeDirectory%'
    to preserve source folder structure.

Destination file name has two additional macros that can be used for dynamically creating destination file name.

  • %SourceFileName% = will be replaced with source file name without extension.
  • %SourceFileExtension% = will be replaced with source file's extension, with the dot '.' included, i.e. if the source file is named 'foo.txt', the %SourceFileExtension% will be expanded as '.txt'. If the source file name does not have an extension, the macro result will be empty, i.e. for original file name "foo", "bar%SourceFileExtension%" will result in "bar"

Details

Frends version
5.5 and newer
Agent
Crossplatform
Target framework
net6.0

Source transfer options

Name Type Default Example Description
Directory String / /upload/

Directory on the server.

FileName String test.txt

File name or file mask of the files to be fetched.

IncludeSubdirectories Boolean False true

Determines if subdirectories should be included when searching source files. Depending on how many subdirectories there are, this operation can be extremely expensive.

Action SourceAction Error SourceAction.Error

What to do if source file is not found. Error = alarm and fail, Info = alarm info and quit with success status, Ignore = quit with success status.

Possible values:

  • Error: What to do if source file is not found. Error = alarm and fail, Info = alarm info and quit with success status, Ignore = quit with success status.

  • Info: What to do if source file is not found. Error = alarm and fail, Info = alarm info and quit with success status, Ignore = quit with success status.

  • Ignore: What to do if source file is not found. Error = alarm and fail, Info = alarm info and quit with success status, Ignore = quit with success status.

Operation SourceOperation Delete SourceOperation.Delete

What to do with the source file after transfer.

Possible values:

  • Delete: Deletes the source file after transfer.

  • Rename: Renames the source file after transfer.

  • Move: Moves the source file after transfer.

  • Nothing: Leaves the source file unchanged.

FileNameAfterTransfer String transferred.txt

Parameter for Rename operation. You can use file macros and also specify a directory where to move the files to, e.g. /subdir/%Date%file.txt. If you don't define a directory path, the source directory is used. When using rename, this parameter must always contain a file name.

DirectoryToMoveAfterTransfer String /upload/transferred/

Parameter for Move operation. Set the full path to the directory without the file name. You can use some macros in the directory name, e.g. /subdir/%Year%_uploaded/.

FilePaths Object #trigger.data.filePaths

The paths to the files to transfer, mainly meant to be used with the file trigger with the syntax: #trigger.data.filePaths

An unhandled error has occurred. Reload 🗙