How to use File Transfer Objects (File Source and File Sink)

system:inmation provides a convenient way to transfer files between different components in the system. The File Transfer feature in in simplest form uses two different objects classes in conjunction:

The File Source is created below the component from which the file will be transferred (for example the Connector) and the File Sink on the component where the file be transferred to (for example, the Core).

It is possible to have multiple File Source objects configured to send files to one File Sink. In this way, files can be sent from different components to the same central component location, for example, a Core.
Overview of File Transfer Objects
Figure 1. Overview of File Transfer Objects

There are two different options for storing the files on the File Sink side:

  • Save to the configured Disk Folder on the the component host machine

  • Database Storage in the Custom File Data Store in the MongoDB repository.

Configuration for both options will be covered here.

File Transfer - Save to Disk Folder

In this example, the transfer of files from a Connector to a Core component is described where the file is saved to a disk folder on the Core component host. This example is broadly analogous to any other component to component file transfer so is useful for other specific use cases involving different components in different arrangements.

To demonstrate the File Transfer from a Connector component to a Core component, it is first necessary to have a Connector service installed on either a local or remote host and a corresponding Connector object configured in the I/O model below the Core.
  1. First we will create a File Sink below the Core object. Right-click on the Core and select Admin  New  Files  File Sink from the context menu.

  2. In the Create Object wizard, give the object a name and click Next >.

    Create File Sink - Common
    Figure 2. Create File Sink - Common
  3. In the Storage Type properties set Storage Type to "Local Storage" and configure a path to a local Disk Folder.

    Create File Sink - Storage Type
    Figure 3. Create File Sink - Storage Type
    If the path to the disk folder is invalid, the File Sink will display a Bad State (red light) in the I/O Model and an error message will be displayed in the log.
  4. Click Create to create the object in the I/O Model.

  5. Next we will create a File Source below a Connector. Right-click on the Connector and select Admin  New  Files  File Source from the context menu.

  6. In the Create Object wizard, give the object a name and click Next >.

    Create File Sink - Common
    Figure 4. Create File Source - Common
  7. In the File Source Parameters properties, configure the Disk Folder to a folder path on the host that the Connector is installed on. The Save in Processed Folder option will save a copy of any transferred files in the "processed" folder underneath the configured Disk Folder (optional, default is off). The Polling Interval property sets how often the Disk Folder is checked for new files, for the purpose of this example we will set it to the lowest value of 1 min so there is not a long wait until the transfer happens. Click Create to create the object in the I/O Model.

    .Create File Source - File Source Parameters
    Figure 5. Create File Source - File Source Parameters
    If the path to the disk folder is invalid, the File Source will display a Bad State (red light) in the I/O Model and an error message will be displayed in the log.
  8. Select the newly created File Source object and in the Object Properties panel. open the File Sink property picker dialog. The File Sink object that you created earlier should be available for selection. Check the box and then click Apply.

    File Source - Select File Sink
    Figure 6. File Source - Select File Sink
    The File Source object will be in a Bad State (red light), until the File Sink property has been configured.
  9. You are now ready to make the first file transfer! Navigate to the File Source Disk Folder in File Explorer and copy a file into the directory (for this example we use the inmationSetup.exe installer file as it is >500 MB and the different stages of the transfer can be observed). When the File Source object polls the configured Disk Folder, it creates 3 folders in the directory "noproc", "processed" and "work".

    File Source Disk Folder
    Figure 7. File Source Disk Folder
  10. If for some reason the transfer doesn’t work, the file will be transferred to the "noproc" file. The "processed" folder is populated with successfully transferred files IF the Saved in Processed Folder property has been selected. The copied file is transferred to the "work" folder during the transfer process.

    File Source Work Folder
    Figure 8. File Source Work Folder
  11. During transfer, the progress of the transfer can be observed in the Status compound in the File Source Object properties panel.

    File Source - Transfer Status
    Figure 9. File Source - Transfer Status
  12. During transfer, the File Sink Status compound will also report the current file that is being transferred.

    File Sink - Status
    Figure 10. File Sink - Status
  13. When transfer begins, the work folder is created the configured File Sink Disk Folder.

    File Sink Disk Folder
    Figure 11. File Sink Disk Folder
  14. This contains the temporary created for the transfer.

    File Sink Disk Folder - Work Folder
    Figure 12. File Sink Disk Folder - Work Folder
  15. When the transfer is complete, a new folder is created in the File Sink Disk Folder that contains the transferred file. It is named according to the Object ID of the corresponding File Source object.

    File Sink - Processed File
    Figure 13. File Sink - Processed File, Object ID
  16. Open the folder to find the transferred file.

File Sink - Transferred File
Figure 14. File Sink - Transferred File

+ NOTE: If multiple File Sources are associated with the same File Sink then a separate folder is created for each File Source object.

  1. Congratulations! File transfer is complete.

    If files with the same filename are transferred to the same File Sink Disk Folder location, the files are not renamed automatically and will be overwritten.

File Transfer - Database Storage

In this example, the transfer of files from a Connector to a Core component is described where the file is archived in the Custom File Data Store. This example is broadly analogous to any other component to component file transfer so is useful for other specific use cases involving different components in different arrangements.

  1. First we will create a Custom File Data Store to . Right click on a Core or a Data Store Group object and select Admin  New  Data Stores  Custom File Data Store from the context menu to open the Create Object wizard.

  2. Give the object a unique name and enter a description if necessary.

    Custom File Data Store Wizard
    Figure 15. Custom File Data Store Wizard - Common Properties
  3. Click on the Upload Restrictions options from the wizard sidebar, here you can select certain file extensions to be whitelisted or blacklisted for upload into the Custom File Data Store. A maximum file size can also be configured here.

    Custom File Data Store Wizard - Upload Restrictions
    Figure 16. Custom File Data Store Wizard - Upload Restrictions
  4. Click on the File Data Store options from the wizard sidebar. The default database name can be changed here, along with data retention period and purge settings (what happens to the object data if the object is deleted). Purge Files - this global option turns on the data store purging that runs according to the Retention Time setting. The Partial Files Purge Time controls how long any partial files (ones that are only partially transferred) are kept for before being purged.

    Partially transferred files do not recommence transfer in the event of disconnection or server downtime. In these cases the file transfer begins again from the start. The uncompleted partial files are then purged according to the Partial Files Purge Time setting.
    Custom File Data Store Wizard - File Data Store
    Figure 17. Custom File Data Store Wizard - File Data Store
  5. If necessary, the MongoDB connection settings can also be changed (for example, if you wish to connect to a remote MongoDB instance or configure replication sets).

    It might be necessary to configure the Authentication Mode and User Name/Password if the MongoDB instance you are using also has this configured. The default connection is to the localhost/port but the other connection settings are not provided automatically.
  6. Click Create to create the object in the I/O model tree.

  7. Next we will create a File Sink below the Core object. Right-click on the Core and select Admin  New  Files  File Sink from the context menu.

  8. In the Create Object wizard, give the object a name and click Next >.

    File Sink Database Storage - Common
    Figure 18. File Sink Database Storage - Common
  9. In the Storage Type properties set Storage Type to "Database Storage". Click on Custom File Data Store and use the instance picker to select the Custom File Data Store you created in the earlier steps.

    File Sink Database Storage - Storage Type
    Figure 19. File Sink Database Storage - Storage Type
  10. Click Create to create the object in the I/O Model.

  11. Next we will create a File Source below a Connector. Right-click on the Connector and select Admin  New  Files  File Source from the context menu.

  12. In the Create Object wizard, give the object a name. Open the File Sink property picker dialog. The File Sink object that you created earlier should be available for selection. Check the box then click Next >. and click Next >.

    File Source Database Storage - Common
    Figure 20. File Source Database Storage - Common
  13. In the File Source Parameters properties, configure the Disk Folder to a folder path on the host that the Connector is installed on. The Save in Processed Folder option will save a copy of any transferred files in the "processed" folder underneath the configured Disk Folder (optional, default is off). The Polling Interval property sets how often the Disk Folder is checked for new files, for the purpose of this example we will set it to the lowest value of 1 min so there is not a long wait until the transfer happens. Click Create to create the object in the I/O Model.

File Source Database Storage - File Source Parameters
Figure 21. File Source Database Storage - File Source Parameters
  1. You are now ready for the transfer! Navigate to the File Source Disk Folder in File Explorer and copy a file into the directory (for this example we use the inmationSetup.exe installer file as it is >500 MB and the different stages of the transfer can be observed). When the File Source object polls the configured Disk Folder, it creates 3 folders in the directory "noproc", "processed" and "work".

    File Source Disk Folder
    Figure 22. File Source Disk Folder
  2. If for some reason the transfer doesn’t work, the file will be transferred to the "noproc" file. The "processed" folder is populated with successfully transferred files IF the Saved in Processed Folder property has been selected. The copied file is transferred to the "work" folder during the transfer process.

    File Source Work Folder
    Figure 23. File Source Work Folder
  3. During transfer, the progress of the transfer can be observed in the Status compound in the File Source Object properties panel.

    file transfer database storage source status
    Figure 24. File Source - Transfer Status

File Callback Lua

The File Source class has a Lua Callback property File Callback that can be used when transferring files for storage in the Custom File Data Store.

File Source - File Callback
Figure 25. File Source - File Callback

The callback returns a boolean value, if "true" is returned, an extra comment parameter can be added which will be added as custom metadata stored with the file in MongoDB. Extra logic can also be added, for example in the following callback script to restrict files with certain file extensions from being transferred.

local filepath = ...

local ext = filepath:match("^.+%.(.+)$")
	if ext == "exe" then
		return false, {}
	end

return true, { comment = "file for transfer", author = "Current User" }

This callback will return false for any files with the ".exe" extension, the transfer will not proceed and an error will be written in the log.

File Callback - Error Message
Figure 26. File Callback - Error Message

If the callback returns true (with an allowed file extension) then the file is transferred and stored in MongoDB with accompanying custom metadata.

mongodb filetransfer custom metadata
Figure 27. MongoDB Entry - Custom Metadata