Dropzone: Use with File Monitor and File Transfer

For different use cases, it might not be convenient for the Dropzone processing to take place on the same host machine as the source of the files. In these cases, system:inmation offers options for both transferring the files between different host directories and the monitoring of the file directories to subsequently feed the Dropzone files for processing.

File Monitor and File Transfer Options with Dropzone
Figure 1. File Monitor and File Transfer Options with Dropzone

Dropzone and File Monitor

It is strongly recommended to use Dropzone in conjunction with the File Monitor object in instances where files are being transferred from remote directories or when using automation to transfer files. The File Monitor gives extra control, monitoring when a new file has been created in the monitored directory or the modification timestamp of a file has changed. The File Monitor item also can be configured to only recognize certain file types or file names by entering regex expressions.

In this example, it will be demonstrated how to use Dropzone in conjunction with the File Monitor.

  1. Create a Dropzone Datasource under a Connector as described in Dropzone Basics. It is not important which processing option is configured for this example, it should just match the example file that you choose to transfer.

  2. Right click on the same Connector where the Dropzone is created and select Admin  New  Files  File Monitor from the context menu.

  3. In the Create Object wizard, enter an object name and click Create.

    The File Monitor is created along with a child Folder object named "_Monitor". The "_Monitor" folder is used by the File Monitor as the location to create items that record file changes. It is important that this folder is not renamed as this will prevent the File Monitor from functioning correctly.
  4. Select the newly created object in the I/O Model and look at the object properties panel. Open the Common property compound and select Source Folders table property and open the table.

  5. Add the file path of a local file location (multiple file locations can be monitored by the same File monitor object) and click Ok.

    This should be a different folder to the Dropzone’s Disk Folder that was configured during the creation of the Dropzone. To reemphasize, these folders should still be local to one another on the same host machine as the parent Connector instance.
  6. Click the Include Subdirectories checkbox in the Common properties to also monitor any sub directories in the specified Source Folders file locations.

    You must have permissions to access any of the file locations specified here in order to monitor files.
  7. In the Common property compound and select Patterns table property and open the table.

  8. Here different regex expressions can be entered in the table to only monitor files of a certain file type or file name. For example the regex expression '*.(csv)$' will only process files with the csv extension. Leaving the table empty will process all file types and file names. Click Ok to close the table and save the entry.

  9. Open the Scheduling property compound and using the default "Simple" Schedule Mode, set the Recurrence to an appropriate time period (during initial configuration it is best to select a short time period such as 10 seconds to help with troubleshooting).

  10. In the Actions compound, click on the "+" sign, the Action Type should be set to "Dropzone" (if not select this from the drop down menu). Enter a Name for the action as well.

  11. Open the Dropzone Action compound and under Target Dropzone, select the Dropzone that was created during the first step.

    File Monitor Dropzone Actions are only configurable with Dropzone Datasources under the same Connector as the File Monitor. If Dropzone’s under different Connector’s are selected, the connection will not work.
  12. Click Apply in the Object Properties panel to confirm the configuration changes.

  13. To test the File Monitor Dropzone action, transfer the relevant example file to the configured File Monitor Source Folder. The File Monitor will process the file (this can be followed under the Diagnostics property compound) and underneath the File Monitor object in the Monitor folder, a new folder hierarchy is created: Year > Month and then a Table Holder object of the day’s date. Open the Table Data property of the Table Holder object to see information and modification dates of the added file.

File Transfer Options from Remote Locations

Once a File Monitor - Dropzone connection has been created, the Source Folder of the File Monitor can be used a transfer location for files to be fed to the Dropzone Datasource for processing. The options for the transfer of files from remote locations can be done either using Microsoft file transfer tools such as Robocopy or using the system:inmation File Transfer functionality.

It is strongly recommended to use the File Transfer object which allows the transfer of files between file locations on different Connector hosts but requires that a Connector instance also be installed on the host of the file source. The advantages to this are that it bypasses the need for file shares and configuration of permissions on Windows as it uses the system’s secure communication to transfer the files.

If it is not possible to install a Connector instance on the file source then Robocopy can be used to copy files between remote host machines. In these cases, secure file shares and network permissions will need to be configured by system administrators.

Please follow the instructions on How to use File Transfer Objects (File Source and File Sink) to set up the File Transfer. The File Sink can be created below same Connector instance as the Dropzone and the File Monitor. The Disk Folder property for the File Sink can be configured the same as the Source Folder of the File Monitor. In this way a direct pipeline between a remote file source and the Dropzone can be set up.