FAQ's FileCatalyst

Our support team is at your disposal for any software questions you may have.

Please find the FAQ for the FileCatalyst solution, to help you with basic questions or configurations.

FileCatalyst uses multiple techniques to yield exceptional results; in many cases, resulting transfers have virtual speeds faster than actual line speed. These techniques include: 

  • FileCatalyst UDP-based protocol: FileCatalyst uses a patent-pending UDP-based protocol that is faster in transferring large data sets when there is high latency or packet loss over the network. 
  • On-the-fly Compression: FileCatalyst compresses the data it sends over the network in real time, meaning there is no time-consuming compression or decompression at the beginning or end of each transfer. File sizes may be reduced by 50% or more, creating the illusion of transfers that exceed line speed. 
  • Multiple TCP Streaming: In situations where UDP is not possible, FileCatalyst can achieve some measure of acceleration by running transfers through multiple concurrent TCP streams, along with on-the-fly compression. 
  • Delta Transfers: If a file has changed since its last transfer, FileCatalyst can send only the “deltas” (incremental differences) of the file, rather than resending it in its entirety. 

On-the-fly compression, one method used by FileCatalyst to accelerate file transfers, allows digital files to be reduced in size as they are sent. It uses the same principles as WinZip, Gzip and other compression utilities. 

What differentiates FileCatalyst is that compression occurs as the file is being transferred, saving preparation time. As the files reach the recipient, they are decompressed and automatically stored in their original formats. 

Less setup and teardown is required with on-the-fly compression, which becomes important when transferring a large number of files. Imagine the overhead involved in trying to send 1,000 files, as each file is individually created and closed by the server. Standard compression techniques interrupt the data flow, adding to the total time and making the file transfer appear slower. When sending one large archive, there is only one setup and teardown involved, which greatly speeds up the overall transfer process. 

 

FileCatalyst uses the UDP protocol for data transport and TCP for control commands and retransmission requests. FileCatalyst also provides a secondary firewall-friendly transfer method that enhances the performance of TCP by opening up multiple concurrent streams of data. 

The UDP-based protocol used in FileCatalyst is proprietary. It is a highly efficient, patent-pending, retransmission and congestion control mechanism that adds a reliability layer to UDP. The flow of data can achieve full line speed with an amazingly low 0.25% overhead. 

The UDP-based protocol used in FileCatalyst is proprietary. It is a highly efficient, patent-pending, retransmission and congestion control mechanism that adds a reliability layer to UDP. The flow of data can achieve full line speed with an amazingly low 0.25% overhead.

Both FTP and HTTP use TCP as the transport protocol. The inherent characteristics of TCP make it highly susceptible to network latency and packet loss. Even on a relatively stable network, TCP goodput is always lower than the actual available line speed. For example, on a T3 network (45 Mbps) with packet loss of 0.1% and a delay of 10 ms, FTP transfers can peak at only 30 Mbps. 

In contrast, FileCatalyst yields goodput of 44 Mbps—only slightly less than maximum available line speed. When network conditions deteriorate to 2% packet loss and a delay of 150 ms, FTP transfers can be expected to perform at 450 Kbps, or 1% of the actual available bandwidth. FileCatalyst maintains its 44Mbps goodput. 

Rather than reading and sending the file sequentially, multiple threads read from the file and transfer pieces over their own TCP streams. These pieces are received and reassembled on the fly by an equivalent number of receiver threads. There is no latency during reconstruction as the pieces are written using random access. The number of streams can be tuned to achieve the desired throughput. This method of transferring files is effective when network degradation is at a reasonable level. Due to the large number of concurrent threads that must run in order to sustain high speeds, this method is not as scalable as FileCatalyst UDP-based transfers.

FileCatalyst supports an advanced “delta” transfer algorithm. Once a file is transferred in full, any new revisions will only require these incremental changes to be sent rather than the whole new file. Imagine, for example, a large database file. Sometimes only small portions of the database are changed, such as a single name or location field. FileCatalyst calculates these modifications as “deltas” and transfers only the new data. At the destination, FileCatalyst automatically amends the changes, bringing the file up to speed with the source. This ensures that bandwidth usage is kept to a minimum and results in a very high effective goodput.

Your firewall may be blocking incoming TCP requests on port 21 (or whatever port you set for the control connection). 

Make sure the user under which FileCatalyst server is running has sufficient permissions to write files to the data directory specified. 

Your firewall or router must allow incoming UDP traffic so data can be received from the client applications. Further, if the server is behind a NAT, the packets must be forwarded to the proper IP address. Another possible issue is that the client side is behind a firewall and doesn’t allow outgoing UDP data. 

It is possible that you have set the encoding unit size greater than 1472, which results in fragmented UDP packets. Some routers and firewall’s will automatically drop fragmented packets. If this is the case try lowering the packet size to 1472 or less. Windows operating systems tend to perform better with a value of 1024. 

The answer is likely that your disk is not able to keep up with the transfers. The only option is to upgrade your disk to something faster. To receive at hundreds of Mbps, you will need to have a very efficient storage device to keep up. You could start with a fast SATA drive at 10K or 15K RPM, a fast RAID, or a fast SAN or NAS connected over GigE. 

Hubs are more susceptible to collisions at high speeds which will result in an additional packet loss. As you increase the packet size, the UDP packets FileCatalyst sends will no longer fit in an ether frame. The OS will fragment the Jumbo packets into many smaller fragmented packets that match the MTU of your network (usually 1500 bytes). The larger the packet, the more pieces it is broken into. 

If your concurrent connection speed is 100 Mbps and FileCatalyst server is storing data to network storage (NAS or SAN) on a 100Mbps or lower switch; this will cause FC to compete with itself for bandwidth when receiving and writing. The transfer rate using this network topology would be approximately 50Mbps or lower. 

Performance may be further affected if you are connected to a 100 Mbps or lower hub as this is a potential source of packet loss due to congestion. If replacing the hub with a fast switch is not an option, do not attempt to send at the full capacity of the hub with a packet size greater than 1472 (or whatever setting does not cause fragmentation) or significant performance degradation could occur. 

Windows machines which have mapped network drives may not being visible to the Server due to Microsoft’s implementation of User Access Control (UAC) starting with Windows Vista. 

There are examples online on how to disable this or set the system to allow drives to be visible to services. 

To implement a workaround: 

  1. Unzip the file enablelinkedconnections.zip in the application directory (C:\Program Files\FileCatalyst Server\) 
  2. Run the registry entry enablelinkedconnections.reg to enable the Server to see mapped network drives. 
  3. A reboot will be required for this to take effect.
  4. To uninstall, run the undo.reg the same way as enablelinkedconnections.reg, a restart is also required on uninstall 

On Window or Linux, the logs can be found in the logs directory in the application’s install directory. On OS X, the logs will be found in logs folder of the appropriate application folder in « Library/Application Support/FileCatalyst » for the user running the application, or in the root library if the application is being run as a service. The path to the logs can also be found in the configuration file as log.location. 

FileCatalyst Requirement

FileCatalyst software components must be installed on systems meeting certain recommended requirements.

The following recommended settings are for a typical server deployment up-to 1Gbps and 20 concurrent connections.

  • Virtual Machine Recommended
  • Multi-core x64 CPU with 2Ghz+ (recommended 4+ cores)
  • 16GB of RAM
  • Fast HDD or SSD with Min of 100GB of disk space + user data
  • Windows Server 2016 or higher, Windows 10/11, Linux (Kernel version 4.5 or higher)
  • Java is pre-packaged into the Linux and Windows Installers.
  • Properly configured Firewall/NAT to allow UDP/TCP traffic to the Server.
  • Sub-domain or dedicated public static IP with sub-domain
  • Access to SMTP (Outgoing mail server)

 

For speeds over 1 Gbps and 50 concurrent connections.

  • Physical Machine Recommended
  • 8 cores @ 3.5 GHz (faster cores rather than more cores preferred)
  • 32GB of RAM
  • 500 GB SSD Drive – NVMe drive preferred
  • At least Two 10GbE ports
  • NAS/SAN read/write at 3Gbps or higher tested with iperf3
  • Wide Area Network send/receive at 3Gbps or higher; tested with iperf3 –udp
  • Configure Jumbo packets on the WAN link (required for speeds over 4Gbps)
  • Disable Deep packet introspection on the firewalls
  • Consult with FileCatalyst sales partner for speeds above 3Gbps

 

Recommended VM sizes for public cloud services

Cloud ServiceVM SizeMax. Possible Throughput
AWSM5.XLarge500 Mbps
AWST3.XLarge250 Mbps
AzureStandard D4s v3400 Mbps
AzureStandard D8s v3600 Mbps
  • Multi-core x64 CPU
  • 8GB of RAM recommended on the machine.
  • Windows Server 2016 or higher, Windows 10/11, macOS 10.15 or higher, Linux (Kernel version 4.5 or higher)
  • HDD or SSD with 10GB of install space for software.
  • HDD/SSD or fast network storage with sufficient IO capability for data transfer.
  • Java is pre-packaged into the macOS, Linux and Windows Installers for HotFolder, TransferAgent and Express. For the API, SDK and Command-line tools an Open JDK 8 Amazon Corretto (64-bit) needs to be installed.
  • Network connection to the Server
  • Properly configured Firewall/NAT to allow outgoing UDP/TCP traffic to the server

The following recommended settings are for a typical deployment of up to 25 concurrent connections. Virtual server configurations are welcomed.

  • Multi-core CPU with 2Ghz+ per core with minimum 2 cores
  • 8GB RAM
  • 50 GB of disk space (excluding user data)
  • Operating system:
  • Windows Server 2016 or higher, Windows 10/11, Linux (Kernel version 4.5 or higher)
  • Architecture: 64-bit
  • Tomcat 9.0.X (Tomcat 6/7/8/10 are NOT supported); other containers are not supported.
  • Maria DB 10.6 or MySQL 5.7 with admin rights (MySQL v8 or higher is not supported)
  • Access to SMTP (Outgoing mail server)
  • Open JDK 8 Amazon Corretto (64-bit)
  • Internet connectivity: Sub-domain or dedicated static IP with proper firewall rules
  • Recent web browsers such as Chrome, Firefox, Safari, Edge, Opera.
  • 64-bit architecture is required if TransferAgent must be installed
  • Windows Server 2016 or higher, Windows 10/11, macOS 10.15 or higher, Linux (Kernel version 4.5 or higher) if TransferAgent must be installed
  • Email account to receive emails
  • Broadband Internet connection

The following recommended settings are for a typical deployment of up-to 40 nodes being monitored. Virtual server configurations are welcomed.

  • Multi-core x64 CPU 2Ghz+ (recommended 4+ cores)
  • 16GB RAM or more
  • 300GB of free disk space
  • Operating system:
  • Windows Server 2016 or higher, Windows 10/11, Linux (Kernel version 4.5 or higher)
  • Architecture: 64-bit
  • Access to SMTP (Outgoing mail server)
  • Open JDK 8 Amazon Corretto (64-bit) is embedded into the installers for Windows and Linux.
  • Internet connectivity: Sub-domain or dedicated static IP with proper firewall rules

The following recommended settings are for a typical server deployment up-to 1Gbps and 20 concurrent connections.

  • Multi-core x64 CPU with 2Ghz+ (recommended 4+ cores)
  • 8GB of RAM
  • 5GB of install space for software
  • Windows Server 2016 or higher, Windows 10/11, Linux (Kernel version 4.5 or higher)
  • Open JDK 8 Amazon Corretto (64-bit) is included in the Windows and Linux installers.
  • Properly configured Firewall/NAT to allow UDP/TCP traffic to the service
  • Sub-domain or dedicated public static IP with sub-domain
  • Single-core x64 CPU, 4GB of RAM
  • Windows Server 2016 or higher, Windows 10/11, Linux (Kernel version 4.5 or higher)
  • 250MB of install space for software.
  • Open JDK 8 Amazon Corretto (64-bit) is embedded into the Linux and Windows installers.
  • Network connection to the servers and client applications
  • Properly configured Firewall/NAT to allow outgoing TCP traffic to the clients and server

Still need our help? 🙂

Contact our support. Our technical team we'll be happy to help you. 

📢26 nov - 10:15 10:45

Le Tech Coffee Break revient pour une nouvelle édition ! ☕ Ne ratez pas le Live !