File Transfers from/to AWS S3 Buckets
Today’s businesses operate in a truly cloud-centric world and storing data in the cloud is a task that most modern IT teams need to handle. At BlueFinch, we help hundreds of customers automate and secure their file transfers. In the process, we learn a lot about how organisations trade files with business partners, suppliers, customers, clients, and the like. Recently, we noticed an interesting trend: the use of AWS S3 rather than the traditional SFTP or FTP servers, to exchange files.
by Punit Bafna
Integrating with these popular cloud storage platforms and efficiently automate file transfers using Amazon S3 can be quite challenging.
Advantages of using Amazon S3 for file storage
High scalability
Most file transfer servers store their files in hard disks that are directly attached (or built-in) to the physical server itself. These disks naturally have limited space. Once the maximum limit of these local disks is reached, they must be replaced, or additional disks must be added to the server. This can be time-consuming and can cause a substantial amount of downtime.
By comparison, Amazon’s massive and highly scalable infrastructure enable file transfer servers to have virtually unlimited storage. That means, your users and trading partners can upload gigabytes upon gigabytes or terabytes upon terabytes of files and your server won’t even bat an eyelash. An Amazon S3 bucket, which is where your uploaded files will be stored, has unlimited capacity. So, your users can upload as many files, regardless of size, as they want.
High availability and disaster recovery capabilities
One problem with local storage is that, if the local hard disk crashes, your users will not be able to access their files. Similarly, if the hard disk just works fine but the server itself crashes, the files still won’t be accessible either.
You won’t be subjected to the same level of risk if you store your files on Amazon S3. That’s because, first, Amazon S3 is built on top of a reliable, fault-tolerant, and highly available infrastructure. The chances of an Amazon S3 bucket ‘crashing’ or becoming unavailable is extremely low.
Highly secure
During a file transfer session, i.e., when files are being uploaded to your server, files are protected via data-in-motion encryption technologies like SSL/TLS (in the case of FTPS or HTTPS) or SSH (in the case of SFTP). But once they’re already on your server, they will have to be secured with some form of data-at-rest encryption solution.
For locally stored files, GoAnywhere MFT Server offers PGP-encryption. But how about files uploaded to an Amazon S3 bucket? For this purpose, Amazon offers server-side encryption. Amazon S3 encrypts data using AES-256, a strong and widely accepted block cipher that’s also a federal government standard.
Automating File Transfer to S3 Bucket using GoAnywhere
GoAnywhere MFT from HelpSystems is an enterprise-level Managed File Transfer application that has been ahead of the game for a while. It easily integrates with storage platforms like Amazon’s Simple Storage Service (Amazon S3) buckets, WebDAV servers and Azure Blob Storage.
GoAnywhere allows an Amazon S3 bucket to be configured as a resource and then specified as file repositories from within Domains or as a Web User virtual folder. Using the intuitive workflow builder, you can use the Amazon S3 component to build projects that retrieve or modify object metadata using the S3 task, or upload/download/manage documents by using qualified file paths.
You can also connect to Amazon S3 compatible storage solutions like Ceph, Cleversafe, and Caringo. By specifying the Endpoint URL, GoAnywhere will use its Amazon S3 client to integrate with these compatible services.
Key Features:
- Automate file transfers in real-time.
- Drag-and-drop S3 tasks into your workflows.
- File Transfers can be triggered using the GoAnywhere REST & SOAP API.
The following file transfer commands are supported:
- Upload file(s) to an S3 bucket.
- Download file(s) from an S3 bucket.
- Transfer files between S3 buckets.
- Get Metadata of file(s) & Set Metadata of file(s).
Additional Info:
- Security is ensured by using the HTTPS protocol with support for an optional proxy server.
- GoAnywhere supports Amazon’s server-side SSE-S3 method for encrypting data at rest by using the AES256-bit Advanced encryption standard.
- Supports AWS IAM Role-Based Access (RBCA).
- No Universal Agent needs to be installed on the AWS Cloud – the communication goes via HTTPS.
- AWS canned ACLs are supported, e.g., to grant full access to the bucket owner.
Worth mentioning here is that we also have a variety of Web API connectors (a.k.a., Cloud Connectors) in the GoAnywhere marketplace to connect to AWS services like Lambda, EC2, SNS, SQS and Azure Blob, Data Lake Storage.
For more information take a look at our solutions feel free to reach out to us at sales@bluefinch.com . We’d be more than happy to share our knowledge and experience.
