8

We have an office that has increasing demand for accessing large files from our own Amazon S3 directories. Being able to access them quickly is important for our business, so we believe it is time to start keeping copies of the files onsite. This is not my area of expertise, so I'm hoping for some advice.

A "normal" cache will not be sufficient for us on its own, since we want to speed up even the first request for any given file. The AWS CLI has the ability to keep a local directory in sync with S3, so one idea is to run that on a schedule during low traffic times, then configure a proxy to treat that directory as its cache, if that is possible.

Another idea is to issue get requests to a caching proxy from a script to keep the cache warm, on a similar schedule.

One caveat is that the S3 assets are private, so we sign their URLs before making each request. This means the proxy will need to be able to serve the local copy based on the URL excluding any query parameters. For example, both of these urls should resolve to the same cached/mirrored file:

  • https://example.com/asset1.txt?signature=1
  • https://example.com/asset1.txt?signature=2

The size of the cache will be in the single digit terabytes, and process the traffic for about 300 active users.

So finally, my questions:

  • Do either of these approaches sound sane?
  • Can anyone recommend proxying software that can be configured the the way we need?
  • Are there resources I can consult to determine hardware requirements for this load?
  • Any other thoughts/suggestions?
Improve this question
2
  • 1
    aws.amazon.com/storagegateway
    – ceejayoz
    Feb 13, 2018 at 20:09
  • 1
    That is an interesting offering I had not seen yet, @ceejayoz. Thank you. It looks like, if we were to use that, it would replace the AWS CLI synchronization in the first option I mentioned. I will add that to our list of options.
    – Eric Simonton
    Feb 13, 2018 at 20:31

2 Answers 2

Reset to default
6

If you need just synchronize your local repository with cloud-based object storage, I would take a look at Rclone or CloudBerry. Rclone has a command-line interface to synchronize directories and files between the clouds. It works for most popular cloud storage like Azure, AWS (both S3 and Glacier), etc. https://rclone.org/

Also, if you want to backup all the data to the cloud, there is an opportunity to take Virtual Tape Library backups with an additional offload to the cloud. So if you need to backup your existing infrastructure, you can take ransomware-proof backups with automatic offload to the cloud. It has dedupe and compression, but as far as I know, right now Starwind gives it for free. https://www.starwindsoftware.com/starwind-virtual-tape-library

Both solutions are mature and reliable, you only need to choose the needed option. Hope it was helpful.

Improve this answer
3
  • You can also use the AWS S3 sync command if all you want to do is sync using a command line interface. I do that to make a copy of an S3 bucket on my PC.
    – Tim
    May 21, 2018 at 1:20
  • What about something like automatization in AWS S3 sync? Can I schedule synchronization over command prompt?
    – Stuka
    May 21, 2018 at 8:19
  • On my PC I use Windows Task Scheduler to run a batch file that runs aws s3 sync. On Unix you'd use cron.
    – Tim
    May 21, 2018 at 9:12
1

Depending on your requirements AWS Storage Gateway may provide you with what you need. Storage Gateway is an AWS offering that is deployed into your on-premises environment as a virtual machine.

There are two flavors of Storage Gateway that immediately spring to mind as potentially suitable:

  • File Gateway presents an S3 bucket as an NFS mount and includes transparent local caching.

  • Volume Gateway - Cached Volumes presents as an iSCSI target and also includes local caching of frequently accessed data.

There are some downsides to Storage Gateway:

  • It is NOT designed to support multi-master scenarios, so locking mechanisms are scoped to the Storage Gateway (rather than the underlying S3 bucket). Out of the two, a multi-master scenario would be more suitable with File Gateway as it does support a RefreshCache API call which will update the metadata in your on-premises VM with objects that were added / removed / replaced since the gateway last listed the bucket's contents.

  • Volume Gateway does not provide access to the underlying S3 bucket. So while File Gateway is backed by a customer controlled S3 bucket, Volume Gateway is backed by an AWS controlled S3 bucket. This means that for Volume Gateway you do not see the S3 bucket in your account, and you cannot access the data in it as a normal S3 object. (I can't find documentation backing this up, but I'm 95% sure this is correct)

There are other types of Storage Gateway and you can read about how AWS Storage Gateway works.

If you're not already using Direct Connect, then you may want to consider using it for high bandwidth, low latency access to AWS services. (My guess is you're already using it given the amount of data you mentioned)

Edit 2018-05-21: Storage Gateway Pricing With Storage Gateway you are charged for the underlying storage (size of data + requests) and data transfer. That's it. Any other solution that uses S3 for storage will cost you the same.

Improve this answer
2
  • 1
    AWS Gateway is too expensive for what it does IMHO.
    – NISMO1968
    May 19, 2018 at 14:08
  • If you're going to down vote please give me a quick comment with an explanation so I can do better in future. Also if it's because of price then I really think one of us is missing something and if it's me that's missing something please explain. I'm hear to learn and improve as much as anything else.
    – Alex Hague
    May 20, 2018 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.