We have an office that has increasing demand for accessing large files from our own Amazon S3 directories. Being able to access them quickly is important for our business, so we believe it is time to start keeping copies of the files onsite. This is not my area of expertise, so I'm hoping for some advice.
A "normal" cache will not be sufficient for us on its own, since we want to speed up even the first request for any given file. The AWS CLI has the ability to keep a local directory in sync with S3, so one idea is to run that on a schedule during low traffic times, then configure a proxy to treat that directory as its cache, if that is possible.
Another idea is to issue get requests to a caching proxy from a script to keep the cache warm, on a similar schedule.
One caveat is that the S3 assets are private, so we sign their URLs before making each request. This means the proxy will need to be able to serve the local copy based on the URL excluding any query parameters. For example, both of these urls should resolve to the same cached/mirrored file:
The size of the cache will be in the single digit terabytes, and process the traffic for about 300 active users.
So finally, my questions:
- Do either of these approaches sound sane?
- Can anyone recommend proxying software that can be configured the the way we need?
- Are there resources I can consult to determine hardware requirements for this load?
- Any other thoughts/suggestions?