We have a web (Rails) application where a small number of somewhat large (1 - 10MB) business logic files can be uploaded by administrators. The content files will not change frequently, maybe once a week.

When users interact with the application, another backend (Java) EC2 instance will frequently (multiple times per minute) have to process the contents of the same (single) file.

I am considering using an S3 bucket to store the files, and to use the AWS SDK to retrieve the files.

The goal is to make the application perform well and to prevent reading the file content over and over again. It is acceptable if changes to a file is not immediately visible, though this would be nice.

Is pure S3 the right approach here? Should I implement caching in Java myself, preventing an S3 request? Or is there another AWS approach that should be leveraged here?

  • 1
    Caching certainly seems like the right approach, as long as you build in a way to invalidate it after updating the S3 data. – ceejayoz Jan 8 at 16:45

It seems like in memory caching would be appropriate. I'd just read it into a data structure in memory. You could alternately use a memcached / redis server if for example you had many GB of data you don't want to store in the RAM of each application server. In Java memory would be faster.

You could check if the file had changed each time you accessed the data or at the interval you require by checking the file modification date and comparing with the date the data was last read.

Reading from S3 would likely be slower than reading from an EBS volume, I don't really see any advantage there.

  • Seems like i can use docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/… to check if file has been modified without downloading the content (yet) right? Could you elaborate on this in your answer? – Boris van Katwijk Jan 9 at 8:27
  • If you put the file on S3 there will be an API call to check the file modification date, and that one looks right. I wouldn't bother with S3 unless you have space issues on your disk or another good reason, having it on your EBS volume is likely faster. However, if it's cached, it makes little difference where it's stored. – Tim Jan 9 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.