Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have 2 servers using the same data located in AWS S3 storage. One is processing HTTP traffic, the other does longer lived jobs. How it works is that HTTP one saves data to S3, dispatches event to backend, and the data is processed.

However, it seems that every couple of minutes there's an error in the backend because the data on S3 does not exist yet. Is there a delay between uploading files to S3 and having them available to other client connections?

I'm sure the upload has finished by the time backend access those files, and the HTTP server uses keep-alive connection to S3.

share|improve this question

1 Answer 1

Due to how S3 distributes data once uploaded and how they load-balance their front end requests, it would be quite conceivable that an object might take a couple of seconds to fully propogate around their infrastructure. I would recommend just putting a delay of a few seconds in your logic (between upload and retrieve) or possibly just detect a failure then wait a few seconds and retry.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.