I have 2 servers using the same data located in AWS S3 storage. One is processing HTTP traffic, the other does longer lived jobs. How it works is that HTTP one saves data to S3, dispatches event to backend, and the data is processed.
However, it seems that every couple of minutes there's an error in the backend because the data on S3 does not exist yet. Is there a delay between uploading files to S3 and having them available to other client connections?
I'm sure the upload has finished by the time backend access those files, and the HTTP server uses keep-alive connection to S3.