I have a cluster with lots of different projects.
I thought it would be good to have a single S3 service that would manage all the functionality of sending files to the external S3 provider.
It would be passed files, specific project information and could be authenticated per project. It would also mean just one code-base where I wouldn't have to duplicate code and maintenance.
Client => API-1 => S3 File Service => External S3Provider
Client => API-2 => S3 File Service => External S3Provider
My only concern would be performance. Would uploading a file from the client end to the API and then copy this from the api to the S3 File Service add a considerable delay?
What if this is done on the same hardware in the cluster (unless the reached API is on a different cluster node than the S3 file service?)
Wouldn't the time increase grow longer per the size of uploaded files.
Would it just be better to remove the S3 File service and do all the S3 file uploading directly from each API?