Is uploading a 500mb file to s3 an expensive operation? Any way to limit cpu usage etc?

Question

On a production server, I'm planning on uploading a single file or a set of files that may be 500 megs in size.

I havent' test it just yet, but just want some opinions on whether you think this will significantly impact the server?

The reason I ask, when I am SSH'd into the server, and on my laptop if I scp a file to my server that is maybe 10 megs I notice the SSH terminal to be very slow to respond. I understand I am on a simple vps with only 512ram, but I wouldn't have thought it would impact it so much.

My product server is a xeon 3450, but again the file size is 500 megs. I know I can break the file down into smaller parts, but was wondering if I could rate limit the upload process somehow also?

Answer 1

Uploading a file is likely to consume almost no resources on the server. The resources that uploading a file can consume are (in probable, though not definite, order):

• Your outgoing bandwidth
• Server's incoming bandwidth
• Server's disk IO
• Server's CPU

Your outgoing bandwidth is likely the problem when you report that the terminal is being slow (ceejayoz mentioned this in a comment).

Your server's CPU is likely not even going to notice the transfer (unless you can somehow upload > 10MB/s). And this will likely be due to having to decompress the stream of whatever transfer protocol youre using. If youre not compressing during transfer, CPU is even less likely to bottle neck. Additionally, as the transfer will take place over a single thread, you cant use more than 1 core of the machine at the absolute most anyway.

As for how to limit it, the simplest way would be if whatever transfer mechanism youre using supported transfer rate limiting. This is unknown though as you dont state how you are transferring the files.