I run batch compute jobs on EC2 instances (specifically i3.8xlarge) and upload large (10's GB) binary files to each instance. We have "gigabit" (Century Link) at the office and I typically upload (via scp) these files to EC2 at 30-40 MBps. The local machine is a RH7 Linux box. Often, however, I observe a curious thing: the upload speed drops precipitously to 300-400 kBps.
- I verify (via speedtest.net) that the network is performing very adequately
- The reduced upload speed is ALWAYS 300-400 kBps
- The phenomenon started about a month ago
- I can upload the same file, to the same instance (different output path, obviously), at the same time, from a different tab of the Gnome terminal, and I get good performance
- The upload will typically resume a good rate unpredictably
So the bottleneck seems tied to a single scp process. Amazon doesn't advertise any type of upload throttling. Our network guys verify that the network is in good working order when I observe this.