2

I run batch compute jobs on EC2 instances (specifically i3.8xlarge) and upload large (10's GB) binary files to each instance. We have "gigabit" (Century Link) at the office and I typically upload (via scp) these files to EC2 at 30-40 MBps. The local machine is a RH7 Linux box. Often, however, I observe a curious thing: the upload speed drops precipitously to 300-400 kBps.

  • I verify (via speedtest.net) that the network is performing very adequately
  • The reduced upload speed is ALWAYS 300-400 kBps
  • The phenomenon started about a month ago
  • I can upload the same file, to the same instance (different output path, obviously), at the same time, from a different tab of the Gnome terminal, and I get good performance
  • The upload will typically resume a good rate unpredictably

So the bottleneck seems tied to a single scp process. Amazon doesn't advertise any type of upload throttling. Our network guys verify that the network is in good working order when I observe this.

2 Answers 2

3

I suspect your EBS Burst Balance has run out. That blog link gives you instructions on how to show your EBS volume burst credits.

The volume types page has a little information on EBS and bursting.

General Purpose SSD (gp2) volumes offer cost-effective storage that is ideal for a broad range of workloads. These volumes deliver single-digit millisecond latencies and the ability to burst to 3,000 IOPS for extended periods of time. Between a minimum of 100 IOPS (at 33.33 GiB and below) and a maximum of 16,000 IOPS (at 5,334 GiB and above), baseline performance scales linearly at 3 IOPS per GiB of volume size. AWS designs gp2 volumes to deliver their provisioned performance 99% of the time. A gp2 volume can range in size from 1 GiB to 16 TiB.

In short, you accumulate burst credits, and as you use your disk a lot you run out of credits. When that happens you get baseline performance. The larger the disk, the more credits, and the less likely you are to run out of credits. There are various articles (e.g. this one) you can read about them.

Instance store volumes do not have a burst system, which is why they work fast all the time.

The solution here is probably to buffer your writes to the instance store, then transfer them to the EBS volume in the background. rsync might be suitable.

6
  • Tim, thanks. I can't upvote you on my own question due to lack of "reputation". But your explanation seems to plausibly explain the observations. I am looking at my "Burst Balance" and weirdly, it's only showing data for the last 15 hours. That said, the burst balance is at 100% for all instances. Mar 16, 2020 at 15:54
  • Look at that metric when the bandwidth drops to very low, see if it goes down to zero.
    – Tim
    Mar 16, 2020 at 18:10
  • Will do. Unfortunately I typically terminate my instances when I'm done with a run. The EBS volumes also are terminated. I guess the Burst Balance tool only shows active volumes. Mar 16, 2020 at 18:19
  • You'll probably have to have a look when it's happening. I wonder if the EBS burst balance starts at max or some intermediate value.
    – Tim
    Mar 16, 2020 at 18:56
  • Well, I'm uploading data again, only now I'm uploading to the onboard SSD, not EBS. Performance is running in the slow 300-400 kB/sec band. I don't have tech support but I'm a big spender so I get a "Solution Architect" via the sales team. Will see what they say. Mar 18, 2020 at 3:19
1

Well, so I was able to "solve" (?) the problem by writing the files to the onboard NVMe SSDs instead of the EBS storage. Still, this makes no sense. Surely EBS has better write capabilities than 300-400 kBps!

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .