1

When booting up an application from an AMI we noticed increased response times and increased rate of request time-out errors, slowly tapering off and going back to normal. I figured this is due to EBS lazy initialization (a well-documented performance characteristic of EBS). The application has a 24 GB EBS data volume.

I tried increasing instance sizes and noticed no difference. So, taking a step back to try to isolate the performance bottleneck, I ran some benchmarks with different instance sizes to try to find the one with the best pure EBS initialization performance, under the assumption that this will serve as a good proxy for "performance with lazy initialization during normal use of the application".

And I ran into a major surprise:

A t3.medium instance performs the same as a c5.18xlarge!

How can this be?

I'm using the fio command recommended by AWS here:

sudo fio --filename=/dev/nvme0n1 --rw=read --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize

(modified for device /dev/nvme0n1)

the larger instance has nominally 5x the network performance of the smaller one (25 Gbps vs "Up to 5 Mbps").

Both plod along at about 35 MiB/s.

Bonus question: What instance type will give me the fastest EBS and S3 peformance, including, EBS initialization from snapshot?

UPDATES

  1. Adding an S3 endpoint to the VPC made no difference.
  2. When I increase the EBS volume size to the maximum 10,000 IOPS (i.e. 3333 GB), the speed goes up to about 45 MiB/s. I'm only testing on the c5.18xlarge at this point
  • What problem is this moderately slow restore speed causing you? Why do you need to restore from snapshots super fast? Perhaps we can help solve your problem in another way. – Tim Dec 1 '18 at 7:37
  • I updated the question to explain this – Alex R Dec 2 '18 at 0:51
  • With more information we could help more. Are you trying to create an instance from an AMI because it has particular software installed, because it has a lot of data, both, or something else? I don't understand why you need super fast restores where the volume gets 100% of performance immediately. You could scale out and send traffic to the instance gradually to warm it, most of the traffic going to other instances until the new one is ready. – Tim Dec 2 '18 at 1:10
  • Getting 100% of performance immediately is not the goal, it is an indirect way to measure the performance penalty amortized during gradual warming, which is just as real but much harder to measure directly. "If you can't measure it, you can't improve it". – Alex R Dec 2 '18 at 3:01
0

Background

EBS snapshots are stored on S3 (this is documented on the link you provided above). When you restore a snapshot it pulls in blocks from S3 when they're required (documented here, copied below).

New volumes created from existing EBS snapshots load lazily in the background. This means that after a volume is created from a snapshot, there is no need to wait for all of the data to transfer from Amazon S3 to your EBS volume before your attached instance can start accessing the volume and all its data. If your instance accesses data that hasn't yet been loaded, the volume immediately downloads the requested data from Amazon S3, and continues loading the rest of the data in the background.

Updated Idea Again

As Michael points out below, the bottleneck here is likely between S3 and EBS. I'm surprised it's so low at 35MB/sec aka 280Mbps, but I guess it's a single S3 object it's retrieving from. S3 can sustain huge bandwidth, but typically with multiple objects.

Based on what Michael says I think you just have to put up with this relatively slow restore. You don't have to pre-warm the volume, you can let it happen on demand and take the hit over the first minutes / hours / days the instance is up.

|improve this answer|||||
  • I'm using fio which is multi-threaded. The AWS account is a sandbox with default settings, default VPC, no NAT. There is no S3 endpoint, I would like to understand how it might help with EBS initialization? Is this documented somewhere? – Alex R Nov 30 '18 at 15:59
  • Have a read of my edited answer. It's only a guess, someone else might have a better idea – Tim Nov 30 '18 at 19:57
  • 1
    @Tim you're on the right track -- there is a bottleneck, but the limitation is between the EBS service and S3 -- it is not between the instance (or anything inside the VPC) and S3. You can only optimize this (e.g. with fio) up to the capability of EBS to restore the blocks, and essentially any modern instance class/size has enough EBS bandwidth to max-out what EBS provides for volume warm-up on demand. – Michael - sqlbot Nov 30 '18 at 22:33
  • Ah, good point @Michael-sqlbot. 35MB/sec (280Mbps) seems pretty slow between S3 and EBS. – Tim Nov 30 '18 at 23:16
  • 1
    @Tim I expanded the question to describe the overall situation and use-case in a little more detail. – Alex R Dec 2 '18 at 0:40
0

Answer: The initialization speed for EBS Snapshots is not affected by EC2 instance type.

As of 12/1/2018.

Apparently 42 MiB/s is the maximum pre-warm / initialization rate that can be achieved from an EBS Snapshot to a single 10,000 IOPS volume. While the speed is not impacted by instance type, it does drop to 35 MiB/s on smaller volumes (100 IOPS). The speed is also not impacted by the presence of an S3 Endpoint on the VPC.

For comparison, copying directly from a live EBS volume to another, bypassing the snapshot process, performs at 128 MiB/s on an r5d.large instance, single-threaded, using a tar|pv|tar pipe, over an ext4 filesystem.

|improve this answer|||||
  • I wonder if volumes created from an AMI have this same limitation. You could try creating an AMI from the snapshot and creating a new instance from that, maybe it'll go faster. I suspect the AMI is stored in S3 and backed by a snapshot, but it's worth a shot. – Tim Dec 2 '18 at 0:08
  • They were AMIs already. I'll update – Alex R Dec 2 '18 at 0:35
  • I suspect a multi-threaded copy would go significantly faster, up to the limits of the EBS / network connection of the instance. – Tim Dec 2 '18 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.