When booting up an application from an AMI we noticed increased response times and increased rate of request time-out errors, slowly tapering off and going back to normal. I figured this is due to EBS lazy initialization (a well-documented performance characteristic of EBS). The application has a 24 GB EBS data volume.
I tried increasing instance sizes and noticed no difference. So, taking a step back to try to isolate the performance bottleneck, I ran some benchmarks with different instance sizes to try to find the one with the best pure EBS initialization performance, under the assumption that this will serve as a good proxy for "performance with lazy initialization during normal use of the application".
And I ran into a major surprise:
t3.medium instance performs the same as a
How can this be?
I'm using the
fio command recommended by AWS here:
sudo fio --filename=/dev/nvme0n1 --rw=read --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize
(modified for device /dev/nvme0n1)
the larger instance has nominally 5x the network performance of the smaller one (25 Gbps vs "Up to 5 Mbps").
Both plod along at about 35 MiB/s.
Bonus question: What instance type will give me the fastest EBS and S3 peformance, including, EBS initialization from snapshot?
- Adding an S3 endpoint to the VPC made no difference.
- When I increase the EBS volume size to the maximum 10,000 IOPS (i.e. 3333 GB), the speed goes up to about 45 MiB/s. I'm only testing on the c5.18xlarge at this point