I have a set of servers inside an Amazon VPC, whose job is to get image files from S3 and process them. The S3 downloads would frequently stall, sometimes for over 2 minutes, but all would eventually succeed. To rule out application issues, I created a simple list of URLs in a text file, and piped them through curl:

time <images.txt xargs -I% curl -O %

There are 352 images listed in the file, with sizes varying between 13K and 1.3MB. Repeated executions of the script exhibited the same problems as the application was seeing. I could not reproduce the problem when running the script directly on the NAT instance, and I also noticed a significant overall performance improvement when running it on the NAT.

After some research, I found this serverfault question: "Amazon EC2 VPC: NAT instance download speed performance drop". The solution was to use ethtool to turn off the TCP Offload Engine scatter-gather mode option. This solved the stalling problem for me, but it does not fix the overall performance difference between the NAT and the clients behind the NAT. After many invocations, the average time to complete the script on the NAT is about 15 seconds, and about 35 seconds on clients behind the NAT.

The NAT instance type is a stock Amazon NAT t2.small (3.14.26-24.46.amzn1.x86_64). Even though CPU load was tiny on the NAT, I have tried different instance types including m1.medium and m3.2xlarge. None had any significant impact on NAT performance. I have tried tweaking a few different settings, including disabling more TOE features and increasing ip_conntrack_max and conntrack hash size. I didn't really expect these to make much difference, and my tests showed no significant changes.

Is a ~50% reduction in download speed to be expected when going through a NAT in this scenario? I understand that NAT/PAT is adding an extra hop and some extra packet processing, but it still seems like a massive drop in performance to me. If this is not to be expected, what can I do to diagnose the problem?

  • It could be the network on your downloading instance and not the NAT. What type are those? Have you tried downloading to the same type from within a public subnet? – Matt Houser Dec 19 '14 at 13:27
  • @MattHouser I have both t2.medium and r3.large instances in the private subnet, and both types exhibit the same problem. I'm going to try attaching an Elastic IP directly to one of those instances to see what happens with a (more) direct connection. – Mike Dec 19 '14 at 13:34
  • I enabled direct internet access on one of the servers in the private subnet, and it completed the download task in about the same time as doing it directly on the NAT. – Mike Dec 19 '14 at 15:04

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.