I have a set of servers inside an Amazon VPC, whose job is to get image files from S3 and process them. The S3 downloads would frequently stall, sometimes for over 2 minutes, but all would eventually succeed. To rule out application issues, I created a simple list of URLs in a text file, and piped them through curl:
time <images.txt xargs -I% curl -O %
There are 352 images listed in the file, with sizes varying between 13K and 1.3MB. Repeated executions of the script exhibited the same problems as the application was seeing. I could not reproduce the problem when running the script directly on the NAT instance, and I also noticed a significant overall performance improvement when running it on the NAT.
After some research, I found this serverfault question: "Amazon EC2 VPC: NAT instance download speed performance drop". The solution was to use ethtool to turn off the TCP Offload Engine scatter-gather mode option. This solved the stalling problem for me, but it does not fix the overall performance difference between the NAT and the clients behind the NAT. After many invocations, the average time to complete the script on the NAT is about 15 seconds, and about 35 seconds on clients behind the NAT.
The NAT instance type is a stock Amazon NAT t2.small (3.14.26-24.46.amzn1.x86_64). Even though CPU load was tiny on the NAT, I have tried different instance types including m1.medium and m3.2xlarge. None had any significant impact on NAT performance. I have tried tweaking a few different settings, including disabling more TOE features and increasing ip_conntrack_max and conntrack hash size. I didn't really expect these to make much difference, and my tests showed no significant changes.
Is a ~50% reduction in download speed to be expected when going through a NAT in this scenario? I understand that NAT/PAT is adding an extra hop and some extra packet processing, but it still seems like a massive drop in performance to me. If this is not to be expected, what can I do to diagnose the problem?