EDITED: I have and issue in my AWS system. Every few requests takes almost exactly 130 seconds to answer. When I say a few I mean 5 to 25 or so. Normally if you cancel the slow request and send again it just answers fast. I also noticed this happens with ANY request, not only specific ones. The servers and back-end do not look overloaded. the system is as follows:
ALB with sticky sessions | 2 Web servers | DB on RDS
The system when using curl most times responds fine, but when it takes long, this is the response output:
This is a curl measuring time on any URL.
time_namelookup: 0.004136 time_connect: 130.117558 time_appconnect: 130.125254 time_pretransfer: 130.125340 time_redirect: 0.000000 time_starttransfer: 130.172553 ---------- time_total: 130.172615
Aside from the
time_connect, the request is fine in the sense that the page loads after that. normal response time of the system is under 0.5 seconds.
I was reading about this and the docs indicate
time_connect, is related to
"time_connect is the TCP three-way handshake from the client’s perspective. It ends just after the client sends the ACK - it doesn't include the time taken for that ACK to reach the server. It should be close to the round-trip time (RTT) to the server..."
This was taken from here.
Added: The system it self is nginx-Python running on ec2 instances with a MySQL DB on RDS and it serves static content from s3 and Users can upload their own files as well. from within the server (nginx-python ec2 instances) on the localhost curl is always FINE, it never takes LONG TIME. This leads me to believe is something related to the LB and the nginx listening on the python hosts.
Added: I have also tried to leave only one of the machines in the back-end and the problem does not go away.
I can not find anything meaningful on AWS Cloudwatch, the app logs or the DB monitoring. Any ideas about what I should look into or how to troubleshoot this issue?
EDIT 3 thanks to the comment bellow:
# curl -v -I -L -k -w "@time.txt" -s "https://my-site.com/url/" * Trying " * Trying IP.ONE.from.AWS... * connect to IP.ONE.from.AWS port 443 failed: Connection timed out * TCP_NODELAY set * Connected to my-site.com (IP.TWO.from.AWS) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt
IP-ONE-from-AWS and IP-TWO-from-AWS are IPs from the AWS region where I am supposed to connect.