2

EDITED: I have and issue in my AWS system. Every few requests takes almost exactly 130 seconds to answer. When I say a few I mean 5 to 25 or so. Normally if you cancel the slow request and send again it just answers fast. I also noticed this happens with ANY request, not only specific ones. The servers and back-end do not look overloaded. the system is as follows:

ALB with sticky sessions | 2 Web servers | DB on RDS

The system when using curl most times responds fine, but when it takes long, this is the response output:

This is a curl measuring time on any URL.

    time_namelookup:  0.004136
       time_connect:  130.117558
    time_appconnect:  130.125254
   time_pretransfer:  130.125340
      time_redirect:  0.000000
 time_starttransfer:  130.172553
                    ----------
         time_total:  130.172615

Aside from the time_connect, the request is fine in the sense that the page loads after that. normal response time of the system is under 0.5 seconds.

I was reading about this and the docs indicate time_connect, is related to

"time_connect is the TCP three-way handshake from the client’s perspective. It ends just after the client sends the ACK - it doesn't include the time taken for that ACK to reach the server. It should be close to the round-trip time (RTT) to the server..."

This was taken from here.

Added: The system it self is nginx-Python running on ec2 instances with a MySQL DB on RDS and it serves static content from s3 and Users can upload their own files as well. from within the server (nginx-python ec2 instances) on the localhost curl is always FINE, it never takes LONG TIME. This leads me to believe is something related to the LB and the nginx listening on the python hosts.

Added: I have also tried to leave only one of the machines in the back-end and the problem does not go away.

I can not find anything meaningful on AWS Cloudwatch, the app logs or the DB monitoring. Any ideas about what I should look into or how to troubleshoot this issue?

EDIT 3 thanks to the comment bellow:

# curl -v -I -L -k -w "@time.txt" -s "https://my-site.com/url/"
*   Trying "
*   Trying IP.ONE.from.AWS...
* connect to IP.ONE.from.AWS port 443 failed: Connection timed out
* TCP_NODELAY set
* Connected to my-site.com (IP.TWO.from.AWS) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt

IP-ONE-from-AWS and IP-TWO-from-AWS are IPs from the AWS region where I am supposed to connect.

6
  • 2
    Your question is really difficult to understand. Can you edit it to give more context, such as it's purpose, architecture, and components? Your numbers on curl are also someone imprecise about exactly what it's measuring. Also, you've said 1 minute 30 seconds but then written 130 seconds, which is 2 minutes 10 seconds - this can be important due to a standard 30 second timeout, so multiples of 30 seconds can indicate where the problem lies.
    – Tim
    Jun 30, 2020 at 1:12
  • Your comment is very useful to make the question better. Appreciate it. thanks. I will fix and come back to you.
    – wti
    Jun 30, 2020 at 1:16
  • @Tim please check again if this makes more sense to you.
    – wti
    Jun 30, 2020 at 1:22
  • 2
    Your question is still really difficult to understand. All I can suggest is to put logging at every point you can (load balancer, web, app, db, etc) and correlate them so you can isolate the tier where the problem's occurring, and from there you can drill down.
    – Tim
    Jun 30, 2020 at 2:07
  • 2
    When curl encounters the 130 second delay, what's logged at the top (curl -v) about connection establishment? Are there multiple IP addresses mentioned? Are they IPv4/v6 addresses? Any mention of retries? Or timeouts? Jun 30, 2020 at 9:51

1 Answer 1

3

You have placed your load balancer on one public subnet and one private subnet, which is an invalid configuration and will result in behavior similar to what you observe, because a balancer is assigned at least one public IP for each subnet to which it is attached... but by definition, public IP addresses don't work unless the subnet is a public subnet.

Internet-facing load balancers need to be attached only to public subnets. They do not need to be attached to the private subnets where the instances behind them are (or should be) deployed, or on any other private subnet.

Alternately, you may have intended to place the balancer on two public subnets, but one of them has a misconfigured VPC route table or Network ACL, which has the same net effect and is blocking traffic when you connect to that IP address.

1
  • Thanks, this saved a lot of time.
    – arv1989
    Apr 14, 2022 at 12:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .