4

I have two c3.2xlarge EC2 machines with Ubuntu environment both in us-west-2a AZ. Both contains same code with mySQL database from AWS RDS (db.r3.2xlarge). Both instances are added to an ELB. Both has one cron scheduled that runs twice in a day.

ELB has been configured to raise the alarm once the threshold crosses 5.0. The CPU utilization of both the instances are by average 30 - 50. At peak hours hits 100% for a minute or two and then returns to normal. But ELB constantly raises alarm thrice a day. At this time, both instances has

CPU     - ~50%
Memory  - total - 14979
          used  - ~6000
          free  - ~9000
RDS CPU - ~30%
          Connections - 200 to 300 /5,000

According to this https://aws.amazon.com/premiumsupport/knowledge-center/elb-latency-troubleshooting/ I could find nothing wrong with the instances. But still latency hits the peak and both instance fails to respond.

Till now, I am just removing one of the instance from the load balancer, restart the apache and then load it back and do the same for other instance. This does the job perfectly alright and the instances and ELB works good for next 6-10 hours. But this is not acceptable since, every day twice or thrice one has to take care of the server, needs it to restart.

I need to know, if there is anything wrong or any steps to be taken to resolve this problem.

Latency

Memory

Apache server-status contains too many such (~200/250 processes):

7-0 23176   1/2373/5118 C   30.95   3986    0   0.0 7.01    15.78   127.0.0.1   ip-xxx-xxx-xxx-xxx.us-west-2.comp   OPTIONS * HTTP/1.0
5
  • What happens if you turn off the cron tasks? Mar 7, 2016 at 20:12
  • @MattHouser Cron is scheduled to run only twice and it doesn't take much resource, since some 10-20 update queries and some file write operations in it that gets completed in less than 10s. I mentioned that to ensure that no cron is running other than this.
    – Thamilhan
    Mar 8, 2016 at 6:01
  • Can you tell, on your instances, what the disk and/or network I/O looks like? I.e. could your mysql instances be waiting on disk I/O (and thus adding latency there), or on network I/O? Another possibility is the TCP backlog, as described here.
    – Castaglia
    Mar 11, 2016 at 19:35
  • @Castaglia Disk reads - 100,000 to 150,000, Disk writes - 0 ; N/w In - 100,000,000 to 125,000,000, N/w out - 25,000,000 to 35,000,000.
    – Thamilhan
    Mar 14, 2016 at 8:57
  • @Castaglia Please check the updated images in the question.
    – Thamilhan
    Mar 14, 2016 at 9:25

1 Answer 1

2
+50

CPU utilization (%) is not the key, key is CPU load average (queue) and networking metrics, apache metrics, buffers, etc. Load balancers are very simple devices, problems, where LB's are involved in architecture usually are not related to the ELB's, but to the nature of how rest of the things work.

To see where is the problem, you most go through following steps:

  • Check if apache is responding to the local requests, if not - problem is NOT the ELB
  • Check states of apache workers (i.e. mod_status), tune MPM settings accordingly
  • Check CPU load average, if load average grows above CPU count and iowait grows - you have trouble with IO
  • Check if connection persistency is enabled and if it is really really required, if you really use sessions on webservers which require access to the same web instance
  • Check keepalive settings for apache, disable it or set very low timeout value
  • Check if you have iptables enabled on the instance and if nf_conntrack_max and nf_conntrack_count kernel parameters are configured with higher values. If you don't need it - disable and do not load modules at all
  • Stress test single instances wit http requests (hint: ab, jmeter)
  • Check and tune kernel parameters accordingly:

    net.core.wmem_max
    net.core.rmem_max
    net.core.netdev_max_backlog
    
    net.core.somaxconn
    net.ipv4.tcp_rmem
    net.ipv4.tcp_wmem
    net.ipv4.tcp_no_metrics_save
    net.ipv4.tcp_timestamps
    net.ipv4.tcp_fin_timeout
    net.ipv4.tcp_max_tw_buckets
    net.ipv4.tcp_tw_recycle
    net.ipv4.tcp_synack_retries
    net.ipv4.tcp_keepalive_time
    
    net.netfilter.nf_conntrack_acct
    net.netfilter.nf_conntrack_generic_timeout
    net.netfilter.nf_conntrack_tcp_timeout_syn_sent
    net.netfilter.nf_conntrack_tcp_timeout_syn_recv
    net.netfilter.nf_conntrack_tcp_timeout_established
    net.netfilter.nf_conntrack_tcp_timeout_fin_wait
    net.netfilter.nf_conntrack_tcp_timeout_close_wait
    net.netfilter.nf_conntrack_tcp_timeout_last_ack
    net.netfilter.nf_conntrack_tcp_timeout_time_wait
    net.netfilter.nf_conntrack_tcp_timeout_close
    net.netfilter.nf_conntrack_tcp_timeout_max_retrans
    net.netfilter.nf_conntrack_tcp_timeout_unacknowledged
    net.netfilter.nf_conntrack_icmp_timeout
    net.netfilter.nf_conntrack_events_retry_timeout
    net.ipv4.netfilter.ip_conntrack_generic_timeout
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_close
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans
    net.ipv4.netfilter.ip_conntrack_icmp_timeout
    net.netfilter.nf_conntrack_tcp_loose
    net.netfilter.nf_conntrack_max net.nf_conntrack_max
    net.netfilter.nf_conntrack_count
    

Apache is not responding after that? It's not the fault of ELB at all.

5
  • 1. Apache server status won't work, but site loads. (Possibly due to already allocated slot) 2. Keep Alive is enabled with 100 alive requests with 5s timeout.
    – Thamilhan
    Mar 14, 2016 at 9:15
  • Please check the updated images in the question.
    – Thamilhan
    Mar 14, 2016 at 9:25
  • If you can't check status from localhost, then ELB will encounter same problem with site too, which means, you have problem with the server/service (TCP/IP stack, tracking, apache, backend) not the ELB. ELB is working as expected. After restart, slots are freed and connection reset, which means everything will work again.
    – GioMac
    Mar 16, 2016 at 12:39
  • So how should I fix that? checked mySQL connections, all are closed properly in process list. But I detect two problems here: 1. Too many close connections in apache server status, 2. More than 15 apache process running in memory in both ec2 instances
    – Thamilhan
    Mar 17, 2016 at 1:42
  • You should check your application and apache mpm settings
    – GioMac
    Mar 17, 2016 at 12:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .