0

My web server (Ubuntu, Apache and Redis-server) is hitting 100% CPU on peak times. There is a rapid increase on CPU usage after redirecting all traffic to HTTPS. We are using PHP 5.6, Apache 2.2, Redis server and GeoIP on server.

Server has enough memory and it only using very less memory.

Is it possible to do anything to reduce CPU usage?

top - 11:30:01 up 115 days,  2:48,  1 user,  load average: 190.86, 270.08, 159.08
Tasks: 1612 total, 155 running, 1456 sleeping,   0 stopped,   1 zombie
%Cpu(s): 75.5 us, 19.3 sy,  0.0 ni,  2.1 id,  1.8 wa,  0.0 hi,  1.2 si,  0.0 st
KiB Mem:  61836576 total, 58087948 used,  3748628 free,  1391660 buffers
KiB Swap:        0 total,        0 used,        0 free. 36999240 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 
106731 www-data  20   0  534768  34936  16644 S  27.5  0.1   0:11.64 /usr/sbin/apache2 -k start                                                                                             
105089 www-data  20   0  536356  36036  16148 S  27.2  0.1   0:06.96 /usr/sbin/apache2 -k start 
105942 www-data  20   0  534580  38780  20696 R  22.3  0.1   0:14.38 /usr/sbin/apache2 -k start 
106552 www-data  20   0  534516  38404  20364 S  20.3  0.1   0:09.20 /usr/sbin/apache2 -k start 
104105 www-data  20   0  538160  40876  19756 R  20.0  0.1   0:16.93 /usr/sbin/apache2 -k start 
102426 www-data  20   0  534728  40544  22268 S  19.7  0.1   0:14.81 /usr/sbin/apache2 -k start                                                                                           
106733 www-data  20   0  537724  39960  19292 R  19.7  0.1   0:11.38 /usr/sbin/apache2 -k start 

My SSL configuration is:

<VirtualHost *:443>
        ServerAdmin support@domain.com
        ServerName domain.com
        ServerAlias www.domain.com
        DocumentRoot /home/domain/public_html
        <Directory /home/domain/public_html/>
                Options -Includes -ExecCGI +FollowSymLinks
                SSLRequireSSL
                AllowOverride All
                Order allow,deny
                allow from all
                Require all granted
        </Directory>
        SSLEngine on
        SSLVerifyClient none
        SSLCertificateFile /etc/apache2/ssl/ssl.cert
        SSLCertificateKeyFile /etc/apache2/ssl/ssl.key
        SSLCertificateChainFile /etc/apache2/ssl/sslIntermediate.crt
        SSLHonorCipherOrder On
        SSLCipherSuite ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:!RC4:HIGH:!MD5:!aNULL:!EDH:!3DES:!DES:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!MEDIUM

        SSLProtocol -ALL +TLSv1.1 +TLSv1.2
        SSLCompression Off
        ErrorLog ${APACHE_LOG_DIR}/domain-error.log
        LogLevel error
        CustomLog ${APACHE_LOG_DIR}/domain-access.log combined
</VirtualHost>

mpm_prefork.conf:

<IfModule mpm_prefork_module>
        StartServers             50
        MinSpareServers          20
        MaxSpareServers          20
          ServerLimit          1200
        MaxClients           1200
        #MaxRequestWorkers       800
        MaxConnectionsPerChild   1000
</IfModule>

Monitoring systems shows Apache is using more CPU, on peak time more than 1200 apache process running, and it seems it's hitting the limit. MySQL server is also getting load.

Will installing SSL on ELB solve the issue? Or any other best practices or change in turning values will resolve my issue?

  • "Is it possible to do anything to reduce CPU usage?" -- do less work. – womble Oct 17 '17 at 6:30
  • Page caching can reduce load by an order of magnitude if there are a significant number of anonymous users. Not having to invoke PHP is a huge saving of CPU. – Tim Oct 17 '17 at 6:55
  • your issue is php, debug it and enable all debugging options you can in php.ini and forget apache httpd directives, because changing them won't make magic. Basically you are burdening your http server with php interpreting capabilities through mod_php, and there is where you should research the issue. There are lots of questions like this everywhere. – ezra-s Oct 19 '17 at 8:15
1

Yes, offloading SSL to an Amazon Load Balancer is a good idea. Encryption / decryption is CPU intensive. Plus your SSL certificates are free and auto renew.

Take a look at your apache log files and review your web site traffic. If you are seeing normal traffic, then SSL offload or increasing your instance size might be necessary (or both). However, if you are seeing a lot of bad traffic (hackers), then consider adding both a load balancer and a firewall (Amazon WAF). I always setup an ALB or NLB plus WAF for all web sites.

If you do add a load balancer, make sure that customers cannot access your web site via its IP address, otherwise hackers will go around your load balancer (not on purpose as they just scan IP addresses). This can be accomplished by either putting your web site inside a private VPC subnet (recommended) or specifying a security group entry only allowing traffic from your load balancer.

If your web site traffic is normal, but tends to increase during certain time periods often (daily) consider adding an ASG (auto scaling group) to handle the traffic increases during those busy hours. This will also add higher availability to your web site setup.

I recommend reviewing your Security Groups to make sure that all non-required ports are blocked. If you are allowing RDP or SSH, only allow access to these ports via your IP address and not 0.0.0.0/0. You can change this IP address via the console whenever your IP address changes.

  • Can I use my current ssl certificate on ELB ? Is it possible to reduce load by changing turning values before adding ELB ? – adminz Oct 16 '17 at 18:58
  • Yes, you can use your current SSL certificate provided that you have your private key. Amazon has published documents for importing SSL certificates. Note: ELB is now a family of load balancers. What used to be called ELB is now referred to as "Classic Load Balancer". You should consider ALB or NLB instead of CLB. For your second question, review your log files to find out why your CPU load is so high. Then you will have better information to take the next steps. – John Hanley Oct 16 '17 at 19:03
  • seems the traffic is legitimate, we already getting high traffic on peak times and after switching to ssl it gone beyond. – adminz Oct 16 '17 at 19:19
  • 1
    If the traffic is normal and the high CPU usage started after switching all traffic to HTTPS, then you need more CPU power. Either add a load balancer to offload SSL processing, increase your instance size, or auto scale when CPU usage is higher than XX % (something like 50% - 75%). I use 40% for ASG. Another item to consider is your page load time. This needs to be quick (usually under a 1000 ms). If your page load times are high then you will lose traffic and is another sign of poor performance. – John Hanley Oct 16 '17 at 19:36
1

To reduce the number of calls to your server, if you have many static images and they do not change often, you could add this to your Apache:

ExpiresActive On
ExpiresDefault "access plus 1 week"

Or use ExpiresByType depending on your need. Where I work, it went from dying server to quiet.

Also, MaxConnectionsPerChild to 1000 is a bit small if you have a high traffic site. This means that every 1000 requests processed, the client dies and is restarted by the parent. If your Apache is otherwise stable, you could go much much higher and avoid restarting processes all the time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.