2

I have Linode 768mb RAM server on Linode. And i have Wordpress blog. On my server installed ubuntu, nginx as frontend and apache2 as backend. And i have APC and memcache modules. Sometimes site is crashing. But CPU usage of server is less than the critical levels (only max 60-70). However, during site crashing i can see the critical levels of hard disk I/O usage. I have read that this may be related to incorrect settings of mysql.

My nginx.conf:

worker_processes 2;
events {
    worker_connections  1024;
    # multi_accept on;
}
http {
    sendfile        on;
    #tcp_nopush     on;
    #keepalive_timeout  0;
    keepalive_timeout  12;
    tcp_nodelay        on;
    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

My nginx proxy.conf:

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
#client_header_buffer_size 64k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffer_size   16k;
proxy_buffers       32   16k;
proxy_busy_buffers_size 64k;

My nginx site conf:

server {
        listen   80;
        server_name mysite.org;

        location / {
                proxy_pass  http://127.0.0.1:8080;
                include     /etc/nginx/conf.d/proxy.conf;
                root   /home/mysite/www/;
                index  index.html index.htm index.php;
        }

        location ~* ^.+\.(jpg|jpeg|cur|flv|avi|gif|png|ico|zip|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf)$ {
            root   /home/mysite/www/;
        }

        location ~* ^.+\.(htm|html|js|htc|css|tgz|gz|rar|bz2)$ {
           root   /home/mysite/www/;
           gzip_static on;
       }

My.cnf

#
# * Fine Tuning
#
key_buffer              = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M

Hardware:

cpu (4x): Intel(R) Xeon(R) CPU L5520  @ 2.27GHz, 2260 MHz 
storage: Xen Virtual Storage 0, Xen Virtual Storage 1
memory: 768mb

Apache conf:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_event_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

Linode statistics links: http://ridna.ua/temp/mysite_io_rate.png and http://ridna.ua/temp/mysite_cpu_rate.png

How i can optimize my nginx+apache2+mysql settings to avoid site downs? Thanks..

5
  • What is the actual problem? What goes wrong? Do you get error messages? Are all the services still running? Mar 16, 2012 at 1:16
  • Server goes blank page.. Nginx says: *3301 upstream timed out (110: Connection timed out) while reading response header from upstream Mar 16, 2012 at 6:47
  • I need server settings optimization for avoid server crashes.. Mar 16, 2012 at 6:51
  • Please post the following: HW specs, apache error_log, my.cnf, 'du -sh mysqls_data_dir/*', mysqls slow and error logs.
    – 3molo
    Mar 16, 2012 at 8:39
  • hardware and my.cnf i just updated in main post. apache error log dont have any suspicious errors. mysql error log of last crash says about crashed one of table of database.. Mar 16, 2012 at 13:24

2 Answers 2

2

I had these problems and fixed it by some tunings on apache and mysql

"upstream timed out" error in nginx

or

INFO: task: apache2 (or mysql or nginx) blocked for more than 120 seconds.

this problem will happening when your apache resource usage is so high and its unavailable. if mysql wont reply fast and make delay on responding to queries apache will increase his treads and your memory will go full and ... BoooOOM

your major problem is from MYSQL and one easy way to fixing it is installing mysqltuner app and do recommendations that

you will need to tune your apache at second step too! first use "top" or something similar (in heavy traffic on server) and find the maximum active thread of apache size in mb. now you must to take the rest of free server ram to apache by configuring MaxClients

for example if your ram is 12 and your mysql taked 5GB of ram - and your maximum apache tread that find is about 70mb you must to config your MaxClients about 70~80 and let the rest of ram be for OS. its so important that you config your services as good that they dont full all of available memory in heavy traffics!

0

You need to identify the actual cause of the disk I/O, rather than just guess at it. The first step in this is to use the historical data collected by sar (if you haven't installed sysstat on your machine and tweaked it to collect data every minute, do that '''now'''). Look for swapping first and foremost; both volume used and pages in/out are potentially useful.

If swap doesn't appear to be the case, double-check which block device has the disk I/O problem. That'll help to narrow down which part of the system is likely to be causing the disk I/O, if you have multiple partitions.

If you can catch the problem in action, running iotop can be really helpful to identify the offending process. It's downside is that running it non-interactively is a pest, because to get useful data you really need to be polling every second, and writing all that out to disk can cause enough disk I/O to put your machine underwater even quicker.

Once you've worked out who the offender is, you need to fix it. If MySQL is, indeed, the culprit, then yes, you'll need to tune it. I'm not going to go into a detailed article on MySQL tuning, as the Internet (and this site) are full of them already. Suffice it to say, yes, default MySQL parameters are complete crap and you always need to tune your DB.

A couple of general points on your setup:

  • Running nginx and Apache is unnecessary, wastes memory, and hence won't be helping either a swap problem or a general disk I/O problem (because there'll be less memory available for disk caching). Get rid of Apache and run php-fpm (or old school PHP FCGI) for serving your dynamic content.
  • Linode's disk I/O is generally pretty poor (we just finished migrating a big customer off Linode, in part because the disk I/O available was so poor it was causing performance problems). If you're actually going to be doing a decent amount of traffic, you should be able to afford a better class of VPS.
4
  • But i dont just guess at it. Linode have own graph daily statistic. and i can see, where is problem: on cpu or disk. See here ridna.ua/temp/mysite_io_rate.png and here ridna.ua/temp/mysite_cpu_rate.png Mar 16, 2012 at 12:48
  • Thank you for instructions. I'll try to track statistic changes.. Mar 16, 2012 at 13:00
  • You are guessing at the cause of the disk I/O, you said "I have read that this may be related to incorrect settings of mysql.". Those graphs don't tell you what is causing the disk I/O, merely that it's happening. Hence the remaining 6 paragraphs or so telling you how to solve your problem scientifically.
    – womble
    Mar 16, 2012 at 21:29
  • i installer php-fpm and stopped apache2 Mar 16, 2012 at 23:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .