Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have Linode 768mb RAM server on Linode. And i have Wordpress blog. On my server installed ubuntu, nginx as frontend and apache2 as backend. And i have APC and memcache modules. Sometimes site is crashing. But CPU usage of server is less than the critical levels (only max 60-70). However, during site crashing i can see the critical levels of hard disk I/O usage. I have read that this may be related to incorrect settings of mysql.

My nginx.conf:

worker_processes 2;
events {
    worker_connections  1024;
    # multi_accept on;
}
http {
    sendfile        on;
    #tcp_nopush     on;
    #keepalive_timeout  0;
    keepalive_timeout  12;
    tcp_nodelay        on;
    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

My nginx proxy.conf:

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
#client_header_buffer_size 64k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffer_size   16k;
proxy_buffers       32   16k;
proxy_busy_buffers_size 64k;

My nginx site conf:

server {
        listen   80;
        server_name mysite.org;

        location / {
                proxy_pass  http://127.0.0.1:8080;
                include     /etc/nginx/conf.d/proxy.conf;
                root   /home/mysite/www/;
                index  index.html index.htm index.php;
        }

        location ~* ^.+\.(jpg|jpeg|cur|flv|avi|gif|png|ico|zip|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf)$ {
            root   /home/mysite/www/;
        }

        location ~* ^.+\.(htm|html|js|htc|css|tgz|gz|rar|bz2)$ {
           root   /home/mysite/www/;
           gzip_static on;
       }

My.cnf

#
# * Fine Tuning
#
key_buffer              = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M

Hardware:

cpu (4x): Intel(R) Xeon(R) CPU L5520  @ 2.27GHz, 2260 MHz 
storage: Xen Virtual Storage 0, Xen Virtual Storage 1
memory: 768mb

Apache conf:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_event_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

Linode statistics links: http://ridna.ua/temp/mysite_io_rate.png and http://ridna.ua/temp/mysite_cpu_rate.png

How i can optimize my nginx+apache2+mysql settings to avoid site downs? Thanks..

share|improve this question
    
What is the actual problem? What goes wrong? Do you get error messages? Are all the services still running? –  David Schwartz Mar 16 '12 at 1:16
    
Server goes blank page.. Nginx says: *3301 upstream timed out (110: Connection timed out) while reading response header from upstream –  Bohdan Hdal Mar 16 '12 at 6:47
    
I need server settings optimization for avoid server crashes.. –  Bohdan Hdal Mar 16 '12 at 6:51
    
Please post the following: HW specs, apache error_log, my.cnf, 'du -sh mysqls_data_dir/*', mysqls slow and error logs. –  3molo Mar 16 '12 at 8:39
    
hardware and my.cnf i just updated in main post. apache error log dont have any suspicious errors. mysql error log of last crash says about crashed one of table of database.. –  Bohdan Hdal Mar 16 '12 at 13:24

2 Answers 2

I had these problems and fixed it by some tunings on apache and mysql

"upstream timed out" error in nginx

or

INFO: task: apache2 (or mysql or nginx) blocked for more than 120 seconds.

this problem will happening when your apache resource usage is so high and its unavailable. if mysql wont reply fast and make delay on responding to queries apache will increase his treads and your memory will go full and ... BoooOOM

your major problem is from MYSQL and one easy way to fixing it is installing mysqltuner app and do recommendations that

you will need to tune your apache at second step too! first use "top" or something similar (in heavy traffic on server) and find the maximum active thread of apache size in mb. now you must to take the rest of free server ram to apache by configuring MaxClients

for example if your ram is 12 and your mysql taked 5GB of ram - and your maximum apache tread that find is about 70mb you must to config your MaxClients about 70~80 and let the rest of ram be for OS. its so important that you config your services as good that they dont full all of available memory in heavy traffics!

share|improve this answer

You need to identify the actual cause of the disk I/O, rather than just guess at it. The first step in this is to use the historical data collected by sar (if you haven't installed sysstat on your machine and tweaked it to collect data every minute, do that '''now'''). Look for swapping first and foremost; both volume used and pages in/out are potentially useful.

If swap doesn't appear to be the case, double-check which block device has the disk I/O problem. That'll help to narrow down which part of the system is likely to be causing the disk I/O, if you have multiple partitions.

If you can catch the problem in action, running iotop can be really helpful to identify the offending process. It's downside is that running it non-interactively is a pest, because to get useful data you really need to be polling every second, and writing all that out to disk can cause enough disk I/O to put your machine underwater even quicker.

Once you've worked out who the offender is, you need to fix it. If MySQL is, indeed, the culprit, then yes, you'll need to tune it. I'm not going to go into a detailed article on MySQL tuning, as the Internet (and this site) are full of them already. Suffice it to say, yes, default MySQL parameters are complete crap and you always need to tune your DB.

A couple of general points on your setup:

  • Running nginx and Apache is unnecessary, wastes memory, and hence won't be helping either a swap problem or a general disk I/O problem (because there'll be less memory available for disk caching). Get rid of Apache and run php-fpm (or old school PHP FCGI) for serving your dynamic content.
  • Linode's disk I/O is generally pretty poor (we just finished migrating a big customer off Linode, in part because the disk I/O available was so poor it was causing performance problems). If you're actually going to be doing a decent amount of traffic, you should be able to afford a better class of VPS.
share|improve this answer
    
But i dont just guess at it. Linode have own graph daily statistic. and i can see, where is problem: on cpu or disk. See here ridna.ua/temp/mysite_io_rate.png and here ridna.ua/temp/mysite_cpu_rate.png –  Bohdan Hdal Mar 16 '12 at 12:48
    
Thank you for instructions. I'll try to track statistic changes.. –  Bohdan Hdal Mar 16 '12 at 13:00
    
You are guessing at the cause of the disk I/O, you said "I have read that this may be related to incorrect settings of mysql.". Those graphs don't tell you what is causing the disk I/O, merely that it's happening. Hence the remaining 6 paragraphs or so telling you how to solve your problem scientifically. –  womble Mar 16 '12 at 21:29
    
i installer php-fpm and stopped apache2 –  Bohdan Hdal Mar 16 '12 at 23:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.