Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a small website with a streaming-chat. We really have not much traffic and a dedicated server. But now something bugs me. Sometimes the Server takes up to 20 seconds to respond to a request. While I highly suspect mySQL as the Troublemaker (as at the same time a static Html page such as munin is served near instantly). But soon afterwards it runs just as expected. I checked the logs and looked over the croinjobs... but nothing correlates with the slowdowns or indicates them.

The apache2 is prefork with mod_php5 and xcache. So I don't think PHP is causing the slowdown.

What I did to find out whats wrong: I installed munin, checked iostats, mytop, htop, optimized my Queries and changed a frequently called Query that I suspected earlier to delayed as well as removing the indexes from the table, as it's hardly queried but often written to.

RAM:

root@server ~ # free -m                                                          
             total       used       free     shared    buffers     cached       
Mem:          2002       1817        184          0        122       1051       
-/+ buffers/cache:        644       1357                                        
Swap:         2046          0       2046

So RAM is okay with 1300 MB free.

iostat:

root@server ~ # iostat -x 1 3
Linux 3.2.0-26-generic (server)  07/18/2012      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.24    1.72    1.43    2.15    0.00   89.47

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.20     2.56    1.35    4.20    26.19    74.90    36.41     0.12   21.17    3.12   26.99   6.29   3.49
md1               0.00     0.00    0.00    0.00     0.02     0.00     6.55     0.00    0.00    0.00    0.00   0.00   0.00
md2               0.00     0.00    2.83    6.10    49.23    72.25    27.22     0.00    0.00    0.00    0.00   0.00   0.00
md0               0.00     0.00    0.02    0.05     0.08     0.20     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.19     2.56    1.11    4.20    23.18    74.90    36.94     0.13   23.88    3.34   29.32   6.30   3.35

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.02    0.00    1.01    0.00    0.00   96.97

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    3.03     0.00    12.12     8.00     0.03    9.33    0.00    9.33   9.33   2.83
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md2               0.00     0.00    0.00    1.01     0.00     4.04     8.00     0.00    0.00    0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    3.03     0.00    12.12     8.00     0.02    6.67    0.00    6.67   6.67   2.02

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00    8.08    0.00   91.92

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     5.05    0.00    6.06     0.00    44.44    14.67     0.12   19.33    0.00   19.33  19.33  11.72
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md2               0.00     0.00    0.00    9.09     0.00    36.36     8.00     0.00    0.00    0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     5.05    0.00    6.06     0.00    44.44    14.67     0.11   18.00    0.00   18.00  18.00  10.91

load average:

load average: 0.36, 0.39, 0.45       

ps ax:

PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 /sbin/init
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:04 [ksoftirqd/0]
    5 ?        S      0:00 [kworker/u:0]
    6 ?        S      0:00 [migration/0]
    7 ?        S      0:00 [watchdog/0]
    8 ?        S<     0:00 [cpuset]
    9 ?        S<     0:00 [khelper]
   10 ?        S      0:00 [kdevtmpfs]
   11 ?        S<     0:00 [netns]
   12 ?        S      0:00 [sync_supers]
   13 ?        S      0:00 [bdi-default]
   14 ?        S<     0:00 [kintegrityd]
   15 ?        S<     0:00 [kblockd]
   16 ?        S<     0:00 [ata_sff]
   17 ?        S      0:00 [khubd]
   18 ?        S<     0:00 [md]
   22 ?        S      0:00 [khungtaskd]
   23 ?        S      0:00 [kswapd0]
   24 ?        SN     0:00 [ksmd]
   25 ?        SN     0:00 [khugepaged]
   26 ?        S      0:00 [fsnotify_mark]
   27 ?        S      0:00 [ecryptfs-kthrea]
   28 ?        S<     0:00 [crypto]
   36 ?        S<     0:00 [kthrotld]
   63 ?        S<     0:00 [devfreq_wq]
  137 ?        S      0:00 [scsi_eh_0]
  142 ?        S      0:00 [scsi_eh_1]
  207 ?        S      0:00 [scsi_eh_2]
  208 ?        S      0:00 [scsi_eh_3]
  209 ?        S      0:00 [kworker/u:3]
  223 ?        S      0:04 [kworker/0:2]
  236 ?        S      0:00 [md1_raid1]
  243 ?        S      0:08 [md2_raid1]
  250 ?        S      0:00 [md0_raid1]
  314 ?        S      0:01 [jbd2/md2-8]
  315 ?        S<     0:00 [ext4-dio-unwrit]
  377 ?        S      0:00 upstart-udev-bridge --daemon
  385 ?        Ss     0:00 /sbin/udevd --daemon
  485 ?        S      0:00 /sbin/udevd --daemon
  489 ?        S      0:00 /sbin/udevd --daemon
  551 ?        S<     0:00 [edac-poller]
  554 ?        S      0:04 [kworker/0:3]
  587 ?        S      0:00 [kjournald]
  638 ?        Sl     0:03 rsyslogd -c5
  673 ?        S      0:00 upstart-socket-bridge --daemon
  693 ?        Ss     0:00 dbus-daemon --system --fork --activation=upstart
  715 ?        Ss     0:00 /usr/sbin/sshd -D
  805 ?        Ss     0:01 /usr/sbin/munin-node
  836 tty4     Ss+    0:00 /sbin/getty -8 38400 tty4
  847 tty5     Ss+    0:00 /sbin/getty -8 38400 tty5
  864 tty2     Ss+    0:00 /sbin/getty -8 38400 tty2
  865 tty3     Ss+    0:00 /sbin/getty -8 38400 tty3
  868 tty6     Ss+    0:00 /sbin/getty -8 38400 tty6
  870 ?        Ss     0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
  878 ?        Ss     0:00 cron
  879 ?        Ss     0:00 atd
  958 ?        S      0:01 [flush-9:2]
 1350 ?        Ss     0:02 amavisd (master)
 1392 ?        Ss     0:00 policyd-weight (master)
 1393 ?        Ss     0:00 policyd-weight (cache)
 1402 ?        Ss     0:00 /usr/sbin/postgrey --pidfile=/var/run/postgrey.pid --daemonize --inet=10023
 1427 ?        S      0:00 /usr/sbin/courierlogger -pid=/var/run/courier/authdaemon/pid -start /usr/lib/courier/courier-authlib/authdaemond
 1428 ?        S      0:00 /usr/lib/courier/courier-authlib/authdaemond
 1451 ?        S      0:00 /usr/sbin/courierlogger -pid=/var/run/courier/imapd.pid -start -name=imapd /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=
 1452 ?        S      0:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20 -nodnslookup -noidentlookup 143 /usr/lib/courier/courier/imaplogin /usr/
 1462 ?        S      0:00 /usr/lib/courier/courier-authlib/authdaemond
 1463 ?        S      0:00 /usr/lib/courier/courier-authlib/authdaemond
 1464 ?        S      0:00 /usr/lib/courier/courier-authlib/authdaemond
 1476 ?        S      0:00 /usr/sbin/courierlogger -pid=/var/run/courier/pop3d.pid -start -name=pop3d /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnsloo
 1477 ?        S      0:00 /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 /usr/lib/courier/courier/courierpop3login
 1649 ?        Ss     0:00 /usr/lib/postfix/master
 1669 ?        S      0:00 qmgr -l -t fifo -u
 2165 ?        S      0:07 amavisd (ch5-avail)
 2170 ?        S      0:08 amavisd (ch5-avail)
 2173 ?        S      0:01 amavisd (ch4-avail)
 2294 ?        Ss     0:01 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 104:110
 2330 ?        Ss     0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
 2354 ?        Ss     0:00 proftpd: (accepting connections)
 2477 ?        S      0:00 /var/www/ispcp/daemon/ispcp_daemon -p /var/run/ispcp_daemon.pid
 2570 tty1     Ss+    0:00 /sbin/getty -8 38400 tty1
 4781 ?        Ss     0:04 /usr/sbin/apache2 -k start
 4785 ?        S      1:01 /usr/bin/perl /var/www/ispcp/engine/ispcp-apache-logger -t error
 4786 ?        S      0:23 /usr/bin/perl /var/www/ispcp/engine/ispcp-apache-logger
 4821 ?        Ssl    2:56 /usr/sbin/mysqld
 5674 ?        S      0:29 /usr/bin/php -f /var/yacrp/yac.php
10494 ?        S      0:00 policyd-weight (child)
22572 ?        S      0:00 /usr/lib/gamin/gam_server
23944 ?        S      0:01 /usr/sbin/apache2 -k start
24025 ?        S      0:01 /usr/sbin/apache2 -k start
24571 ?        S      0:01 /usr/sbin/apache2 -k start
24572 ?        S      0:01 /usr/sbin/apache2 -k start
25166 ?        S      0:00 /usr/sbin/apache2 -k start
25167 ?        S      0:00 /usr/sbin/apache2 -k start
25174 ?        S      0:00 pickup -l -t fifo -u -c
25192 ?        S      0:01 /usr/sbin/apache2 -k start
25194 ?        S      0:01 /usr/sbin/apache2 -k start
25207 ?        S      0:00 /usr/sbin/apache2 -k start
25743 ?        S      0:00 /usr/sbin/apache2 -k start
25744 ?        Ss     0:00 sshd: root@pts/1
25760 pts/1    Ss     0:00 -bash
25816 ?        S      0:00 /usr/sbin/apache2 -k start
25817 ?        S      0:00 /usr/sbin/apache2 -k start
25819 ?        S      0:00 /usr/sbin/apache2 -k start
25821 ?        S      0:00 /usr/sbin/apache2 -k start
25827 ?        S      0:00 /usr/sbin/apache2 -k start
25829 ?        S      0:00 /usr/sbin/apache2 -k start
26483 pts/1    R+     0:00 ps ax

vmstat:

root@server ~ # vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  1    172 119388 129544 1142364    0    0    51    78   77  304  7  1 89  2

iotop:

root@server ~ # iotop -b -n1
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    5 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u:0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    8 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [cpuset]
    9 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [khelper]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kdevtmpfs]
   11 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [netns]
   12 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [sync_supers]
   13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [bdi-default]
   14 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kintegrityd]
   15 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kblockd]
   16 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ata_sff]
   17 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [khubd]
   18 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [md]
27667 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
   22 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [khungtaskd]
   23 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kswapd0]
   24 be/5 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksmd]
   25 be/7 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [khugepaged]
   26 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [fsnotify_mark]
   27 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ecryptfs-kthrea]
   28 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [crypto]
   36 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthrotld]
  551 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [edac-poller]
  554 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:3]
22572 be/4 vmail       0.00 B/s    0.00 B/s  0.00 %  0.00 % gam_server
 4786 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % perl /var/www/ispcp/engine/ispcp-apache-logger
27656 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
27706 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % -bash
 2570 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % getty -8 38400 tty1
   63 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [devfreq_wq]
27659 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
  587 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kjournald]
25166 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
25167 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
26552 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
27690 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % sshd: root@pts/1
25174 be/4 postfix     0.00 B/s    0.00 B/s  0.00 %  0.00 % pickup -l -t fifo -u -c
 1649 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % master
 2165 be/4 amavis      0.00 B/s    0.00 B/s  0.00 %  0.00 % amavisd (ch5-avail)
 2170 be/4 amavis      0.00 B/s    0.00 B/s  0.00 %  0.00 % amavisd (ch5-avail)
 2173 be/4 amavis      0.00 B/s    0.00 B/s  0.00 %  0.00 % amavisd (ch5-avail)
  638 be/4 syslog      0.00 B/s    0.00 B/s  0.00 %  0.00 % rsyslogd -c5
 1669 be/4 postfix     0.00 B/s    0.00 B/s  0.00 %  0.00 % qmgr -l -t fifo -u
  137 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [scsi_eh_0]
27671 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
  142 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [scsi_eh_1]
27658 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
  673 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % upstart-socket-bridge --daemon
 4781 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
 4785 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % perl /var/www/ispcp/engine/ispcp-apache-logger -t error
  690 be/4 syslog      0.00 B/s    0.00 B/s  0.00 %  0.00 % rsyslogd -c5
  691 be/4 syslog      0.00 B/s    0.00 B/s  0.00 %  0.00 % rsyslogd -c5
  692 be/4 syslog      0.00 B/s    0.00 B/s  0.00 %  0.00 % rsyslogd -c5
  693 be/4 messageb    0.00 B/s    0.00 B/s  0.00 %  0.00 % dbus-daemon --system --fork --activation=upstart
  715 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % sshd -D
  207 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [scsi_eh_2]
  208 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [scsi_eh_3]
  209 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u:3]
 4821 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
25816 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
27866 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
25819 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
25821 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
  223 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:2]
 4833 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4834 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4835 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4836 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4837 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4838 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4839 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4840 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4841 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4842 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
  236 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [md1_raid1]
 4845 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4846 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 4847 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
  243 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [md2_raid1]
 2294 be/4 ntp         0.00 B/s    0.00 B/s  0.00 %  0.00 % ntpd -p /var/run/ntpd.pid -g -u 104:110
 4856 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
  250 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [md0_raid1]
 5674 be/4 vu2004      0.00 B/s    0.00 B/s  0.00 %  0.00 % php -f /var/yacrp/yac.php
10494 be/4 polw        0.00 B/s    0.00 B/s  0.00 %  0.00 % policyd-weight (child)
29057 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 2330 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
  805 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % munin-node
 2354 be/4 nobody      0.00 B/s    0.00 B/s  0.00 %  0.00 % proftpd: (accepting connections)
29065 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
28473 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
  314 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [jbd2/md2-8]
  315 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ext4-dio-unwrit]
  836 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % getty -8 38400 tty4
 1350 be/4 amavis      0.00 B/s    0.00 B/s  0.00 %  0.00 % amavisd (master)
26539 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
  847 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % getty -8 38400 tty5
28474 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
  864 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % getty -8 38400 tty2
  865 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % getty -8 38400 tty3
  868 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % getty -8 38400 tty6
  870 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % acpid -c /etc/acpi/events -s /var/run/acpid.socket
  878 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % cron
  879 be/4 daemon      0.00 B/s    0.00 B/s  0.00 %  0.00 % atd
 1392 be/4 polw        0.00 B/s    0.00 B/s  0.00 %  0.00 % policyd-weight (master)
 1393 be/4 polw        0.00 B/s    0.00 B/s  0.00 %  0.00 % policyd-weight (cache)
  377 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % upstart-udev-bridge --daemon
 1402 be/4 postgrey    0.00 B/s    0.00 B/s  0.00 %  0.00 % postgrey.pid --daemonize --inet=10023
29053 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
29054 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
  385 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % udevd --daemon
29058 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
29059 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
29060 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
29061 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
 4844 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
29066 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 1427 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % courierlogger -pid=/var/run/courier/authdaemon/pid -start /usr/lib/courier/courier-authlib/authdaemond
 1428 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % authdaemond
27650 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
26538 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
 1451 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % courierlogger -pid=/var/run/courier/imapd.pid -start -name=imapd /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20 -nodnslookup -noidentlookup 143 /usr/lib/courier/courier/imaplogin /usr/bin/imapd Maildir
 1452 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % couriertcpd -address=0 -maxprocs=40 -maxperip=20 -nodnslookup -noidentlookup 143 /usr/lib/courier/courier/imaplogin /usr/bin/imapd Maildir
 2477 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % ispcp_daemon -p /var/run/ispcp_daemon.pid
 5552 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
18866 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
 1462 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % authdaemond
 1463 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % authdaemond
 1464 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % authdaemond
  958 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [flush-9:2]
 1476 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % courierlogger -pid=/var/run/courier/pop3d.pid -start -name=pop3d /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir
 1477 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir
29140 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % python /usr/sbin/iotop -b -n1
  485 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % udevd --daemon
  489 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % udevd --daemon
24571 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start

So while I never dealed with performance troubles and I might just not see it... I think all of this looks pretty okay. I would be eternally grateful for any input what I can try or pointing out something I may have overlooked.

Thanks in advance.

share|improve this question
    
It's very interesting to see your thought process in trouble shooting. You've managed to over look on thing. The Network!! You're server might be fine, however you've got switches and routers between you and the clients, so start to check network services. MTG, Tracert, netstat -s, ping, are all your friends. –  The Unix Janitor Jul 18 '12 at 9:36
    
I forgot to mention that we switched servers and datacenters. We had a vServer previously and we switched from this hoster (because of the lags) to a different one and a Dedicated server and still experienced those lags. So I pretty much ruled that out. But you are right, I should have mentioned this. Sorry. –  genuineparts Jul 18 '12 at 9:58
    
the devil is in the details. If something changes..usually that change has knock on effects. –  The Unix Janitor Jul 18 '12 at 11:54
add comment

2 Answers

up vote 0 down vote accepted

I suggest checking external connections of the system in question especially the DNS. Unexplainable delays are known to be often caused by late- or nonresponding nameservers. tcpdump with a filter on dns queries as well as checking the /etc/hosts and /etc/resolv.conf files might help here.

share|improve this answer
    
That was a good hint. The NS was not slow but the server queried it for the subdomain the AJAX runs on alot. Putting it in the hosts file has reduced the queries. We'll see if this helps. Thanks! –  genuineparts Jul 18 '12 at 9:45
    
Seems that was really the cause for all my troubles. Thank you! –  genuineparts Jul 19 '12 at 11:07
add comment

As the load figures are relatively low, I suspect I/O wait or locking issues are the culprit.

One definitive way to analyse this, is to strace all Apache/PHP processes with the -ttt flag:

strace -f -o /tmp/strace.log -Tttt -p <PID_of_main_Apache_process>

This way, you will have a timestamp per syscall. However, this may generate lots of data, especially if the problem only appears randomly.

A possible shortcut is to check the slow-query log of your mysql server to find troublemakers. If you find slow queries, use the 'EXPLAIN' feature to see whether indices are missing. If this doesn't yield anything, add timing measurements to your PHP code and log them to a separate file.

share|improve this answer
    
I forgot to mention that I have 0 slow queries logged. But I will try the strace. –  genuineparts Jul 18 '12 at 9:25
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.