Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a custom built Ubuntu 11.04 server with a 6 disk software RAID 10 primary drive. On it I'm primarily running a PostgreSQL and a few other utilities that stream data from the web. I often find after a few hours of uptime the server starts to lag with all kinds of processes. For example, it may take 10-15 seconds after log-in to get a shell prompt. It might take 5-10 seconds for top to come up. An ls might take a second or two.

When I look at top there is almost no CPU usage. There's a fair amount of memory used by the PostgreSQL server but not enough to bleed into swap.

I have no idea where to go from here, other than to suspect the RAID10 (I've only ever had software RAID 1's before).

Edit: Output from top:

top - 11:56:03 up  1:46,  3 users,  load average: 0.89, 0.73, 0.72
Tasks: 119 total,   1 running, 118 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 93.5%id,  6.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16325596k total,  3478248k used, 12847348k free,    20880k buffers
Swap: 19534176k total,        0k used, 19534176k free,  3041992k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                   
 1747 woodsp    20   0  109m  10m 4888 S    1  0.1   0:42.70 python                                                    
  357 root      20   0     0    0    0 S    0  0.0   0:00.40 jbd2/sda3-8                                               
    1 root      20   0 24324 2284 1344 S    0  0.0   0:00.84 init                                                      
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd                                                  
    3 root      20   0     0    0    0 S    0  0.0   0:00.24 ksoftirqd/0                                               
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0                                               
    7 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/0                                                
    8 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1                                               
   10 root      20   0     0    0    0 S    0  0.0   0:00.02 ksoftirqd/1                                               
   12 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/1                                                
   13 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/2                                               
   14 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/2:0                                               
   15 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/2                                               
   16 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/2                                                
   17 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/3                                               
   18 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/3:0                                               
   19 root      20   0     0    0    0 S    0  0.0   0:00.02 ksoftirqd/3                                               
   20 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/3                                                
   21 root       0 -20     0    0    0 S    0  0.0   0:00.00 cpuset                                                    
   22 root       0 -20     0    0    0 S    0  0.0   0:00.00 khelper                                                   
   23 root      20   0     0    0    0 S    0  0.0   0:00.00 kdevtmpfs                                                 
   24 root       0 -20     0    0    0 S    0  0.0   0:00.00 netns                                                     
   26 root      20   0     0    0    0 S    0  0.0   0:00.00 sync_supers                                               

df -h

rpsharp@ncp-skookum:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       1.8T  549G  1.2T  32% /
udev            7.8G  4.0K  7.8G   1% /dev
tmpfs           3.2G  492K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.8G     0  7.8G   0% /run/shm
/dev/sda2       952M  128K  952M   1% /boot/efi
/dev/md0        5.5T  562G  4.7T  11% /usr/local

free -m

psharp@ncp-skookum:~$ free -m
             total       used       free     shared    buffers     cached
Mem:         15942       3409      12533          0         20       2983
-/+ buffers/cache:        405      15537
Swap:        19076          0      19076

tail -50 /var/log/syslog

Jul  3 06:31:32 ncp-skookum rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="1070" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul  3 06:39:01 ncp-skookum CRON[14211]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul  3 06:40:01 ncp-skookum CRON[14223]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul  3 07:00:01 ncp-skookum CRON[14328]: (woodsp) CMD (/home/woodsp/bin/mail_tweetupdate  # email an update)
Jul  3 07:00:01 ncp-skookum CRON[14327]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul  3 07:00:28 ncp-skookum sendmail[14356]: q63E0SoZ014356: from=woodsp, size=2328, class=0, nrcpts=2, msgid=<201207031400.q63E0SoZ014356@ncp-skookum.Stanford.EDU>, relay=woodsp@localhost
Jul  3 07:00:29 ncp-skookum sm-mta[14357]: q63E0Si6014357: from=<woodsp@ncp-skookum.Stanford.EDU>, size=2569, class=0, nrcpts=2, msgid=<201207031400.q63E0SoZ014356@ncp-skookum.Stanford.EDU>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jul  3 07:00:29 ncp-skookum sendmail[14356]: q63E0SoZ014356: to=Spencer Wood <woodsp@stanford.edu>,Martin Lacayo <mlacayo@stanford.edu>, ctladdr=woodsp (1004/1005), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=62328, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (q63E0Si6014357 Message accepted for delivery)
Jul  3 07:00:29 ncp-skookum sm-mta[14359]: STARTTLS=client, relay=mx3.stanford.edu., version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256
Jul  3 07:00:29 ncp-skookum sm-mta[14359]: q63E0Si6014357: to=<mlacayo@stanford.edu>,<woodsp@stanford.edu>, ctladdr=<woodsp@ncp-skookum.Stanford.EDU> (1004/1005), delay=00:00:01, xdelay=00:00:00, mailer=esmtp, pri=152569, relay=mx3.stanford.edu. [171.67.219.73], dsn=2.0.0, stat=Sent (Ok: queued as 8F3505802AC)
Jul  3 07:09:08 ncp-skookum CRON[14396]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul  3 07:17:01 ncp-skookum CRON[14438]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul  3 07:20:01 ncp-skookum CRON[14453]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul  3 07:39:01 ncp-skookum CRON[14551]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul  3 07:40:01 ncp-skookum CRON[14562]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul  3 08:00:01 ncp-skookum CRON[14668]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul  3 08:09:01 ncp-skookum CRON[14724]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul  3 08:17:01 ncp-skookum CRON[14766]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul  3 08:20:01 ncp-skookum CRON[14781]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul  3 08:39:01 ncp-skookum CRON[14881]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul  3 08:40:01 ncp-skookum CRON[14892]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)

Output of hdparm -t /dev/sd{a,b,c,d,e,f} This looks suspicious?

/dev/sda:
 Timing buffered disk reads:   2 MB in  4.84 seconds = 423.39 kB/sec

/dev/sdb:
 Timing buffered disk reads: 420 MB in  3.01 seconds = 139.74 MB/sec

/dev/sdc:
 Timing buffered disk reads: 390 MB in  3.00 seconds = 129.87 MB/sec

/dev/sdd:
 Timing buffered disk reads: 416 MB in  3.00 seconds = 138.51 MB/sec

/dev/sde:
 Timing buffered disk reads: 422 MB in  3.00 seconds = 140.50 MB/sec

/dev/sdf:
 Timing buffered disk reads: 416 MB in  3.01 seconds = 138.26 MB/sec
share|improve this question
2  
Post top/df -h/free -m commands please –  AliGibbs Jun 30 '12 at 17:58
    
I have run into this before, the culprit was indeed my raid controller too. Can you post what controller you are using? –  Lucas Kauffman Jun 30 '12 at 18:10
    
@LucasKauffman I'm using a software raid. –  Rich Jun 30 '12 at 19:20
1  
Check dmesg for disk error messages. –  mgorven Jul 1 '12 at 6:21
1  
I'm surprised nobody's asked for your software raid cfg yet: (/etc/mdadm.conf?) -- the fact that you mention a 6 disk raid 10, your hdparm shows 6 disks, and your machine is clearly using pieces of one of those disks in non-raid makes me suspect something is badly misconfigured there. –  quadruplebucky Jul 7 '12 at 22:24

5 Answers 5

up vote 1 down vote accepted
+100

You're actually using both a physical storage controller that exposes your disks to the Linux kernel (whether or not you're using its built-in RAID capabilities) and software RAID. You can't rule out the possibility that your storage controller is poorly-supported. Use the output of hdparm -t /dev/sd{a,b,c,d,e,f} to diagnose the issue (this command will take a while).

Since you see some inordinate slowness on /dev/sda, I'd suspect disk failure or controller failure. Double-check that your storage controller is well-supported and try to replace /dev/sda as quickly as possible.

share|improve this answer

I've got an idea. As you posted the output of hdparm, it says that the SDA drive is very slow. It could be because:

a) You have your / and (part of) your RAID 10 on the same disk, or...

b) There's a problem with some driver.

If you upgraded the kernel, try using the default that comes with Ubuntu.

As @Oneiroi pointed, you should try iotop and, in background, run programs. You can run ls where the RAID is mounted, alone; and later run both ls on / and on the RAID. If it slows down, then it could be reason a.

Try using grep to search in /var/log/dmesg, syslog and messages for words like hdd, kernel, raid or postgresql.

Plus, I would try making the sda fail and unmount from the RAID. Then try hdparm again. If it works, then the problem is sda disk.

Another possible case is that the problem is PostgreSQL. If possible, you could start the server without PostgreSQL and see if the problem's still there. If there's still the problem, shutdown any other services you may have. You could also try shutting down everything but PostgreSQL. If you can do this you may know if the problem is produced by

a) PostgreSQL

b) Other Services

c) Manipulation of Big amounts of data

d) System itself.

Depending on what you tried before, you may specify what proble (a,b,c or d) you have, and the get better help.

Plus, if @SilverbackNet has the opportunity, he could tell us about his server; so we now what is simmilar between both servers and have a solution.

PD:Sorry for bad English. Edit and correct errors; there must be a lot.

PD2:I hope this is helpful, but it's just a bunch of theories i thought could help :)

share|improve this answer

Two other possibilities:

  1. Way too much logging going on, or log files not getting cleaned up properly. If they have gotten very large, it is taking time to load / save them during regular operations.

  2. Network connection or SSH issue. I have had similar symptoms with Ubuntu 11, where when I SSH into the machine the SSH connection seems to hang or respond very slowly after even a short period of time. Directly hooking a monitor up seems to be as fast as ever, however. With Ubuntu 12 server, the problems disappeared.

share|improve this answer

There's one possible thing completely left out from the details as I type this.

Perhaps something triggers lots and lots of context switches and/or interrupts? That would probably show as lots of system% in top, but anyway, take a look at vmstat 1 and watch the in and cs columns. And paste the result to your question, too.

share|improve this answer

Looks like you've checked the obvious things - it's a bit of a puzzler.

You're not using LVM anywhere are you? (I don't see anything here that looks like an LVM device) snapshots kill performance on LVM.

Check your irqbalance is running and distributing interrupts sensibly across physical cores (not hyperthreaded ones).

Which io scheduler are you using? Assuming you don't have a hardware RAID controller with lots of memory, then deadline's probably the most sensible option, but try CFQ if you've currently got deadline configured.

What's the filesystems and mount options? How have you configured the disks (for ide/data what does hdparm tell you - check the accoustic settings, DMA, readahead and cache)?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.