Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need some help trying to identify something that is making one of my servers getting some high load spikes (it goes up to 120 during 5-10m al least one time per hour).

The thing is that this server is basically a mirror of another one, both of them running the same apps, and both of them connecting to the same dbb server (oddly, the one with the least users is the one that is giving this high load spikes).

I was debugging just in one of this spikes when I compared both server processes with ps ax. I found some differences between them, primarily with these processes and it´s time running:

576 ? S< 2:25 [ata/0]
577 ? S< 0:12 [ata/1]
578 ? S< 4:39 [ata/2]
579 ? S< 1:04 [ata/3]
580 ? S< 0:00 [ata_aux]
586 ? S 8:41 [scsi_eh_3]

6502 ? Ss 15:07 hald

Can this be the cause of the problems? Any help will be appreciated. Kind regards, Simon.

share|improve this question

1 Answer 1

up vote 2 down vote accepted

You mention that one of the systems has more traffic than the other, how significant is the difference? If the machine giving problems is mostly idle, it may just be the costs of startup, cache building, etc. It it is constantly seeing traffic then you may have a serious problem with the hardware.

dmesg, top and vmstat are commonly installed. iostat is less common but is usually available as a package. See the man pages for more information about them.

Use dmesg to find out if the kernel is seeing any problems like IO or filesystem errors.

top is very useful, it provides ongoing data about the process table, memory usage, load average and CPU utilization. Is there a significant difference in IO or memory utilization.

vmstat is used to provide statistics for virtual memory, IO, and cpu activity. The first data line is based on the results since since bootup. The subsquent lines are what has happened since the previous line. If called like vmstat 3 it updates the information every 3 seconds. If called like vmstat 3 3 it runs until it provides 3 data lines.

Another tool is iostat. This provides a bit more detailed information about what is happening. The basic use is similar to vmstat. In other words a single bare number at the end indicates the collection interval and 2 numbers indicates how often and how many times to collect the data. For IO issues, iostat -x 5 and/or iostat -p ALL 5 are very useful in tracking down bottlenecks.

If you are using RAID5 are any of the drives in the array having problems? mdadm is the tool if you are using software RAID. Hardware raid adapters have their own tools to tell you whats going on.

I had a case where a vendor supplied replacement drive in an array topped out at 35% of the performance of the original drives causing a serious performance hit for the array it was in. The replacement drive theoretically had the same basic specs but failed to handle queuing as well as the originals.

share|improve this answer
    
Thanks for this great answer. I´m gonna start debugging with all of them ;)! –  Simon Aug 30 '09 at 8:10
    
Good luck. There is lot more information under /proc and /sys. The problem is that a lot of it is quite cryptic. Take care. –  Rik Schneider Aug 31 '09 at 1:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.