1

I'm having a serious issue with dell server PE2950. The server has really slow IO rates, so slow that I'm not able to use it anymore

I tried few things to solve this:

  1. changing disks to new disks (configured them as raid1)
  2. changing perc card + perc cables
  3. reinstalling the OS of course, had to cause of changing of disks, centos 5.5 x64bit
  4. firmware update to everything
  5. virtual disks policy: No Read Ahead,Write Back, disk cache policy disabled.
  6. openmanage doesn't alert about anything, also i ran dell's diag tests, everything passed, also dell didn't see anything in deset log.
  7. dell offered to reseat everything, including the cpu, we did that as well, still io rates are slow

I have several PE2950 servers, and I never had such a thing with any of those. All have similar or exact hardware as this one, all configured the same, with the same os centos 5.5 x64, same disks, same raid, same policy.

Just for comparison:

the problematic PE2950 server:

[root@bad ~]# time sh -c "dd if=/dev/zero of=/tmp/ddfile bs=8k count=200000 && sync"  

200000+0 records in  
200000+0 records out  
1638400000 bytes (1.6 GB) copied, 27.7946 seconds, 58.9 MB/s  

real    0m33.968s  
user    0m0.531s  
sys     0m26.000s

good PE2950 server (with the exact same hardware):

[root@good ~]# time sh -c "dd if=/dev/zero of=/tmp/ddfile bs=8k count=200000 && sync"  

200000+0 records in  
200000+0 records out  
1638400000 bytes (1.6 GB) copied, 3.19999 seconds, 512 MB/s  

real    0m7.694s  
user    0m0.053s  
sys     0m4.057s 

Hopefully you will have an idea what can cause the problem.

3
  • What are the virtual disk policies on the "bad" and "good" server. Are they the same?
    – joeqwerty
    Mar 21, 2011 at 13:32
  • Are you seeing anything going on with the PERC battery in the OpenMange logs? I have seen when the battery goes bad the learn cycle happens a lot. Patrol will run > quaterly and you will see your disks switch over to write through. Probably a long shot if Dell didn't see anything in the logs about it.
    – Chadddada
    Apr 13, 2011 at 1:54
  • I would definately enable write-back cache policy if your controller battery is in good condition. Also since these are new disks, you probably want to check the firmware levels on them. Dell will have a recommened firmware for them that work nicely with the percs.
    – Bad Dos
    Jun 3, 2012 at 5:07

3 Answers 3

2

If you can I'd try moving the disk controller, cables and disks from the good server to the bad and vice versa, see if the problem stays with the disks/controller or the systemboard. If it stays with the systemboard then put the disks back and swap the CPU/s, again see if the problem moves/stays, then possibly the memory. If it stays with the disks, then try split the disks and controllers. Basically divide and eliminate.

1
  • cables and disks has been changed to a new one. and we had the problem before changing those, so i guess it eliminate the perc, cables and disks. changing the cpu and the motherboard is not such an option at the moment. since cant take another server down..
    – OrenM
    Mar 21, 2011 at 21:00
1

I had similar issues with PE2900 and PERC 6/i, very slow writes (~ 40 MB/s) with Ubuntu 12.04 and I noticed that when I boot from the CD in rescue mode the speeds were good (~ 200 MB/s), so I started blacklisting linux modules and found out the cause - i5000_edac and edac_core.

vi /etc/modprobe.d/blacklist-edac.conf

blacklist i5000_edac
blacklist edac_core
1
  • quite incredibly worked with my Debian jessie on the same hardware/bios! Dec 23, 2015 at 13:21
0
  1. What PERC type is used?
  2. Is there a BBU? Is it alive?
  3. What is the array state when you run the test?
  4. Do you see the same results when booted from a liveCD?
  5. Is there anything loading the disk IO?
  6. Are those SAS or SATA disks?
  7. When updating all firmware, did you also update the HDD firmware?
3
  • 1. perc 6i integreted. as i mentioned before, i have replaced it with a new one 2. yes. stated as "ready" 3. iostat status: running dd command: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 4958.50 0.00 227.00 0.00 20788.00 183.15 1.04 4.61 2.53 57.45 idle: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.50 0.00 2.00 8.00 0.00 0.00 0.00 0.00
    – OrenM
    Mar 21, 2011 at 12:53
  • 4. i didnt try running these from a livecd, i will do that soon, but i guess the problem is not in the OS, since i reinstalled it. 5. no, the server is after a fresh clean OS installation, and it's idle since i can't use it like that. 6. it was sas 300GB 15K disks, i changed them to SAS 600GB 15K disks, it was slow before and after. 7. i've updated everything using dell repository, so yes, it updated that aswell.
    – OrenM
    Mar 21, 2011 at 12:56
  • in #3 I meant the array states in OMSA. If the array is in a "rebuilding" or "reconfiguring" or "initializing" state, the overhead will be huge. As for additional steps, Chopper3 made a good point
    – dyasny
    Mar 21, 2011 at 13:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .