Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

i'm having big trouble accessing my production servers (runnung xen); i recently bought brand new Western Digital (WD5000AAKX 500Gb) drives to replace the old drives; re-installed the system as it was.

Since this morning the server behaves very erratically. I 'suspect' a disk problem; but I replace 3 drives in 3 separate nodes (in a SuperMicro server) and each of them has problems at the same time.

Keyboard handling is very erratic too; on KVM access to server sometimes my keystrokes are being buffered and then released at once; especially for the command line 'tab' completion command. I suspect a disk; but how can I be sure ?

All the servers that are running those drives failed 'at the same time' ?!

Smartctl doesn't show any error :

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   143   143   021    Pre-fail  Always       -       3816
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       11
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       186
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       9
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   115   112   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

Some dd's

[root@node1 ~]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.145882 seconds, 719 MB/s

[root@node1 ~]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.103641 seconds, 1.0 GB/s

[root@node1 ~]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.171583 seconds, 611 MB/s

[root@node1 ~]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.127497 seconds, 822 MB/s

[root@node1 ~]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.093312 seconds, 1.1 GB/s

[root@node1 ~]# dd if=/dev/zero of=prout bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 327.451 seconds, 3.2 MB/s

iostat:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.11    0.09    0.16    6.97    0.01   92.66

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00    2.06    0.00   97.94

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.06    0.06    0.00    0.00   99.88

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00    0.25    0.00   99.75

top

top - 13:02:44 up 8 min, 1 user, load average: 0.53, 0.65, 0.36 Tasks: 199 total, 2 running, 197 sleeping, 0 stopped, 0 zombie

update: even more erratic results

[root@node1 log]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 22.5026 seconds, 4.7 MB/s

[root@node1 log]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 29.5306 seconds, 3.6 MB/s

[root@node1 log]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.156836 seconds, 669 MB/s

[root@node1 log]# dd if=/dev/zero of=prout bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.199494 seconds, 526 MB/s

[root@node1 log]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   21764 MB in  1.98 seconds = 10981.45 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
 Timing buffered disk reads:   92 MB in  4.22 seconds =  21.78 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
share|improve this question
    
While this doesn't sound like a CPU/RAM failure, you should run Memtest for a few hours, though that would mean taking the server completely down for the time - if it's not the problem, it's best to rule it out as soon as possible. –  Sašo May 9 '12 at 11:14
    
ok will do that; not a problem to take offline. I edited the question to include latest results of dd; as you see sometimes it's under 5 MB/sec and sometimes over 500 MB/s. What do you think ? –  Disco May 9 '12 at 11:16
1  
If I had to place a bet, I'd say a faulty/incompatible controller or problematic harddrives. –  Sašo May 9 '12 at 11:23
1  
If you take a look at the datasheet for these drives : WD Caviar Blue Hard Drives are not recommended for and are not warranted for use in RAID environments utilizing Enterprise HBAs and/or expanders and in multi-bay chassis, as they are not designed for, nor tested in, these specific types of RAID applications. For all Business Critical RAID applications, please consider WD’s Enterprise Hard Drives that are specifically designed with RAID-specific, time-limited error recovery (TLER), are tested extensively in 24x7 RAID applications. –  drcelus May 9 '12 at 12:59
1  
What is your RAID hardware and configuration? I'm pretty sure this is not RAM failure otherwise you would be seeing all sorts of problems, not just occasional poor IO performance. –  grifferz May 11 '12 at 19:44
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.