1

I am facing a problem on a server I didn't install, the former guy isn't here anymore to explain me anything.

Some processes crashed due to what seems to be a disk failure, so I searched in the logs and found lines like this in /var/log/messages :

 kernel: [277146.149980] sd 0:1:0:0: [sda] Sense Key : Medium Error [current] 
 kernel: [277146.149986] sd 0:1:0:0: [sda] Add. Sense: Unrecovered read error

Rebooting and using fsck didn't fix the issue.

I have no idea if the disks are on a hardware or software RAID. I know that LVM isn't install on the machine. Backup is not an issue here, this server doesn't contain info on its disks, important data are mounted via NFS. So I can if necessary reinstall the machine without worrying about the current content of the disks.

The machine is a Dell PowerEdge R510 running a Debian Squeeze 64 bits. The manufacturer (Dell) diagnosis tools don't run on a Debian Squeeze, I tried to install and run them anyway, but they wouldn't work, so I have to diagnose it manually. Here is some info I can provide you.

Also, I have physical access to the machine if needed, and I saw that only three disks were present : three SCSI Seagate disks of 600GB each.

 # lshw -class disk
  *-disk:0                
       description: SCSI Disk
       product: Virtual Disk
       vendor: Dell
       physical id: 1.0.0
       bus info: scsi@0:1.0.0
       logical name: /dev/sda
       version: 1028
       size: 1675GiB (1798GB)
       capacity: 1675GiB (1798GB)
       capabilities: 15000rpm partitioned partitioned:dos
       configuration: ansiversion=6 signature=cf241336

So apparently we are talking about a virtual disk of 1800 GB, containing the three 600GB physical disks. Apparently, configuring virtual disks can be done using the BIOS. So if I have to reinstall the whole machine with new disks, I must go in the BIOS before booting one iso/liveCD/PXE, am I right ?

# fdisk -l
Disk /dev/sda: 1798.7 GB, 1798651772928 bytes
3 heads, 52 sectors/track, 22519177 cylinders
Units = cylinders of 156 * 512 = 79872 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcf241336

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         421       32812   de  Dell Utility
/dev/sda2   *         434       27320     2097152    c  W95 FAT32 (LBA)
/dev/sda3           27320      277715    19530752   83  Linux
/dev/sda4          277728    22519165  1734832129    5  Extended
/dev/sda5          277728      402918     9764864   82  L

I suppose this is the confirmation that there is only one disk, a virtual one.

# smartctl -i /dev/sda
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Device: Dell     Virtual Disk     Version: 1028
Device type: disk
Local Time is: Mon Jun 15 10:28:59 2015 CEST
Device does not support SMART

It seems legit because it seems that smartctl doesn't work with virtual drives.

# cat /proc/scsi/scsi 
Attached devices:
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: TEAC     Model: DVD-ROM DV-28SW  Rev: R.2B
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi0 Channel: 01 Id: 00 Lun: 00
  Vendor: Dell     Model: Virtual Disk     Rev: 1028
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: SEAGATE  Model: ST3600057SS      Rev: ES64
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: SEAGATE  Model: ST3600057SS      Rev: ES64
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: SEAGATE  Model: ST3600057SS      Rev: ES64
  Type:   Direct-Access                    ANSI  SCSI revision: 05

So this is what I've got so far. I suppose that I have to buy new disks and reinstall the machine. So I want to know : does this machine have some RAID configured ? How do I know ? Because I want to reproduce the exact current configuration. I will be glad to provide you more info if you need some.

[EDIT] Here is the lspci command output.

# lspci
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
03:00.0 InfiniBand: QLogic Corp. InfiniPath QME7342 QDR HCA (rev 02)
06:03.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a)

[EDIT2] To know the number of controllers available :

# megacli -adpCount                 
Controller Count: 0.
Exit Code: 0x00

To print all the infos of the controllers :

# megacli -adpallinfo -aALL
Exit Code: 0x00

So I suppose this means there is no hardware RAID ?

[EDIT3]

# ls -l /dev/disk/by-path
total 0
lrwxrwxrwx 1 root root  9 Jun 12 12:19 pci-0000:00:1f.5-scsi-0:0:0:0 -> ../../sr0
lrwxrwxrwx 1 root root  9 Jun 15 13:40 pci-0000:02:00.0-scsi-0:1:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 10 Jun 12 12:21 pci-0000:02:00.0-scsi-0:1:0:0-part6 -> ../../sda6

# megaclisas-status
-- Controller informations --
-- ID | Model

-- Arrays informations --
-- ID | Type | Size | Status | InProgress

-- Disks informations
-- ID | Model | Status

[EDIT4] The sas2ircu command was helpful (see this page).

I finally manage to find if RAID was used : reboot the server, enter a Configuration Utility menu (Ctrl + C), and see that RAID0 is configured. I used the smartctl command on the /dev/sg* disks, (smartctl wasn't working on /dev/sda*).

smartctl -t long /dev/sg2
smartctl -l selftest /dev/sg2

It finally shows me which one of the disks had failed segments. So problem solved !

  • Please add output of lspci. Then we will know what type of RAID controller do you have (each one has differents CLI commands for management) – Marki555 Jun 15 '15 at 11:39
  • I just added the lspci command output. – dbourcet Jun 15 '15 at 11:43
  • As it is LSI card, try these megacli diagnostic commands: hwraid.le-vert.net/wiki/LSIMegaRAIDSAS – Marki555 Jun 15 '15 at 11:46
  • I just added the megacli command output. – dbourcet Jun 15 '15 at 11:57
  • Did you try also other commands from that page like megaclisas? It's strange, because you have this controller connected, so it should show up the details. Try also ls -l /dev/disk/by-path, it will show you the PCI ID of the controller where the disk comes from (compare with first column of lspci output). – Marki555 Jun 15 '15 at 12:12
0

This message indicates that one or more of the three disks is failing, or that the controller/cables are failing. The whole setup seems to be a RAID0 of three disks - a very risky configuration. I would do something like this: backup the data (if you want to), dismantle the RAID and get JBOD. Test each individual disk. Establish what is the failing part. Replace it, and configure RAID 1. Restore/reinstall the system.

  • I finally manage to find if RAID was used (it was), which RAID was used (RAID 0, you were right) and which disks had failed segments. I will keep the RAID 0 configuration, the purpose of this machine is to perform good I/O, so RAID0 is doing fine. Thank you anyway. – dbourcet Jun 16 '15 at 8:05
0

The lspci output:

02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

indicates the presence of an hw raid card.

The correct tool to manage this kind of card is: sas2ircu

You can find more informations here:

http://hwraid.le-vert.net/wiki/LSIFusionMPTSAS2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.