1

I am facing a problem on a server I didn't install, the former guy isn't here anymore to explain me anything.

Some processes crashed due to what seems to be a disk failure, so I searched in the logs and found lines like this in /var/log/messages :

 kernel: [277146.149980] sd 0:1:0:0: [sda] Sense Key : Medium Error [current] 
 kernel: [277146.149986] sd 0:1:0:0: [sda] Add. Sense: Unrecovered read error

Rebooting and using fsck didn't fix the issue.

I have no idea if the disks are on a hardware or software RAID. I know that LVM isn't install on the machine. Backup is not an issue here, this server doesn't contain info on its disks, important data are mounted via NFS. So I can if necessary reinstall the machine without worrying about the current content of the disks.

The machine is a Dell PowerEdge R510 running a Debian Squeeze 64 bits. The manufacturer (Dell) diagnosis tools don't run on a Debian Squeeze, I tried to install and run them anyway, but they wouldn't work, so I have to diagnose it manually. Here is some info I can provide you.

Also, I have physical access to the machine if needed, and I saw that only three disks were present : three SCSI Seagate disks of 600GB each.

 # lshw -class disk
  *-disk:0                
       description: SCSI Disk
       product: Virtual Disk
       vendor: Dell
       physical id: 1.0.0
       bus info: scsi@0:1.0.0
       logical name: /dev/sda
       version: 1028
       size: 1675GiB (1798GB)
       capacity: 1675GiB (1798GB)
       capabilities: 15000rpm partitioned partitioned:dos
       configuration: ansiversion=6 signature=cf241336

So apparently we are talking about a virtual disk of 1800 GB, containing the three 600GB physical disks. Apparently, configuring virtual disks can be done using the BIOS. So if I have to reinstall the whole machine with new disks, I must go in the BIOS before booting one iso/liveCD/PXE, am I right ?

# fdisk -l
Disk /dev/sda: 1798.7 GB, 1798651772928 bytes
3 heads, 52 sectors/track, 22519177 cylinders
Units = cylinders of 156 * 512 = 79872 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcf241336

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         421       32812   de  Dell Utility
/dev/sda2   *         434       27320     2097152    c  W95 FAT32 (LBA)
/dev/sda3           27320      277715    19530752   83  Linux
/dev/sda4          277728    22519165  1734832129    5  Extended
/dev/sda5          277728      402918     9764864   82  L

I suppose this is the confirmation that there is only one disk, a virtual one.

# smartctl -i /dev/sda
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Device: Dell     Virtual Disk     Version: 1028
Device type: disk
Local Time is: Mon Jun 15 10:28:59 2015 CEST
Device does not support SMART

It seems legit because it seems that smartctl doesn't work with virtual drives.

# cat /proc/scsi/scsi 
Attached devices:
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: TEAC     Model: DVD-ROM DV-28SW  Rev: R.2B
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi0 Channel: 01 Id: 00 Lun: 00
  Vendor: Dell     Model: Virtual Disk     Rev: 1028
  Type:   Direct-Access                    ANSI  SCSI revision: 06
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: SEAGATE  Model: ST3600057SS      Rev: ES64
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: SEAGATE  Model: ST3600057SS      Rev: ES64
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: SEAGATE  Model: ST3600057SS      Rev: ES64
  Type:   Direct-Access                    ANSI  SCSI revision: 05

So this is what I've got so far. I suppose that I have to buy new disks and reinstall the machine. So I want to know : does this machine have some RAID configured ? How do I know ? Because I want to reproduce the exact current configuration. I will be glad to provide you more info if you need some.

[EDIT] Here is the lspci command output.

# lspci
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
03:00.0 InfiniBand: QLogic Corp. InfiniPath QME7342 QDR HCA (rev 02)
06:03.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a)

[EDIT2] To know the number of controllers available :

# megacli -adpCount                 
Controller Count: 0.
Exit Code: 0x00

To print all the infos of the controllers :

# megacli -adpallinfo -aALL
Exit Code: 0x00

So I suppose this means there is no hardware RAID ?

[EDIT3]

# ls -l /dev/disk/by-path
total 0
lrwxrwxrwx 1 root root  9 Jun 12 12:19 pci-0000:00:1f.5-scsi-0:0:0:0 -> ../../sr0
lrwxrwxrwx 1 root root  9 Jun 15 13:40 pci-0000:02:00.0-scsi-0:1:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 10 Jun 12 12:21 pci-0000:02:00.0-scsi-0:1:0:0-part6 -> ../../sda6

# megaclisas-status
-- Controller informations --
-- ID | Model

-- Arrays informations --
-- ID | Type | Size | Status | InProgress

-- Disks informations
-- ID | Model | Status

[EDIT4] The sas2ircu command was helpful (see this page).

I finally manage to find if RAID was used : reboot the server, enter a Configuration Utility menu (Ctrl + C), and see that RAID0 is configured. I used the smartctl command on the /dev/sg* disks, (smartctl wasn't working on /dev/sda*).

smartctl -t long /dev/sg2
smartctl -l selftest /dev/sg2

It finally shows me which one of the disks had failed segments. So problem solved !

8
  • Please add output of lspci. Then we will know what type of RAID controller do you have (each one has differents CLI commands for management)
    – Marki555
    Jun 15, 2015 at 11:39
  • I just added the lspci command output.
    – dbourcet
    Jun 15, 2015 at 11:43
  • As it is LSI card, try these megacli diagnostic commands: hwraid.le-vert.net/wiki/LSIMegaRAIDSAS
    – Marki555
    Jun 15, 2015 at 11:46
  • I just added the megacli command output.
    – dbourcet
    Jun 15, 2015 at 11:57
  • Did you try also other commands from that page like megaclisas? It's strange, because you have this controller connected, so it should show up the details. Try also ls -l /dev/disk/by-path, it will show you the PCI ID of the controller where the disk comes from (compare with first column of lspci output).
    – Marki555
    Jun 15, 2015 at 12:12

2 Answers 2

0

This message indicates that one or more of the three disks is failing, or that the controller/cables are failing. The whole setup seems to be a RAID0 of three disks - a very risky configuration. I would do something like this: backup the data (if you want to), dismantle the RAID and get JBOD. Test each individual disk. Establish what is the failing part. Replace it, and configure RAID 1. Restore/reinstall the system.

1
  • I finally manage to find if RAID was used (it was), which RAID was used (RAID 0, you were right) and which disks had failed segments. I will keep the RAID 0 configuration, the purpose of this machine is to perform good I/O, so RAID0 is doing fine. Thank you anyway.
    – dbourcet
    Jun 16, 2015 at 8:05
0

The lspci output:

02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

indicates the presence of an hw raid card.

The correct tool to manage this kind of card is: sas2ircu

You can find more informations here:

http://hwraid.le-vert.net/wiki/LSIFusionMPTSAS2

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .