I am facing a problem on a server I didn't install, the former guy isn't here anymore to explain me anything.
Some processes crashed due to what seems to be a disk failure, so I searched in the logs and found lines like this in /var/log/messages
:
kernel: [277146.149980] sd 0:1:0:0: [sda] Sense Key : Medium Error [current]
kernel: [277146.149986] sd 0:1:0:0: [sda] Add. Sense: Unrecovered read error
Rebooting and using fsck
didn't fix the issue.
I have no idea if the disks are on a hardware or software RAID. I know that LVM isn't install on the machine. Backup is not an issue here, this server doesn't contain info on its disks, important data are mounted via NFS. So I can if necessary reinstall the machine without worrying about the current content of the disks.
The machine is a Dell PowerEdge R510 running a Debian Squeeze 64 bits. The manufacturer (Dell) diagnosis tools don't run on a Debian Squeeze, I tried to install and run them anyway, but they wouldn't work, so I have to diagnose it manually. Here is some info I can provide you.
Also, I have physical access to the machine if needed, and I saw that only three disks were present : three SCSI Seagate disks of 600GB each.
# lshw -class disk
*-disk:0
description: SCSI Disk
product: Virtual Disk
vendor: Dell
physical id: 1.0.0
bus info: scsi@0:1.0.0
logical name: /dev/sda
version: 1028
size: 1675GiB (1798GB)
capacity: 1675GiB (1798GB)
capabilities: 15000rpm partitioned partitioned:dos
configuration: ansiversion=6 signature=cf241336
So apparently we are talking about a virtual disk of 1800 GB, containing the three 600GB physical disks. Apparently, configuring virtual disks can be done using the BIOS. So if I have to reinstall the whole machine with new disks, I must go in the BIOS before booting one iso/liveCD/PXE, am I right ?
# fdisk -l
Disk /dev/sda: 1798.7 GB, 1798651772928 bytes
3 heads, 52 sectors/track, 22519177 cylinders
Units = cylinders of 156 * 512 = 79872 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcf241336
Device Boot Start End Blocks Id System
/dev/sda1 1 421 32812 de Dell Utility
/dev/sda2 * 434 27320 2097152 c W95 FAT32 (LBA)
/dev/sda3 27320 277715 19530752 83 Linux
/dev/sda4 277728 22519165 1734832129 5 Extended
/dev/sda5 277728 402918 9764864 82 L
I suppose this is the confirmation that there is only one disk, a virtual one.
# smartctl -i /dev/sda
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
Device: Dell Virtual Disk Version: 1028
Device type: disk
Local Time is: Mon Jun 15 10:28:59 2015 CEST
Device does not support SMART
It seems legit because it seems that smartctl doesn't work with virtual drives.
# cat /proc/scsi/scsi
Attached devices:
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: TEAC Model: DVD-ROM DV-28SW Rev: R.2B
Type: CD-ROM ANSI SCSI revision: 05
Host: scsi0 Channel: 01 Id: 00 Lun: 00
Vendor: Dell Model: Virtual Disk Rev: 1028
Type: Direct-Access ANSI SCSI revision: 06
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: SEAGATE Model: ST3600057SS Rev: ES64
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: SEAGATE Model: ST3600057SS Rev: ES64
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: SEAGATE Model: ST3600057SS Rev: ES64
Type: Direct-Access ANSI SCSI revision: 05
So this is what I've got so far. I suppose that I have to buy new disks and reinstall the machine. So I want to know : does this machine have some RAID configured ? How do I know ? Because I want to reproduce the exact current configuration. I will be glad to provide you more info if you need some.
[EDIT] Here is the lspci command output.
# lspci
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
03:00.0 InfiniBand: QLogic Corp. InfiniPath QME7342 QDR HCA (rev 02)
06:03.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a)
[EDIT2] To know the number of controllers available :
# megacli -adpCount
Controller Count: 0.
Exit Code: 0x00
To print all the infos of the controllers :
# megacli -adpallinfo -aALL
Exit Code: 0x00
So I suppose this means there is no hardware RAID ?
[EDIT3]
# ls -l /dev/disk/by-path
total 0
lrwxrwxrwx 1 root root 9 Jun 12 12:19 pci-0000:00:1f.5-scsi-0:0:0:0 -> ../../sr0
lrwxrwxrwx 1 root root 9 Jun 15 13:40 pci-0000:02:00.0-scsi-0:1:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jun 12 12:19 pci-0000:02:00.0-scsi-0:1:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 10 Jun 12 12:21 pci-0000:02:00.0-scsi-0:1:0:0-part6 -> ../../sda6
# megaclisas-status
-- Controller informations --
-- ID | Model
-- Arrays informations --
-- ID | Type | Size | Status | InProgress
-- Disks informations
-- ID | Model | Status
[EDIT4]
The sas2ircu
command was helpful (see this page).
I finally manage to find if RAID was used : reboot the server, enter a Configuration Utility menu (Ctrl + C), and see that RAID0 is configured.
I used the smartctl command on the /dev/sg*
disks, (smartctl wasn't working on /dev/sda*
).
smartctl -t long /dev/sg2
smartctl -l selftest /dev/sg2
It finally shows me which one of the disks had failed segments. So problem solved !
lspci
. Then we will know what type of RAID controller do you have (each one has differents CLI commands for management)megaclisas
? It's strange, because you have this controller connected, so it should show up the details. Try alsols -l /dev/disk/by-path
, it will show you the PCI ID of the controller where the disk comes from (compare with first column oflspci
output).