1

I have a dedicated server (Hetzner EX4). One day, I rebooted the server and it wouldn't start. customer service told me that one of the hard drive was faulty, and activated the rescue system (Linux). My server has 2 x 3TB HDD, on a Raid1 (almost sure about this!)

So, I assume at least one of the drives might be usable, but I have no clue about how to get the data out of my server. I've some research, and tried to use partimage (and partimage server), but since I have no understanding of how disks and partitions works in Linux, I don't know how to get the data out of the server.

I don't even know i what I'm seeing are drives, partitions, or what!

Not sure, but I think this may be useful:

root@rescue /dev # cd dev/
-bash: cd: dev/: No such file or directory
root@rescue /dev # dir
1-1            cpu              full   loop4         mouse1              ptyp2  ptype  ram4    sda3  stderr  tty16  tty27  tty38  tty49  tty6   ttyp4  ttyS0    vcs2   vga_arbiter
2-1            cpu_dma_latency  fuse   loop5         net                 ptyp3  ptypf  ram5    sda4  stdin   tty17  tty28  tty39  tty5   tty60  ttyp5  ttyS1    vcs3   vhost-net
2-1.4          disk             hpet   loop6         network_latency     ptyp4  ram0   ram6    sda5  stdout  tty18  tty29  tty4   tty50  tty61  ttyp6  ttyS2    vcs4   watchdog
2-1.6          event0           input  loop7         network_throughput  ptyp5  ram1   ram7    sdb   tty     tty19  tty3   tty40  tty51  tty62  ttyp7  ttyS3    vcs5   watchdog0
autofs         event1           kmem   loop-control  null                ptyp6  ram10  ram8    sdb1  tty0    tty2   tty30  tty41  tty52  tty63  ttyp8  urandom  vcs6   xconsole
block          event2           kmsg   MAKEDEV       port                ptyp7  ram11  ram9    sdb2  tty1    tty20  tty31  tty42  tty53  tty7   ttyp9  usbmon0  vcsa   zero
bsg            event3           kvm    mapper        ppp                 ptyp8  ram12  random  sdb3  tty10   tty21  tty32  tty43  tty54  tty8   ttypa  usbmon1  vcsa1
btrfs-control  event4           log    md            psaux               ptyp9  ram13  rtc     sdb4  tty11   tty22  tty33  tty44  tty55  tty9   ttypb  usbmon2  vcsa2
bus            event5           loop0  mem           ptmx                ptypa  ram14  rtc0    sdb5  tty12   tty23  tty34  tty45  tty56  ttyp0  ttypc  usbmon3  vcsa3
char           event6           loop1  mice          pts                 ptypb  ram15  sda     sg0   tty13   tty24  tty35  tty46  tty57  ttyp1  ttypd  usbmon4  vcsa4
console        fb0              loop2  microcode     ptyp0               ptypc  ram2   sda1    sg1   tty14   tty25  tty36  tty47  tty58  ttyp2  ttype  vcs      vcsa5
core           fd               loop3  mouse0        ptyp1               ptypd  ram3   sda2    shm   tty15   tty26  tty37  tty48  tty59  ttyp3  ttypf  vcs1     vcsa6
root@rescue /dev # fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x8ab49420

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.
root@rescue /dev #

Could anyone please give me some advice, or point me in the right direction to solve my problem? Maybe I'm doing it all wrong, and I should be taking another approach, or maybe this is not possible at all :/

UPDATE 1 First, let me thank you for all your suggestions. I've tried a few things, but nor sure what the results mean..

First, you've already seen the result of fdisk -l (I'm not sure the fact that /dev/sda has Disk Identifier: 0x00000000 is a clue..

I've tried to mount /dev/sda1 and it worked. However, if I go into that directory, I can only see the EFI folder. Is this normal?

Also, if I try to mount /dev/sdb1, I get "mount: you must specify the filesystem type".

If I run cat /proc/mdstat I get this:

Personalities : [raid1]
unused devices: <none>

UPDATE 2 Following Cristian Ciupitu's advise, I ran smartctl on both drives, and here are the results:

sdb:

root@rescue / # smartctl -l error /dev/sdb
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.10.36] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 242 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 242 occurred at disk power-on lifetime: 20101 hours (837 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 8c fe a2 0b  Error: UNC at LBA = 0x0ba2fe8c = 195231372

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 8c fe a2 4b 00   7d+19:32:38.593  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00   7d+19:32:38.559  SET FEATURES [Reserved for Serial ATA]
  27 00 00 00 00 00 e0 00   7d+19:32:38.559  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00   7d+19:32:38.559  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00   7d+19:32:38.559  SET FEATURES [Set transfer mode]

Error 241 occurred at disk power-on lifetime: 20101 hours (837 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 8b fe a2 0b  Error: UNC at LBA = 0x0ba2fe8b = 195231371

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 8c fe a2 4b 00   7d+19:32:35.600  READ FPDMA QUEUED
  60 00 01 8b fe a2 4b 00   7d+19:32:35.600  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00   7d+19:32:35.567  SET FEATURES [Reserved for Serial ATA]
  27 00 00 00 00 00 e0 00   7d+19:32:35.567  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00   7d+19:32:35.566  IDENTIFY DEVICE

Error 240 occurred at disk power-on lifetime: 20101 hours (837 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 8d fe a2 0b  Error: UNC at LBA = 0x0ba2fe8d = 195231373

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 8b fe a2 4b 00   7d+19:32:32.607  READ FPDMA QUEUED
  60 00 01 8c fe a2 4b 00   7d+19:32:32.606  READ FPDMA QUEUED
  60 00 01 8d fe a2 4b 00   7d+19:32:32.606  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00   7d+19:32:32.574  SET FEATURES [Reserved for Serial ATA]
  27 00 00 00 00 00 e0 00   7d+19:32:32.573  READ NATIVE MAX ADDRESS EXT

Error 239 occurred at disk power-on lifetime: 20101 hours (837 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 8a fe a2 0b  Error: UNC at LBA = 0x0ba2fe8a = 195231370

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 8d fe a2 4b 00   7d+19:32:29.563  READ FPDMA QUEUED
  60 00 01 8c fe a2 4b 00   7d+19:32:29.563  READ FPDMA QUEUED
  60 00 01 8b fe a2 4b 00   7d+19:32:29.563  READ FPDMA QUEUED
  60 00 01 8a fe a2 4b 00   7d+19:32:29.563  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00   7d+19:32:29.531  SET FEATURES [Reserved for Serial ATA]

Error 238 occurred at disk power-on lifetime: 20101 hours (837 days + 13 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 8e fe a2 0b  Error: UNC at LBA = 0x0ba2fe8e = 195231374

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 8a fe a2 4b 00   7d+19:32:26.521  READ FPDMA QUEUED
  60 00 01 8b fe a2 4b 00   7d+19:32:26.521  READ FPDMA QUEUED
  60 00 01 8c fe a2 4b 00   7d+19:32:26.521  READ FPDMA QUEUED
  60 00 01 8d fe a2 4b 00   7d+19:32:26.521  READ FPDMA QUEUED
  60 00 01 8e fe a2 4b 00   7d+19:32:26.520  READ FPDMA QUEUED

sda:

root@rescue / # smartctl -t short /dev/sda
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.10.36] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Short INQUIRY response, skip product id
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

UPDATE 3 I did a lsblk to find which partitions have the data:

root@rescue / # lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   2.7T  0 disk
├─sda1   8:1    0   200M  0 part
├─sda2   8:2    0     1M  0 part
├─sda3   8:3    0   127M  0 part
├─sda4   8:4    0   2.7T  0 part /mnt
└─sda5   8:5    0 455.5K  0 part
sdb      8:16   0   2.7T  0 disk
├─sdb1   8:17   0     1M  0 part
├─sdb2   8:18   0   127M  0 part
├─sdb3   8:19   0   200M  0 part
├─sdb4   8:20   0   2.7T  0 part
└─sdb5   8:21   0 455.5K  0 part
loop0    7:0    0   1.5G  1 loop

Then I mounted sda4. I can see the filesystem (C unit in Windows), but when i go inside a directory (let's say "Program Files"), and try to list all files, I get an IO error:

dir: reading directory .: Input/output error

I did try to send anyway all the filesystem through ftp using ncftpput, and most of the files throwed an IO exception.

If I try to mount sdb4, I got this error:

root@rescue / # mount /dev/sdb4 /mnt
ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Failed to read vcn 0x28: Input/output error
Failed to mount '/dev/sdb4': Input/output error
NTFS is either inconsistent, or there is a hardware fault, or it's a
SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows
then reboot into Windows twice. The usage of the /f parameter is very
important! If the device is a SoftRAID/FakeRAID then first activate
it and mount a different device under the /dev/mapper/ directory, (e.g.
/dev/mapper/nvidia_eahaabcc1). Please see the 'dmraid' documentation
for more details.

UPDATE 4

I've tried ntfsfix with no luck:

root@rescue / # ntfsfix /dev/sdb4
Mounting volume... ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Failed to read vcn 0x28: Input/output error
FAILED
Attempting to correct errors...
Processing $MFT and $MFTMirr...
Reading $MFT... OK
Reading $MFTMirr... OK
Comparing $MFTMirr to $MFT... OK
Processing of $MFT and $MFTMirr completed successfully.
Setting required flags on partition... OK
Going to empty the journal ($LogFile)... OK
ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Failed to read vcn 0x28: Input/output error
Remount failed: Input/output error
Improve this question
3
  • 2
    RAID is not a backup. You do have backups, right? Right?
    – Grant
    Apr 28, 2014 at 16:16
  • Replace the disk immediately. The strain of pulling all your data off the drives on a degraded array can cause another drive to fail and the loss of your data.
    – Nathan C
    Apr 28, 2014 at 16:34
  • Run smartctl -a /dev/sda and smartctl -a /dev/sdb to get a better idea about the health of the drives.
    – Cristian Ciupitu
    Apr 28, 2014 at 23:54

4 Answers 4

Reset to default
1

First, check your backups. If all goes well you will not need them, but it help a lot (emotially) when you know your data is safe and that you can risk things without fully understanding them.

Next find out which RAID you used. It can be hardware RAID, but it can also be software RAID like mdadm. Unless you remember paying for HW RAID mdadmn is likely. Confirm this and read the mdadm man pages.

Next figure out which disk is broken and which is still ok.

Mount the disk which is OK to get a degraded RAID1. To to this check cat /proc/mdstat. It you are lucky you will get output similar to this:

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
md0 : active raid1 sda1[0] sdb1[2](F)
      24418688 blocks [2/1] [U_]

In this example mdadm is loaded and recognised that the second disk has failed. If you do not get this output check if your kernel has mdadm support or that the right module is loaded. Optionally create a new md device. Going fro memory the command for that is mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb2. (Check this! And realise that this assumes the second drive failed. If it was the first write then create one with sda as missing!).

Once you get a working /dev/mdX device copy all your data to a spare location. You might not need it, but you want to be sure that your backups are fully up to date. Only then ask your provider to swap the broken disk for a new one and add the new disk to the RAID array.

caveeats:

  1. This assumes you have backups, or can read a manual.
  2. This assumes the disk was partitioned as a single large RAID1. That is likely, but not guaranteed. (One could have partitioned the disk in multiple partitions and RAID'ed those). However the path of the least effort would be one big RAID1 disk, and because it is less work it is mostly used be default.
Improve this answer
5
  • First, let me thank you for all your suggestions. I've tried a few things, but nor sure what the results mean.. First, you've already seen the result of fdisk -l (I'm not sure the fact that /dev/sda has Disk Identifier: 0x00000000 is a clue.. I've tried to mount /dev/sda1 and it worked. However, if I go into that directory, I can only see the EFI folder. Is this normal?
    – Dídac Punyet
    Apr 28, 2014 at 19:52
  • Also, if I try to mount to mount /sdb1, I get "mount: you must specify the filesystem type". If I run cat /proc/mdstat I get this: code root@rescue / # cat /proc/mdstat Personalities : [raid1] unused devices: <none>
    – Dídac Punyet
    Apr 28, 2014 at 19:58
  • Question edited
    – Dídac Punyet
    Apr 28, 2014 at 20:00
  • I've checked this post as answer because even though it didn't solve my problem, pointed me in the right direction. The issue that I have now, I think it should be posted in a different thread.
    – Dídac Punyet
    Apr 30, 2014 at 15:20
  • Could you point us to that thread?
    – Hennes
    Apr 30, 2014 at 16:44
1

You need to replace the disk. Now.

To Linux, your array will appear as a singular drive unless partitioned differently or separated by the controller into different virtual disk(s). You should be able to use the serial console (if provided by the host) to boot into your server's controller and see how it's configured.

But, replace the disk or be sorry. Depending on the age of your drives, the second one may be nearing its failure time and you really want to finish the rebuild before it fails.

Improve this answer
0

You may want to replace the bad hard drive with a good drive, and it should repair itself. It is possible that your RAID implementation is different though, and you may need to consult your vendor's documentation for further guidance. However, if the file system got corrupted by software, and not a bad hard drive, RAID does not protect against this (e.g. the bad writes happen on both hard drives).

Improve this answer
0

You should be able to mount the hard drive that is still online.

Try the command mount /dev/sda1 /mnt

If this command works, it will mount the filesystem under /mnt. Navigate to that folder and you should be able to see files and folders. Just backup the info you need and follow Jonathan's advice.

Improve this answer
1
  • @Cristian Ciupitu Tried that, results on UPDATE 3
    – Dídac Punyet
    Apr 29, 2014 at 7:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .