2

After having made use of a remote power switch, my server did not come back online. When I went to the datacenter and reboot the computer on the spot I see the server booting (I see the centos progress bar with running almost all the way to the end) and eventually giving the following messages:

md/raid:md2: cannot start dirty degraded array.
md/raid:md2: failed to run raid set.
md: pers->run() failed ...
md/raid:md2: cannot start dirty degraded array.
md/raid:md2: failed to run raid set.
md: pers->run() failed ...



Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init not tainted 2.6.32-279.1.1.el6.i686 #1
Call Trace:
 [<c083bfbc>] ? panic+0x68/0x11c
 [<c045a501>] ? do_exit+0x741/0x750
 [<c045a54c>] ? do_group_exit+0x3c/0xa0
 [<c045a5c1>] ? sys_exit_group+0x11/0x20
 [<c083eba4>] ? syscall_call+0x7/0xb
 [<c083007b>] ? cmos_wake_setup+0x62/0x112

The server runs CentOS and has software raid, and I don't have backups of the raid settings. The only backup I have is of /home and the database dumps. (Glad to at least have those though.)

Since the server is an old Dell PowerEdge 1750 with no CD-ROM drive, I have no way of booting the machine from a boot disk. I also remember in the past that the server also wouldn't boot from a bootable USB disk. So the only way I know how to boot the server is to go to the datacenter, pick up the server and take it to the office. Screw open the server. Attach a cdrom drive to an IDE slot on the motherboard. And then boot it. I am hoping you guys could help me avoid this.

I have looked a bit through the boot options and I found the following boot options. When CentOS is about to boot and interrupt the boot-countdown:

CentOS (2.6.32-279.1.1.el63.i686)
CentOS Linux (2.6.32-71.29.1.el6.i686)
centos (2.6.32-71.el6.i686)

I think the first configuration is the default one, because choosing that gets me to the above mentioned kernel panic. The other ones end with something like "Sleeping forever".

I can press 'e' to edit boot commands, press 'a' to modify kernel arguments and press 'c' for grub command line.

The command line gives a grub> prompt. But I have no idea how to get the system to boot without (trying to) access the dirty partitions.

What I want to do is off course: - boot the machine - check hard drive for errors - mark the drive as clean

Improve this question
3
  • Does your Dell have a DRAC? If it has one you can avoid going to the data center and edit the console via that. Disclaimer: if it has one and if it is connected. I think you would know if you bought it (it is a few hundred euro more expensive if you include that option), but it might just have slipped though if this is a old second hand server.
    – Hennes
    Sep 8, 2012 at 15:15
  • Used DRAC cards for the 1750 can be found quite cheaply on eBay these days.
    – Michael Hampton
    Sep 8, 2012 at 15:19
  • Aye. But he will still have to install it if it is not present. If the physical machine gets pulled out of the data centre then I strongly recomment adding a DRAC. (or configuring the already present one).
    – Hennes
    Sep 8, 2012 at 15:33

1 Answer 1

Reset to default
2

From the grub screen, edit your boot commands and add emergency to the end of the boot command line options. Then boot it up. This isn't guaranteed to work (if md2 is your root filesystem it will fail). If this fails to get you a shell, you will have to go find a CD-ROM drive.

Once you have a shell, you can run mdadm to attempt to recover your RAID array.

Find out what devices are supposed to be part of it:

mdadm -D /dev/md2

You'll see a listing of devices. If some are marked as removed or failed then you'll have to deal with the failed disks first.

After that, reassemble your RAID array:

mdadm --assemble --force /dev/md2 /dev/**** /dev/**** /dev/**** ...

(* listing each of the devices which are supposed to be in the array from the previous output.)

Improve this answer
12
  • adding emergency to the end of the boot command changes nothing. However, when I remove some stuff from the boot command and only leave the UUID's, I do see stuff happening. Seeing some md's recognised, seeing something about 2 out of 3 mirrors active. But still ending with a kernel panic.
    – nl-x
    Sep 11, 2012 at 16:51
  • Quoting myself: "If this fails to get you a shell, you will have to go find a CD-ROM drive."
    – Michael Hampton
    Sep 11, 2012 at 16:53
  • I have found a CD-ROM drive btw. I was able to boot with an old Knoppix boot CD. But when I tried to mount /dev/sda1 , it wouldn't. When I did dmesg | tail , it said something about /dev/sda1 containing unsupported features. (Maybe a newer boot-cd will have a linux this features present?!)
    – nl-x
    Sep 11, 2012 at 16:56
  • You're running CentOS 6. Boot with a CentOS 6 disc.
    – Michael Hampton
    Sep 11, 2012 at 16:57
  • By the way, I have 3 partitions ... 1st is /boot ... 2nd is swap ... 3rd is / (in this order. Or so Knoppix told me with gparted )
    – nl-x
    Sep 11, 2012 at 16:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .