Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a cPanel server, running Cloud Linux, it's a dedicated server that originally ran CentOS 6.4 and SoftRaid 1. I converted it to CloudLinux after a fresh re-install. I didn't have a need to restart it, until now. Everything was running smoothly, but when I rebooted the server it shut down, but it didn't come back up. The provider put it into Rescue Mode, since I can't use KVM nor IPMI. I ran a hardware test, and everything is alright. I tried re-installing GRUB, but it doesn't change anything. The server still won't boot. I'm really out of ideas at this point. The provider is an unmanaged one, I'm keeping the reinstall as the last resort.

I used this tutorial to re-install GRUB: http://www.adminlinux.org/2009/10/how-to-reinstall-grub-while-in-rescue.html

EDIT:

The provider told me this shows up on the screen while booting up:

*An error occurred during the file system check*
*Dropping you to a shell; the system will reboot*

I did some research and I need to run fsck.

When I try to run fsck /dev/sda1 I keep getting this:

root@rescue [/]# fsck /dev/sda1
fsck from util-linux-ng 2.17.2
fsck: fsck.linux_raid_member: not found
fsck: Error 2 while executing fsck.linux_raid_member for /dev/sda1

That's when the drive is mounted, then I tried unmounting it,

root@rescue [/]# fsck /dev/sda1
fsck from util-linux-ng 2.17.2
/dev/sda1 is in use.
Aborted

I am doing all this in rescue mode.

I have 2 drives 1TB each in Raid 1 here is fstab:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/md1        /       ext3    errors=remount-ro,usrjquota=quota.user,jqfmt=vfsv0      0       1
/dev/md3        /home   ext3    usrjquota=quota.user,jqfmt=vfsv0        1       2
/dev/sda2       swap    swap    defaults        0       0
/dev/sdb2       swap    swap    defaults        0       0
proc            /proc   proc    defaults                0       0
sysfs           /sys    sysfs   defaults                0       0
tmpfs       /dev/shm tmpfs  noexec,nosuid   0   0
devpts          /dev/pts        devpts  defaults        0       0
/tmp_fs /tmp ext3 noexec,nosuid,loop,rw 1 1

UPDATE:

As Bittylicious suggested I executed the commands and here are the results:

root@rescue:~# mdadm --assemble /dev/md1 /dev/sda1 /dev/sdb1
mdadm: /dev/sda1 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping

Then I assumed the raid was already assembled so I did:

root@rescue:~# fsck /dev/md1
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/: recovering journal
Setting free inodes count to 9410644 (was 9410650)
Setting free blocks count to 35082532 (was 35082538)
/: clean, 190380/9601024 files, 3317196/38399728 blocks

UPDATE #2:

Here is the mount result:

root@rescue:~# mount
none on / type tmpfs (rw,relatime)
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
198.27.85.63:/home/pub/rescue.v7 on /nfs type nfs (ro,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,nolock,proto=udp,port=2049,timeo=11,retrans=3,sec=sys,local_lock=all,addr=198.27.85.63)
198.27.85.63:/home/pub/pro-power on /power type nfs (ro,nodev,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,nolock,proto=udp,port=2049,timeo=11,retrans=3,sec=sys,local_lock=all,addr=198.27.85.63)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=807512k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1615020k)
rpc_pipefs on /nfs/var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

Here is the fsck /dev/md3 result:

root@rescue:~# fsck /dev/md3
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/home: clean, 370962/50429952 files, 5187714/201693680 blocks

mkswap on sda2:

root@rescue:~# mkswap /dev/sda2
Setting up swapspace version 1, size = 16382972 KiB
no label, UUID=b375ef52-9434-4298-be4a-38db9da029af

mkswap on sdb2:

root@rescue:~# mkswap /dev/sdb2
Setting up swapspace version 1, size = 16382972 KiB
no label, UUID=f9318100-c18d-4987-b4e8-6bc58b0c8299

UPDATE #3:

I'm assuming Bittylicious ment the /boot/grub/grub.conf, in that case:

default=0
timeout=5

        title CloudLinux Server (2.6.32-458.6.2.lve1.2.30.el6.x86_64)
        kernel /boot/vmlinuz-2.6.32-458.6.2.lve1.2.30.el6.x86_64 root=/dev/md1 $
        root (hd0,0)
        initrd /boot/initramfs-2.6.32-458.6.2.lve1.2.30.el6.x86_64.img
        title linux centos6_64
        kernel /boot/vmlinuz-2.6.32-358.18.1.el6.x86_64 root=/dev/md1  ro
        root (hd0,0)
        initrd /boot/initrd-2.6.32-358.18.1.el6.x86_64.img

I've also tried rebuilding initramfs. I followed this tutorial: http://advancelinux.blogspot.com/2013/06/how-to-rebuild-initrd-or-initramfs-in.html

share|improve this question

closed as off-topic by Falcon Momot, Ward, Jenny D, Dave M, mdpc Oct 9 '13 at 16:05

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions must be relevant to professional system administration. Server Fault is a site dedicated to professionals; novice questions are off-topic. Please see the Help Center for more information on topicality. The best advice we can give you is to hire a professional to help you out." – Falcon Momot, Ward, Jenny D, Dave M, mdpc
If this question can be reworded to fit the rules in the help center, please edit the question.

    
Check if all the devices in /etc/fstab exist. Run fsck on them. –  Mark Wagner Oct 8 '13 at 23:58
    
Updated the post. –  DzoniDev Oct 9 '13 at 7:32
add comment

1 Answer

up vote 2 down vote accepted

There are so many reasons why it couldn't boot. We could all list a number of reasons (and I will) but at the end of the day there must be some way you can get your provider to help out. Even if they just read what's on the console, it will provide massive clues as to how to fix it.

My suggestions for now are:

  • You haven't set up an initramfs/initrd but yet you're specifying it
  • You've specified one, but it doesn't contain the relevant drivers to mount your root filesystem
  • You're using a hypervisor that can't load an initrd/ramfs/kernel image in the compressed format it's in

Most unmanaged providers do provide support, even it's payable. I suggest you fork out for it as you'll spend way too long trying to guess this without any kind of kernel access.

Update 1 - now that you've added information fsck, it's quite simple. You are trying to run fsck on a RAID member - this won't work. What you need to do is to assemble the RAID array first (and then possibly run fsck on that).

Why are you running fsck on /dev/sda1 anyway? What you probably need to do is something like:

mdadm --assemble /dev/md1 /dev/sda1 /dev/sdb1

(assuming that sda1 and sdb2 are your RAID members)

If that worked, then you can run fsck on /dev/md1. What you really need to figure out is why your OS didn't assemble the RAID array. My best guess is that it's degraded, and you need to add 'bootdegraded' to your kernel options.

Update 2 - Let's get more info:

  1. When you reinstalled GRUB, do you know what the contents of your grub.cfg or boot.lst file were so we can figure out your boot options?

  2. Have you tried rebuilding your initramfs (update-initramfs -k all -u just to start)

share|improve this answer
    
I have got them to provide me with the error that shows up on the console: An error occurred during the file system check Dropping you to a shell; the system will reboot –  DzoniDev Oct 8 '13 at 22:33
    
OK, that's a good start. I would get them to provide more information still (maybe even just ask them to take a photo)? This could be as simple as your /etc/fstab being wrong though, certain mountpoints not existing etc. Honestly, as said before, just pay them a small amount to get your system booted. It'll be easier than going back and forth. –  Bittylicious Oct 9 '13 at 6:58
    
Believe me at this point I'd be glad to pay them to help me. But they keep shooting me down. Anyways, I have update the post. –  DzoniDev Oct 9 '13 at 7:32
1  
1. Get a new hosting provider - that's a poor show. 2. I have updated my answer with a likely solution. –  Bittylicious Oct 9 '13 at 7:58
    
I see, I did the commands, and I know it is poor, I'm looking for a replacement, but I can't leave this server just hanging. I have updated the post with the results from the commands performed. –  DzoniDev Oct 9 '13 at 8:02
show 9 more comments

Not the answer you're looking for? Browse other questions tagged or ask your own question.