Roll back to clean debian

Question

I have a remote server, which is like a "sandbox", where I'm testing all new software before installing them on production servers. I wanted to clean up a little bit this system, but accidentally deleted all /var contents. How can I recover it, or roll back to clean debian, if I can not reinstall system, because I have only ssh connection and no physical access?

Answer 1

What I would do is as follows:

Get a Linux distribution which can run entirely in RAM. In this case, let's assume you will use PLD Rescue Linux (available here: http://rescuecd.pld-linux.org). Make sure to use right CPU architecture (x86 or x86_64).

Mount the iso somewhere (e.g. /mnt) and copy the kernel and initrd to your /boot:

cp /mnt/boot/isolinux/vmlinuz /boot/vmlinuz-rescue (vmlinuz6 for x86_64)
cp /mnt/rescue.cpi /boot/rescue.cpi (rescue6.cpi for x86_64)

Configure your bootloader to boot the rescue system instead of your Debian. Kernel command line should be:

initrd=/boot/rescue.cpi root=/dev/ram0

Remember to make it default, you won't be able to select it since you don't have physical access or KVM.

Now reboot your system. Wait some time till ping comes back and ssh into your box as root. The password is: pld

Now the fun begins. Format all of your system partitions - everything that was previously mounted on /, /var/, /boot etc. You can of course leave /home alone. For example if your system consists of a single filesystem on /dev/sda1, then run:

mkfs.ext4 /dev/sda1

Mount everything under /mnt. Create appropriate mountpoints if necessary.

Now you will bootstrap a basic Debian system:

debootstrap squeeze /mnt

This will install Debian Squeeze into /mnt. Wait for everything to finish (it will take a while). Mount-bind virtual filesystems:

mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
mount --bind /dev /mnt/dev

And chroot into your new environemnt:

chroot /mnt /bin/bash

Once in chroot install the Linux kernel and appropriate bootloader:

apt-get update
apt-get install linux-image-2.6-i686 (or linux-image-2.6-amd64)
apt-get install grub-pc

You will also need SSH meta-package:

apt-get install ssh

Now edit /etc/fstab, /etc/hostname and /etc/hosts to suit your needs. Remember to create /etc/network/interfaces with appropriate network configuration. Double check it for mistakes. You will need networking since you can only SSH into the box.

Setup a password for root:

passwd

Install your bootloader to the MBR of your harddrive. If you installed GRUB2 as in my example then run:

update-grub
grub-install /dev/sda

Leave chroot by typing "exit" and unmount everything under /mnt. Reboot and pray to the God of Ping. ;-)

WARNING:

Reinstalling OS while having only SSH access is always dangerous since you don't have any means to debug broken system without functional networking. First practice in a virtual machine on your PC and then apply those instructions to your server.

Answer 2

Grab the debootstrap package. Debootstrap can install a starter Debian in a directory somewhere, and you can reconstruct some of /var based on that.

You won't be able to install the package normally, of course. Download it and unpack it manually.

dpkg-deb -x debootstrap_*.deb /
mkdir /chroot
debootstrap squeeze /chroot

Now recreate at least the directory structure and permissions, or just copy /chroot/var wholesale.

Of course you won't recover your data, but you should get a limping system. In particular, you won't be able to use the package management tools unless you repair the package database, and that's a lot of work. I advise reinstalling as soon as you can get physical access.

Answer 3

If your server is hosted with a hosting company, they may provide some way of remotely reinstalling the operating system or booting off a rescue disk? Perhaps it might be worth checking with them if this is possible? (You haven't specified if the server is hosted with a company or if it is simply just a remote box somewhere out of reach?)

The company which I host my dedicated server with provides both a web interface and desktop application with which you can manage certain aspects of the server and hosting account. This includes control over the boot source for the machine, so I can boot off one of a selection of rescue disks etc, as well as wiping and reinstalling with a new operating system (as in, using the same method they use to provision the server with a specific OS when you first order the server).