66

A disaster just occurred to me after I ran the command yum remove python and now I can't boot the server up anymore.

How it happened: I tried updating some apps via yum on my CentOS 5 VPS and the command was failing due to some weird python 2.4 error. I noticed that my version of python was old and I tried reinstalling it by first removing it, and so I did yum remove python.

After that it asked me something about removing dependencies and it looked like nothing I could miss so I clicked Y.

So the aftermath of that was that I was unable to run any command what so ever. I even tried cd /var/www but it said something like "command does not exist in /usr/bin". When I used tab to see folder navigation suggestions, the file structure still seemed to be there (at least the /var/www bit which is really important to me). After that I tried restarting the vps (from the admin panel since reboot command did not work) and now it doesn't boot anymore.

Now my question is: how can a command like that possibly destroy my server like this?

Improve this question
13
  • 33
    Incidentally, apt-get remove apt works. I haven't tried dpkg --remove dpkg yet. I bet that's really bad.
    – joshudson
    Apr 16, 2017 at 3:58
  • 19
    You do have a backup, right?
    – vasin1987
    Apr 16, 2017 at 17:55
  • 21
    @joshudson I just spun up a Debian 8.4 live CD in a VM and tried it. Result: # dpkg --remove dpkg spits out dpkg: error processing dpkg (--remove): this is an essential package; it should not be removed. If I add --force-all to dpkg's command line, dpkg spits out a whole bundle of warnings and proceeds to remove itself, along with breaking about two dozen other packages that depend on dpkg. On a real system, I'm pretty sure you'd have some trouble recovering from that, but you probably could (there's little magic to .debs); CentOS may or may not be similar in this regard.
    – user
    Apr 16, 2017 at 20:35
  • 6
    @joshudson It works because there's a copy of apt running in memory when you issue the command... as soon as it's finished (and the program terminates), you won't be able to use apt any longer.
    – SnakeDoc
    Apr 16, 2017 at 21:11
  • 2
    Yum itself is implemented in Python I believe... So removing Python shoots your own foot pretty well!
    – bobflux
    Apr 18, 2017 at 13:29

6 Answers 6

Reset to default
69

I'm sincerely sorry: I can feel the pain to have a server unbootable/unserviceable.

However, I'm lost when reading that:

After that it asked me something about removing dependencies and it looked like nothing I could miss so I clicked [Y]

The list of to-be-removed packages surely was really huge, as python is an essential part of RHEL/CentOS. You should never confirm some warning message you don't really understand.

The best thing you can do, as already suggested, is to boot via a recovery media (ie: livecd), extract the required data files, and reinstall your machine with a newer CentOS release (and as CentOS 6 is quite old, I strongly suggest you to rebase on CentOS 7).

Improve this answer
3
  • 2
    For the future: build servers with the OS on LVM LVs (which ISTR is the RHEL default). Before anything that has even the slightest chance of breaking the system, create snapshot(s). If it does indeed break the system, you can then revert to the snapshot in minutes. Otherwise when all is OK you can remove the snapshot. unix.stackexchange.com/questions/18913. NB snapshots are not backups. You still need backups for situations where the snapshot cannot save you.
    – nigel222
    Apr 18, 2017 at 12:48
  • On Debian/Ubuntu, he could boot recovery image(e.g. finnix), unpack base system with debootstrap, chroot and apt-get install ubuntu-desktop. No idea whether centos/rhel have debootstrap equivalent.
    – Edheldil
    Apr 19, 2017 at 8:56
  • 1
    CentOS 6 is supported until November 2020. There's no rush to upgrade (though going to 7 does have plenty of advantages).
    – psmears
    Apr 19, 2017 at 9:10
107

Frankly, because you did something you didn't fully understand. Python is an essential part of the OS and the things you considered unimportant are very important. Restore from backup.

When you removed Python, yum showed you a long list of packages that would also be removed. This list contains such essentials as yum itself, coreutils, net-tools and others. You confirmed to yum that you know what you are doing and want to proceed anyway. The result of this is a non-working system. This shouldn't be surprising.

For the record, on newer CentOS version this isnt't possible anymore, as certain packages are now marked as protected and can't be removed, only reinstalled or upgraded. And since CentOS 5 is now EOL anyway, this is a good time to upgrade to a newer version.

Improve this answer
10
  • 3
    Haha well that escalated quickly (yum remove python == just wipe whole machine :D ). Do you reckon whether there is any way I can at least restore the files that were in /var/www or is that completely foobard as well?
    – tadoman
    Apr 15, 2017 at 20:02
  • 8
    @tadoman: If you can boot into some kind of rescue system, you will be able to mount the disk and recover data from it (this should be all of your user data, including config files). You have to contact your provider to discuss what is possible in their environment.
    – Sven
    Apr 15, 2017 at 22:30
  • 6
    @RussellBorogove: I don't say either of this. yum works by walking down the dependency graph and it removes packages until all dependencies are satisfied. If package A depends on package B and package B depends on package C, yum will also remove package B and in turn, package A to satisfy all dependencies if you try to remove package C. For central packages (like python), this can result in a large number of removed packages that are seemingly unrelated.
    – Sven
    Apr 16, 2017 at 15:20
  • 4
    @RussellBorogove: Yes, exactly, although coreutils doesn't depend on python directly in CentOS 5, but via one or more intermediate packages - the result is the same though: Removal.
    – Sven
    Apr 16, 2017 at 16:32
  • 4
    The number of intermediate packages to coreutils appears to be two. python -> cracklib -> pam -> coreutils The dependency exists because there are python bindings to cracklib, pam was built with cracklib, and that su is integrated with pam. Of course it took out lots of important things too, like yum to install any more packages...
    – John Mahowald
    Apr 18, 2017 at 5:50
15

You did something without fully understanding the consequences

That install is irrecoverable, would require a lot of work to reinstall centos5. And that is a bad plan because

  1. CentOS 5 is End of Life, and therefore has no updates. This is extra serious given it sounds like a webserver serving content on the public internet and that you use panel apps to control it.
  2. CentOS >5 would have stopped you from doing this update and killing the box. That's a nice airbag to have.
  3. CentOS 7 claims to support major version upgrades in place. I've never used it, but being able to jump from 7 to 8 when it releases will be very good. Debian's had this since forever, but Redhat always required a reinstall for major version jumps.

Solution

Your best bet is to create a new VPS, fresh install CentOS7, and then reattached the old centos5 disk volume and mount it read-only. Then work to copy (not move) your data from the old drive to the new.

Note this would be my method using AWS. If your VPS provider can't attach disks to different VMs then you'll have to tweak the plan.

No matter what you do, please consider setting up automated backups in the future. It wouldn't save you, but it would make recovery somewhat more flexible. Right now you need the data on that disk in a new working server. Don't loose the existing disk.

Improve this answer
9

How could this happen? Well, quite simple: By removing parts that were critical to your server.

Next steps for you: re-deploy a fresh OS and restore your data from backups.

Improve this answer
4
  • 1
    Uninstalling a package should never ever remove data. There really shouldn't be a need to restore anything, unless some package maintainer royally screwed up.
    – Jörg W Mittag
    Apr 16, 2017 at 11:04
  • 6
    If the OP restores a fresh OS like I recommended, restoration of data and configuration will most assuredly be required.
    – EEAA
    Apr 16, 2017 at 11:06
  • @JörgWMittag I think you might have misunderstood the use of the word "restore." EEAA means that the system will have to be reconfigured and the existing data placed somewhere on the new machine. "Restore" might not be the best word for that, although I could see someone using database restore functionality for that piece.
    – jpmc26
    Apr 19, 2017 at 2:42
  • @JörgWMittag Oh, his data is still there, most likely. It's just that with his system nonfunctional he has no way of accessing it.
    – Shadur
    Apr 20, 2017 at 4:54
4

As pointed out by dragon788 and others in the comments, in Gentoo, the developers also maintain a set of tinderbox packages which are just pre-built, binary versions of a set of core packages of the OS for just such situations. If you lose a core package, you just get the system booted to the LiveCD/DVD, mount the OS drive of the broken server and unpack the tinderbox package(s) to the filesystem, unmount, reboot and, if it boots up correctly, rebuild the packages back to your server's specs and configs.

So, to perform something similar in CentOS, I think you would need to find the right versions of the RPMs that were uninstalled, then boot to a LiveCD/DVD, mount the OS drive and chroot in (perhaps... if you are familiar with how to use the "--relocate" flag for rpm, you might not need to chroot), then reinstall those packages, unmount and reboot.

Of course, since support for CentOS 5 ended last month, after you get the system rebooted properly, you can update it to a current version.

HTH.

Improve this answer
0

You can typically boot from the install media and then chroot to run commands or enter into the current install and recover files or perform a reinstall of packages.

Improve this answer
2
  • 4
    You won't be able to chroot into a root directory with no useful binaries in /usr/bin, etc. It would give the same result as booting into the dead system. Chroot also isn't necessary because once you have the filesystem mounted, you can copy out the files that you want to rescue from /var/www.
    – qris
    Apr 16, 2017 at 11:45
  • Definitely true, I'm recalling installing Gentoo/Arch where there are usually some useful binaries available. I've never uninstalled everything first. ;)
    – dragon788
    Apr 16, 2017 at 13:04

Not the answer you're looking for? Browse other questions tagged .