1

During maintenance yesterday I did a mistake moving files and instead of mv ./* /destination-path, I typed mv /* /destination-path ... :( I stopped the process of moving in the middle and then moved files back to / and everything looked fine. But I logged out and now I can't log in on this server, neither through ssh, nor on the console.

Ssh returns that:

MacBook:johns$ ssh -vv root@groom
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to groom [192.168.133.196] port 22.
debug1: Connection established.
debug1: identity file /Users/johns/.ssh/id_rsa type -1
debug1: identity file /Users/johns/.ssh/id_rsa-cert type -1
debug1: identity file /Users/johns/.ssh/id_dsa type -1
debug1: identity file /Users/johns/.ssh/id_dsa-cert type -1
ssh_exchange_identification: Connection closed by remote host

On the physical console I got:

O0o.oops() [console.c(83)]: Opening console failed

The server is pingable and all virtual machines that run on top of it are accessible via ssh.

Am I completely screwed?

P.S. Unfortunately there aren't any backups. This is not a production server and the only valuable things there are few virtual machines, that are intact and running. The OS is CentOS 5.x running Xen and I have DRAC access to the server. I just need to find a way to log in. Any ideas?

1
  • 5
    Dust off the tapes, I see a restore in your future.
    – MDMarra
    Apr 16, 2012 at 16:45

4 Answers 4

8

No you are not completely screwed, that's why you took the time to make good backups. Now just go to your server and restore them.

6

It's a mistake. We've all made it at some point. This one may not be easily recoverable. You didn't provide any detail as to what the server's operating system distribution/version was. Do you have console access to the system (ILO, DRAC, VMWare, etc.)?

Even if we were to get you into the system, I wouldn't trust the integrity of the installation. Do you have backups?

4
  • Unfortunately there aren't any backups. This is not a production server and the only valuable things there are few virtual machines, that are intact and running. The OS is CentOS 5.x running Xen and I have DRAC access to the server. I just need to find a way to log in. Any ideas?
    – OTS
    Apr 16, 2012 at 17:02
  • This may be outside the scope of the site and question, but recovery is possible with surgery... A Linux live-CD or a CentOS rescue DVD, coupled with rsync of the appropriate system directories (from a similar system), followed by an RPM package verification... But that's a heavy project.
    – ewwhite
    Apr 16, 2012 at 17:06
  • If I can get into the server I can move VMs to another host easily and can reinstall OS, so the only problem now really is how to log in.
    – OTS
    Apr 16, 2012 at 17:24
  • @OTS fire up a rescue CD and copy what you need off of it.
    – MDMarra
    Apr 16, 2012 at 22:22
5

A root console is a loaded bazooka. mv, rm, dd and a few other commands have an aiming arc that passes through your foot. You pulled the trigger at the wrong time and lost a leg.

At this point restoring from backups is your best option. In the future you will be more careful when you're operating as root (really stop and read what you typed before hitting enter), and if/when you make such a mistake in the future you will remember not to log out before testing that you can log in again.

All things in life are a learning experience. This one will probably be painful. Pain is great at reinforcing memory.

2
  • Update: At the console I enter root after login and it waits for prompt for password indefinitely. Corrupted /etc/passwd and /etc/shadow?
    – OTS
    Apr 16, 2012 at 19:12
  • @OTS impossible to say without access to the system, and a boot disk.
    – voretaq7
    Apr 16, 2012 at 21:22
2

You're in a bad spot, no doubt. I doubt you're going to be able to get back into the box without rebooting it. You could give a shot to sftp, and depending on where the login is blowing up, it might work. But I wouldn't count on it.

At reboot point, you have a couple of options. You can boot with a LiveCD/Rescue CD and try to clean up or migrate things from there, or you can try booting with the additional boot option of init=/bin/bash on the kernel line. As long as bash is available, that'll let you get into the box (with no services running, and nothing started). But, it'll get you to a prompt where you can try to get what you need.

My recommendation would be to jump straight to LiveCD, as you'll have greater functionality and you're less likely to have to fight a broken system while you fix it. With Dell hardware and the DRAC, you can remotely mount, access, and boot from a CD disk or ISO image that's located on your local computer, if you don't have easy physical access to the box.

This is a good example of why a lot of people alias rm, mv, cp, etc., to rm -i, mv -i, and cp -i, respectively. It gives you an extra sanity check and (minimal) safety net to catch simple mistakes.

1
  • Update: I booted in rescue mode with CentOS CD and successfully mount the fs. Then I migrated all VM files I need with rsync (using the sparse files option) to another machine. Thank you everyone for your help!
    – OTS
    Apr 20, 2012 at 10:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .