Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

During maintenance yesterday I did a mistake moving files and instead of mv ./* /destination-path, I typed mv /* /destination-path ... :( I stopped the process of moving in the middle and then moved files back to / and everything looked fine. But I logged out and now I can't log in on this server, neither through ssh, nor on the console.

Ssh returns that:

MacBook:johns$ ssh -vv root@groom
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to groom [192.168.133.196] port 22.
debug1: Connection established.
debug1: identity file /Users/johns/.ssh/id_rsa type -1
debug1: identity file /Users/johns/.ssh/id_rsa-cert type -1
debug1: identity file /Users/johns/.ssh/id_dsa type -1
debug1: identity file /Users/johns/.ssh/id_dsa-cert type -1
ssh_exchange_identification: Connection closed by remote host

On the physical console I got:

O0o.oops() [console.c(83)]: Opening console failed

The server is pingable and all virtual machines that run on top of it are accessible via ssh.

Am I completely screwed?

P.S. Unfortunately there aren't any backups. This is not a production server and the only valuable things there are few virtual machines, that are intact and running. The OS is CentOS 5.x running Xen and I have DRAC access to the server. I just need to find a way to log in. Any ideas?

share|improve this question
5  
Dust off the tapes, I see a restore in your future. –  MDMarra Apr 16 '12 at 16:45

4 Answers 4

No you are not completely screwed, that's why you took the time to make good backups. Now just go to your server and restore them.

share|improve this answer

It's a mistake. We've all made it at some point. This one may not be easily recoverable. You didn't provide any detail as to what the server's operating system distribution/version was. Do you have console access to the system (ILO, DRAC, VMWare, etc.)?

Even if we were to get you into the system, I wouldn't trust the integrity of the installation. Do you have backups?

share|improve this answer
    
Unfortunately there aren't any backups. This is not a production server and the only valuable things there are few virtual machines, that are intact and running. The OS is CentOS 5.x running Xen and I have DRAC access to the server. I just need to find a way to log in. Any ideas? –  OTS Apr 16 '12 at 17:02
    
This may be outside the scope of the site and question, but recovery is possible with surgery... A Linux live-CD or a CentOS rescue DVD, coupled with rsync of the appropriate system directories (from a similar system), followed by an RPM package verification... But that's a heavy project. –  ewwhite Apr 16 '12 at 17:06
    
If I can get into the server I can move VMs to another host easily and can reinstall OS, so the only problem now really is how to log in. –  OTS Apr 16 '12 at 17:24
    
@OTS fire up a rescue CD and copy what you need off of it. –  MDMarra Apr 16 '12 at 22:22

A root console is a loaded bazooka. mv, rm, dd and a few other commands have an aiming arc that passes through your foot. You pulled the trigger at the wrong time and lost a leg.

At this point restoring from backups is your best option. In the future you will be more careful when you're operating as root (really stop and read what you typed before hitting enter), and if/when you make such a mistake in the future you will remember not to log out before testing that you can log in again.

All things in life are a learning experience. This one will probably be painful. Pain is great at reinforcing memory.

share|improve this answer
    
Update: At the console I enter root after login and it waits for prompt for password indefinitely. Corrupted /etc/passwd and /etc/shadow? –  OTS Apr 16 '12 at 19:12
    
@OTS impossible to say without access to the system, and a boot disk. –  voretaq7 Apr 16 '12 at 21:22

You're in a bad spot, no doubt. I doubt you're going to be able to get back into the box without rebooting it. You could give a shot to sftp, and depending on where the login is blowing up, it might work. But I wouldn't count on it.

At reboot point, you have a couple of options. You can boot with a LiveCD/Rescue CD and try to clean up or migrate things from there, or you can try booting with the additional boot option of init=/bin/bash on the kernel line. As long as bash is available, that'll let you get into the box (with no services running, and nothing started). But, it'll get you to a prompt where you can try to get what you need.

My recommendation would be to jump straight to LiveCD, as you'll have greater functionality and you're less likely to have to fight a broken system while you fix it. With Dell hardware and the DRAC, you can remotely mount, access, and boot from a CD disk or ISO image that's located on your local computer, if you don't have easy physical access to the box.

This is a good example of why a lot of people alias rm, mv, cp, etc., to rm -i, mv -i, and cp -i, respectively. It gives you an extra sanity check and (minimal) safety net to catch simple mistakes.

share|improve this answer
    
Update: I booted in rescue mode with CentOS CD and successfully mount the fs. Then I migrated all VM files I need with rsync (using the sparse files option) to another machine. Thank you everyone for your help! –  OTS Apr 20 '12 at 10:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.