Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have Citrix XenServer 4.1 running. On this I have Debian Etch which is used as our web development server. I have Virtualmin set-up to manage the domains (MySQL, Apache, etc.). The server is only accessible from outside via HTTP, no SSH or FTP.

Suddenly two days ago we were unable to run any commands on the server. I tried to reboot but that failed. While were able to get to the login screen were unable to login. There were boot up errors of the form:

init: cannot execute "/etc/init.d/rc"

I'v never seen these before. Searching suggested it couldn't find the /bin/sh referenced at the top of the script. This lead to further searches that returned a lot of results about disk failing, etc. This didn't really make much sense to me since the Xen host was fine and another another Debian install running on it was also fine.

Eventually I mounted the disk that held the corrupted install and found the /bin directory was missing. This explained everything that was happening so far. I gave the logs a cursory glance to see if anything popped out at me. None did. I copied the /bin directory from another Debian Etch install to see if I could get the server back up. No luck, things like aptitude did not work, etc. I decided to do a fresh install since that would not take too much time and I have now restored everything.

I've been looking at the logs but I can't find anything that indicates what happened so far.

Can anyone suggest or point me in a direction that will help explain how a critical directory, such as /bin, suddenly disappeared? I would hate for this happen again.

Thanks in advance.

share|improve this question

2 Answers 2

I only happened to see such symptoms on machines with a damaged filesystem... or with a dump root-user doing kind of (don't try that at home)

# rm -rf /bin

instead of

# rm -rf ./bin

inside his whatever-app-directory...

So ask yourself, your users and than check your (file-)system. Often fs mounts are configured to remount in read-only when errors occur, causing commands to crash that want to write anything back to disk (think of .lock, .pid or .log). But if your /bin has really gone... I assume:

  1. filesystem crash
  2. a naughty skript/daemon/app running with root-privilege
  3. someone logged in as root made a big big mistake

or... Mr. Murphy

share|improve this answer
    
Thanks Marcus. I am more inclined to think it was a naughty skript/daemon/app but I am not sure how to verify this since the logs are basically useless. –  canen Sep 17 '09 at 18:24
    
You're right, the logs won't show much unless you specifically set up accounting or snoopy or the like. However, check auth.log and ~/.bash_history anyways. You might get lucky. –  Matt Sep 17 '09 at 18:43
    
My guess: someone or something issued rm -rf $FOO_HOME/bin (or some other variable) in a context where the variable in question wasn't set (e.g. cron can give you surprises like that). –  reinierpost Sep 17 '09 at 18:48
    
...and examine syslog to get a closer time-slot... there might be some more processes depending on /bin/<stuff> which possibly yielded errors. Than you might find a cronjob or something that was executed wight before. Good Luck! –  Marcus Spiegel Sep 17 '09 at 19:49
    
Thanks. I'll look further. –  canen Sep 18 '09 at 0:35

You could also examine history for root user

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.