0

I have Citrix XenServer 4.1 running. On this I have Debian Etch which is used as our web development server. I have Virtualmin set-up to manage the domains (MySQL, Apache, etc.). The server is only accessible from outside via HTTP, no SSH or FTP.

Suddenly two days ago we were unable to run any commands on the server. I tried to reboot but that failed. While were able to get to the login screen were unable to login. There were boot up errors of the form:

init: cannot execute "/etc/init.d/rc"

I'v never seen these before. Searching suggested it couldn't find the /bin/sh referenced at the top of the script. This lead to further searches that returned a lot of results about disk failing, etc. This didn't really make much sense to me since the Xen host was fine and another another Debian install running on it was also fine.

Eventually I mounted the disk that held the corrupted install and found the /bin directory was missing. This explained everything that was happening so far. I gave the logs a cursory glance to see if anything popped out at me. None did. I copied the /bin directory from another Debian Etch install to see if I could get the server back up. No luck, things like aptitude did not work, etc. I decided to do a fresh install since that would not take too much time and I have now restored everything.

I've been looking at the logs but I can't find anything that indicates what happened so far.

Can anyone suggest or point me in a direction that will help explain how a critical directory, such as /bin, suddenly disappeared? I would hate for this happen again.

Thanks in advance.

2 Answers 2

1

I only happened to see such symptoms on machines with a damaged filesystem... or with a dump root-user doing kind of (don't try that at home)

# rm -rf /bin

instead of

# rm -rf ./bin

inside his whatever-app-directory...

So ask yourself, your users and than check your (file-)system. Often fs mounts are configured to remount in read-only when errors occur, causing commands to crash that want to write anything back to disk (think of .lock, .pid or .log). But if your /bin has really gone... I assume:

  1. filesystem crash
  2. a naughty skript/daemon/app running with root-privilege
  3. someone logged in as root made a big big mistake

or... Mr. Murphy

4
  • Thanks Marcus. I am more inclined to think it was a naughty skript/daemon/app but I am not sure how to verify this since the logs are basically useless.
    – canen
    Sep 17, 2009 at 18:24
  • You're right, the logs won't show much unless you specifically set up accounting or snoopy or the like. However, check auth.log and ~/.bash_history anyways. You might get lucky.
    – Matt
    Sep 17, 2009 at 18:43
  • My guess: someone or something issued rm -rf $FOO_HOME/bin (or some other variable) in a context where the variable in question wasn't set (e.g. cron can give you surprises like that). Sep 17, 2009 at 18:48
  • ...and examine syslog to get a closer time-slot... there might be some more processes depending on /bin/<stuff> which possibly yielded errors. Than you might find a cronjob or something that was executed wight before. Good Luck! Sep 17, 2009 at 19:49
0

You could also examine history for root user

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .