My system is CentOS 6.1, /var/log directory was gone, what are the possible problems? I created a new /var/log, and 6 hours later it was gone again.

Is there any system log that I can inspect?

  • 3
    I agree with the general feeling that it's likely the server has been compromised. You may wish to look at serverfault.com/questions/218005/… which is the canonical answer on recovering from intrusion.
    – MadHatter
    Dec 29 '11 at 12:18

Most probable answers:

  • your server was hacked
  • you installed a program that removes files
  • you wrote a program that removes files

The first answer is much more likely...


Looks like your server is compromised. I would recommend to check the following:

  • crontab
  • output of ps

Immediately disable root logins. Each and everything requiring superuser privilege should be run using sudo. commands executed via sudo are logged with the username and other details at syslog or you can define the log path in /etc/sudoers. I recommend to log it outside /var/log since your /var/log is vanishing.

As a final measure you can rename /bin/rm to /bin/myrm or something else and tell trusted users to use it instead of rm. This way rouge scripts executing rm will not be able to do anything.

  • 1
    What makes you think they're executing rm, and not just using the rm syscall? Dec 29 '11 at 12:35
  • 7
    Bugger disabling anything. Turn it off at the wall. Dec 29 '11 at 12:35
  • of course I cannot be sure that they are not using the system call. It is just something I would have checked. Dec 29 '11 at 12:48
  • What @TomO'Connor said -- Rip the plug out and do proper forensics later, but if it smells compromised get it the hell off the network.
    – voretaq7
    Dec 29 '11 at 20:23
  • 2
    Also note that if you suspect the system is compromised NONE of the software installed on it can be assumed to be telling the truth -- You cannot "fix" a compromised system because you cannot trust anything on it (right up to and including the kernel - Remember Linux lets you hot-patch running kernels, so an attacker can bury stuff that will hose even "known good" binaries run off a CD).
    – voretaq7
    Dec 29 '11 at 20:26

Booting this system off a rescue CD (after power off) should be ok, as long as you don`t run any binaries off that system. If you have a backup of your server before it misbehaved compare the files, inspect crontab, look at tmpwatch, ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.