1

I'm having a strange issue with a CentOS 6 box. A couple of hours ago I lost most of the files in /bin - ls,ps,mount,rpm for example are gone. It seems that the missing files are those provided by coreutils, but other coreutils files (man pages, stuff in /etc, /usr/bin) are in place.

What on earth could have caused this? Anyone seen anything similar?

Earlier today I noticed that crond is not running on the system, and I restarted it. No idea yet why it was stopped. It had probably been sleeping for some time, as it ran daily weekly and monthly scripts after starting it. My only theory so far is that something cron runs has deleted the files, but I don't see how that would happen. The only custom cronjob is a backup script which runs mysqldump and gzip to an nfs share.

I swear I did not delete the files by accident.. And no other admins have access to the system currently as it's not yet in production.

Here's a listing of what I now have in /bin. "dir" is still working.

# dir /bin
awk                   dd             gettext       lscgroup  sleep        umount
basename              df             grep          mail      sort         uname
bash                  dnsdomainname  gtar          ping      stty         unicode_stop
cgclassify            dumpkeys       gunzip        readlink  su           unlink
cgcreate              ed             hostname      rmdir     sync         zcat
cgexec                ex             ipcalc        rnano     tar
cut                   false          iptables-xml  rview     taskset
dbus-cleanup-sockets  find           link          setfont   touch
dbus-uuidgen          gawk           loadkeys      sh        traceroute6

That's not much.. I have a feeling I'm missing something vital/simple here but it's eluding me. I can provide more info as requested.

Update: I have been browsing through system logs and find no weird logins, rm-commands or errors. Lots of "command not found" messages in root's mailbox from cron jobs, but that's expected in this case. The server is a VMware VM running apache and mysql. Intrusion is not likely (yeah, I know what that sounds like)..

Update 2:

I intalled another Centos 6 VM (from dvd iso, the previous problem host was from the minimal image) and quickly ran into similar problems, but now so far I'm missing just a couple of commands...

Here's what I did, a complete list:

  • installed centos 6 from dvd .iso, into esxi 4.1 vm
  • selected "web server" as base package in install
  • configured network interface
  • configured proxy in /etc/profile (with vi - note this later)
  • installed mysqld and some php packages
  • removed postgresql
  • put some data in mysql db + apache documentroot
  • yum update
  • tried to configure an apache virtualhost - noticed that vi is missing
  • noticed that /bin/cp is missing as well
  • coreutils & vim-minimal are installed according to rpm

yum.log shows that during the yum update vim-minimal was updated, coreutils not, no package removals visible. Can't see anything strange in dmesg, /var/log/secure, /var/log/messages

Improve this question
13
  • 2
    Um...did you run fsck? And have you been keeping it up to date, and check the system logs for odd activity??
    – Bart Silverstrim
    Oct 19, 2011 at 13:52
  • /var/log/auth.log should inform you of superuser commands that have been issued, I would have a look through there to see if there is any mischief - you will at least be able to convise yourself that there are no sudo rm in there.
    – Tom
    Oct 19, 2011 at 13:55
  • Also, you can type last to check and see if there were any "interesting" logins.
    – Matthew
    Oct 19, 2011 at 13:58
  • Have not run fsck. Have done sporadic yum updates (not in a couple of weeks now). I can't find anythig strange in logs, no weird package installs for example - I did install dos2unix and ImageMagick today, that's about the only change as far as I remember. I've been browsing through root+user history and can find nothing weird.
    – mikkoko
    Oct 19, 2011 at 13:59
  • Yeah and I can see no weird logins in lastlog or /var/log/secure. Should've mentioned all this in the question but was too quick to post.
    – mikkoko
    Oct 19, 2011 at 14:02

2 Answers 2

Reset to default
2

Like so often, this was down to human error. It seems the task list I recorded was not complete after all. At some point an unfinished backup script was pushed to the server, which instead of cleaning up old backups, cleaned up ... other stuff.

Improve this answer
1
  • 1
    I LOLed at this... "Tune in with us next time on 'When Automation Goes Bad'..."
    – gWaldo
    Jan 24, 2012 at 14:28
1

Have you tried running a file integrity checker on a known clean version of a completely offline installation verses your current installation? Did you run a check on the installation image/CD? Do these files re-disappear if you re-install the relevant packages? Do they happen at particular intervals or over time (file integrity checker)? Are you losing gaining space on your drive (want to check whether or not files are being hidden as opposed to deleted)? Have you tried booting from a forensic/other OS and running an audit from there? For instance, BackTrack or Knoppix? Have you run a network/packet sniffer to look to look for anomalous activity?

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .