0

It has come to my attention that my CentOS 5.6 5.11 VPS has over 16,000 files with a timestamp during a 3-day period in September this year. I can't figure out why that is, as I did not log in via SSH during that time (nor were there any other logins, which would have been very worrisome) and there are no records of actions in Webmin during that period. Those are the only two ways I would have done anything to change system files. The server appears to run fine, but no one else is supposed to have access (other than the hoster, of course), so I want to know what's going on.

I captured the files with those dates in a file thus:

touch -t 201409160000 start.tmp
touch -t 201409190000 stop.tmp
find / -type f -newer start.tmp \! -newer stop.tmp -exec ls -la {} \; > sep-16-18.txt

But what do I do with this information? The volume is crazy!

[root](21:05:55)[~]$ grep "Sep 16" -c sep-16-18.txt
6079
[root](21:06:30)[~]$ grep "Sep 17" -c sep-16-18.txt
2580
[root](21:06:40)[~]$ grep "Sep 18" -c sep-16-18.txt
7653

The whole rest of the month of September was less than 500 files total, so something radical happened in the middle of the month.

Almost all the files were in the /usr/ directory:

/usr/:  16130
/lib/:     49
/sbin/:    49
/etc/:     45
/lib64/:   37
All others: 3

But the breakdown of subdirectories under /usr/ is spread across the board pretty widely:

/usr/lib/:     6514
/usr/include/: 5109
/usr/share/:   3438
/usr/lib64/:    853
/usr/bin/:      105
/usr/sbin/:      61
/usr/libexec/:   36
/usr/kerberos/:  14

Perhaps the file with the earliest timestamp might shed some light (perhaps it started the cascade), but I can't seem to sort the find results by time. I could repeatedly reset start.tmp and stop.tmp to smaller and smaller time periods, re-running find until I got a reasonable number of results to look through manually, but I would probably have no idea what I'm looking at - I'm really not a Linux expert anyway. Perhaps someone can give me some tips on what "questions" I should be asking my server. It looks as if Linux was reinstalled or something, but...

UPDATE: For those who are asking, here are the first several files with timestamps on the 16th. The first one seems unrelated, but then starting at 15:04, for the next couple minutes over 5000 files were updated (or created, I suppose):

1410846366 Tue Sep 16 14:46:06 2014 /etc/default/nss
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/features.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/gnu-versions.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/limits.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/values.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/lib64/libc.so
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/bits/xopen_lim.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/libc-version.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/lib-names.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/stubs.h
1410847468 Tue Sep 16 15:04:28 2014 /usr/include/gconv.h
1410847468 Tue Sep 16 15:04:28 2014 /usr/include/iconv.h
1410847473 Tue Sep 16 15:04:33 2014 /usr/lib64/gconv/gconv-modules
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/bits/locale.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/langinfo.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/locale.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/xlocale.h
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ANSI_X3.4-1968.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ARMSCII-8.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ASMO_449.gz

After that, it goes on with character maps and locale-related files at some length, then more include files, then after about 5500 of that type of file it takes a break for a little while before more .so files and other things that look pretty normal. Does this tell anyone anything? None of I have seen in the list look suspicious; I'm no expert, but it looks more like an innocent update. But I don't know how how an update would happen besides a shell login or Webmin.

17
  • Um, what files changed? Wouldn't some example file names help?
    – ewwhite
    Nov 14, 2014 at 13:48
  • 2
    Could it have been a software update? is there anything in the syslog that corresponds to that time? Are there any entries in any other logs which give you any more info?
    – NickW
    Nov 14, 2014 at 14:01
  • 1
    when was this VPS installed ? Nov 14, 2014 at 14:03
  • 3
    The downvote is probably because this question borders on too broad. You have to focus on specific challenges on Serverfault, and those challenges should not be duplicates of existing questions. It might have fared better if you had kept the question focused on how you perform mtime forensics, which is the question I answer below. If you're looking for someone to tell you what exactly happened in these circumstances (very situation specific, too many possibilities), this question is probably going to continue on its way to being closed.
    – Andrew B
    Nov 15, 2014 at 6:55
  • 2
    I was focussing hard on software updates, but I noticed the OP mentioned the box is C5.6 - which means it's wildly out of date, can't have been patched recently and is ripe for exploitation. But the OP also mentions a yum update in early Oct, which would explain those timestamps but not the OS version number. @OsakaWebbie, please clarify whether the system really is C5.6. The contents of /etc/redhat-release would be really helpful.
    – MadHatter
    Nov 15, 2014 at 7:47

2 Answers 2

6

This question's on its way to getting closed as a duplicate of How do I deal with a compromised server, and I think that's a mistake, because the very correct advice in that question is to nuke the server and restore from known-good media. However, there's absolutely no evidence that this system has been compromised.

OsakaWebbie, you have admitted that you were wrong about the OS version; the system's at 5.11. That given, the files you list are all part of the glibc-headers package, with the exception of /usr/lib64/libc.so which is part of glibc-devel, a package that travels in lockstep with glibc-headers and is generally built at the same time.

I would have done this on a C5 system, but I don't have one to hand. On an up-to-date C6 system, the information about glibc-headers is

[me@lory 2012]$ rpm -qi glibc-headers
Name        : glibc-headers                Relocations: (not relocatable)
Version     : 2.12                              Vendor: CentOS
Release     : 1.149.el6                     Build Date: Wed 15 Oct 2014 03:00:58 BST
Install Date: Fri 24 Oct 2014 20:42:04 BST      Build Host: c6b9.bsys.dev.centos.org
[...]

Note the build and install dates. Now let's look at the files in question:

[me@lory 2012]$ for file in `rpm -ql glibc-headers`; do ls -la $file ; done
[...]
-rw-r--r--. 1 root root 2416 Oct 15 02:44 /usr/include/gnu-versions.h
-rw-r--r--. 1 root root 3057 Oct 15 02:44 /usr/include/gnu/lib-names.h
-rw-r--r--. 1 root root 1337 Oct 15 02:44 /usr/include/gnu/libc-version.h
-rw-r--r--. 1 root root 315 Oct 15 02:44 /usr/include/gnu/stubs.h
-rw-r--r--. 1 root root 6991 Oct 15 02:44 /usr/include/grp.h
-rw-r--r--. 1 root root 4611 Oct 15 02:44 /usr/include/gshadow.h
-rw-r--r--. 1 root root 1949 Oct 15 02:44 /usr/include/iconv.h
-rw-r--r--. 1 root root 4990 Oct 15 02:44 /usr/include/ieee754.h
[...]

Note how the files all have the same timestamp, and it's a few minutes before the Build Date baked into the RPM. From this we conclude that the modification time of an RPM-controlled system file is the modification time it had on the build system where the RPM was created; since a build happens as a monolithic process, it is completely reasonable that all the files in an RPM should have very similar mtimes, and indeed that those in associated packages should have them also, and that those mtimes will not correspond to the date on which the packages were installed.

If you can confirm that your system is indeed at C5.10/5.11, you can stop panicking; you have presented no evidence that anything bad has happened to your system.

Edit: OsakaWebbie, you having confirmed that the system is now at 5.11, I submit this is a non-issue. The modification times on the files you've shown are exactly as you'd expect them to be.

For reference - and it's quite important that you understand this - C5.11 isn't a major release of C5. C6 is a different major release to C5, but C5.11 is just a fully-patched C5.6 (or any other version of C5). You can read about this at some length in an earlier answer I wrote on C/RH versioning if you'd like, but in all probability you brought the system to 5.11 when you did that yum update in late Sep / early Oct. You can confirm that with grep centos-release /var/log/yum.log*, if you like.

3
  • Thank you for the explanation and reassurance. As you can see in a comment on the OP (I saw your comment there before I saw this answer), I did discover that I'm at 5.11. Sorry for that mistake! I wasn't actually panicking - I figured that it was more likely that there was a reasonable explanation that I was simply ignorant of. BTW, I'm not relying on my memory for dates - /var/log/yum.log has entries on 1/1 and then 10/1, so I know my yum update after learning about Shellshock was on 10/1, not in the middle of Sept. And last says there was no login between 9/10 and 9/19. Nov 15, 2014 at 10:28
  • So what that there was no login between those dates? Please read what I wrote above - the mtimes on those files are inherent to the RPM. That fact that you installed the RPM in early October makes no difference to the mtimes; they remain as the RPM sets them.
    – MadHatter
    Nov 15, 2014 at 10:32
  • Oh, after your second comment, I think I understand what you meant (sorry I'm slow). The RPM files were dated in mid-Sept., packed in a .gz or whatever, and then unpacked onto the server when I updated. That makes total sense now. Thank you! I can't upvote anything because I don't have 15 reputation points yet, but it did let me select it as THE answer. Thanks again! Nov 15, 2014 at 10:36
1

I'm going to largely C&P an answer of mine from an unrelated question. This is a better home for it I think. The instructions below are provided with the assumption that you have already followed the instructions for How do I deal with a compromised server? if applicable. Always disconnect the network cable when in doubt.

find /filesystem1 /filesystem2 -xdev -printf '%T@ %t %p\n' | sort -n

%T@ Modification time, epoch seconds (for sorting)

%t Modification time, human readable

%p Full file path

Season to taste. Run on one mounted filesystem at a time or as many as you think is relevant.

What this does is give you a timestamp sorted audit of the most recent modifications to files. This can occasionally give you insight into other things that were happening at the time of the unknown event. It's certainly not bulletproof, and may not reveal anything at all, but I occasionally find this extremely useful in identifying what work was being performed at the time that the unknown event happened. (i.e. someone performing rm -Rf /lib instead of rm -Rf lib)


Asides:

  • If the output of this command contains a wall of common libraries and executables that cannot be associated with an OS update, the most likely cause is a rootkit being overlaid on top of your operating system.

  • Another possibility (that you should be less inclined to consider) is a clueless admin running a greedy rsync from another system to this one. It's unlikely to be a restore from an archival copy (backup software, tar, etc.) as those are generally good about honoring the original mtime.

  • If you can't figure it out, you have no choice but to treat this as an intrusion.

1
  • Thanks! I had tried about a dozen ways to sort a find result list by time, with no success. As for the "hacked server" advice from you and Dennis, I actually think it's unlikely that I'm hacked, and even if I am, I can't unplug, reformat, or most of the other first-responder things suggested in that other thread - it's not my server. I'm on a hosted VPS. Hmm, I wonder if the hoster did something to my stuff in September - I'll write to support and ask. But if they upgraded my stuff and didn't tell me, I should probably change hosters. Nov 15, 2014 at 8:03

Not the answer you're looking for? Browse other questions tagged .