Files disappearing on linux server

Question

I've got 4 specific files that seem to keep disappearing from a user's home directory. As far as we know, there are no cronjobs or other automated tasks that would be removing them. I've setup auditd on them but the logs aren't really showing anything of interest. I can see our backup utility accessing them every night until the point they aren't there anymore, but nothing else. Is there anything that would be causing those files to be removed that would get around auditd?

The files in question are these:

/home/username/.bashrc
/home/username/.bash_profile

as well as a couple of files in that user's .ssh directory. Copies of these files placed into a subfolder called "keepers" get deleted at the same time as well. Changing the permissions on them to 000 and having them owned by root hasn't helped.

I've currently got inotifywait setup to log create,delete,move on that subfolder, so hopefully that will turn up something, although it doesn't log much aside from when it happened, not what caused it.

Answer 1

Solution 1: systemtap
You can use systemtap to show all PIDs that are trying to use unlink() on the inode of .bashrc and .bash_profile files.

Install systemtap and the debug symbols for your kernel.

Create a file with name unlink.stap with the following content:

probe syscall.unlink
{
    printf ("%s(%d) unlink (%s) userID(%d)\n", execname(), pid(), argstr, uid())
}

Then run it with sudo stap unlink.stap

Solution 2: inotify
You can also use inotify to see when the file is deleted.

Solution 3: ftrace
Another solution is to use ftrace:

trace-cmd record -e \*unlink\*

Wait for the file to be deleted, press CTRL+C to stop trace-cmd record ..., then run:

trace-cmd report

Solution 4: bpftrace
Install bpftrace, then run:

bpftrace -e 'BEGIN {printf("PID\tPPID\tCMD\tunlink path\n")} tracepoint:syscalls:sys_enter_unlink* { printf("%d\t%d\t%s\t%s\n", pid, curtask->parent->pid, comm, str(args->pathname)); }'

Answer 2

Are you absolutely sure the user himself is not (accidentally) deleting them ?

I had some clueless (Windows) users with the same problem. Turned out that they deleted those files themselves every time they visited their home-dir with a ftp client. They noticed the .xxxx files (the ftp client didn't hide them) and removed the "clutter".

It never occurred to me they did it to themselves until one of them complained about the spontaneously re-appearing files he had deleted several days before.

Answer 3

in addition to micea's answer, you can chattr +i the files as root and see if anything logs an error when trying to remove them.

Answer 4

We use bash logout scripts (~/.bash_logout) to clean out certain files upon logout - you might check to see if you have that setup, perhaps with a fat-fingered glob in it.

Answer 5

More seems like an intruder, who is doing a find /home/user -name filename -exec rm -f {} \; after all his sneaking :). Just guessing, because you mentioned that the backup files are also getting deleted.

Answer 6

To prevent losing the files and their content you can setup libtrash via LD_PRELOAD. Using libtrash you can do number of things but those that might be interesting to you are

INTERCEPT_UNLINK
INTERCEPT_RENAME
INTERCEPT_FOPEN
INTERCEPT_OPEN

Good article about libtrash can be found here

Other thing you mention, that you chowned files to root and they still got removed. It is because /home/username is owned by username; and if dir has say mod 755; then any file or dir in that dir owned by no matter who can be removed by user; even if it is roots file or dir. It's basically due to fact that removing file in dir means changing the dir contents and user has 7 (in 755) of that dir so he can do whatever he wants.

There are ways to block this as other people already suggested via chattr on ext file systems to set files as immutable (+i). Then one would need to unset the immutable flag before making any changes to file/dir which has the +i flag. Immutable flag / chattr can only be used by root.