How to go about rotating logs which are arbitrary named and placed in deeply nested directories?

Question

I have a couple of hosts which are basically a playground for developers. On these hosts, each of them has a directory under /tmp where he is free to do all he wants - store files, write logs etc.

Of course, the logs are to be rotated, or else the disc will be 100% full in a week. The files can be plenty, but I've dealt with it with paths like /tmp/[a-e]*/* and so on and lived happily for a while, but as they try new cool stuff on the machine logrotate rules grow ugly and unmanageable, and it's getting more difficult to understand which files hit the glob. Also, logrotate would segfault if asked to rotate a socket.

I don't feel like trying to enforce some naming policies in that environment, I think it's going to take quite a lot of time and get people annoyed and still would fail at some point.

And I still need to manage the logs, not just rm the dirs at night.

So is it a good idea in circumstances like these to write a script which would handle these temporary files? I prefer sticking with standard utilities whenever possible, but here I think logrotate is getting less and less manageable.

And probably someone heard of some logrotate alternatives which would work well in such an environment? I don't need emailing logs or some other advanced features, so theoretically some well commented find | xargs would do.

P.S. I do have a log aggregator but this stuff is not going to touch my little cute logstash machine.

Answer 1

In development environments like this I was a fan of setting up a specific directory that folks needed to symlink their log directories into, usually with a forced naming convention like 'username_branch' or 'username-ProductInTesting' I'd then logrotate all the symlinked directories.

The couple times that folks forgot to add the symlink and the disk got filled up the other devops folks got pretty steamed at them.. So I was off the hook.

Answer 2

Since these are developer machines, have you asked them to manage this stuff themselves? Give them the tools and teach them the process, and then it's on them.

Alternatively, if you can't fob this off on them, can you require them to alert you when they add a new log directory so you don't have to find the new ones?

Both of these approaches come down to "Communicate between teams and force accountability", you'll note. How do you handle this in prod? I hope the same way, right?