Huge /var/log directory inode in ext3 after creating and removing many files

Question

I have Debian Jessie, /var/log on root partition of type ext3 (rw,relatime,data=ordered). Four days ago logrotate dumped tens of thousands files, this was fixed and files deleted. After this:

# du -hc /var/log/
511M    total

Files and subdirectories are only 255M:

#du -hc --exclude=.. --exclude=. /var/log/* /var/log/.*
255M    total

The rest is directory itself:

# ls -ldh /var/log
drwxr-xr-x 15 root root 257M Feb 18 06:55 /var/log
# stat /var/log
  File: ‘/var/log’
  Size: 268496896       Blocks: 524936     IO Block: 4096   directory

Some operations (look random for me) on /var/log take very long time (up to 5 minutes). I can neigher umount nor remount it readonly right now for non-technical reasons.

Why such large directory may arise and why it causes huge lags? How can i "deflate" it without unmounting? What may be caveats of stopping rsyslog, creating a new directory, moving /var/log content to it and renaming to /var/log, similar to suggested in https://serverfault.com/a/487245 ?

Answer 1

With ext3, the size of a directory never decreases. If its size have been high at a moment, it remains high, even after cleaning. In contrast, for example, with xfs, the size of a directory cleaned is adjusted (in real time).

Answer 2

As you stated, there were tens of thousands of files in this directory. The directory entry list is large and reading it all is expensive and slow. If you happen to look at recently accessed file with this still in cache it will be much faster.

Moving the directory and creating a new one is an efficient way to mitigate it. You can remove the big slow one at your convenience even if it takes a while.

fsck might also optimize the directory but that requires it to be unmounted.

For syslog, remote logging could be an alternative to local files, if that helps with your brief logging outage.