after some wrong php code was exceuted on one of our servers, which tried to open up a file which didn't exist, we ended up with huge apache log files which weren't configured yet to rotate. We just deleted those logfiles. After which we saw that our disk didn't really clean up. The output of a df -h shows the following
Filesystem Size Used Avail Use% Mounted on /dev/sda1 18G 16G 1.1G 94% / udev 999M 12K 999M 1% /dev tmpfs 403M 848K 402M 1% /run none 5.0M 0 5.0M 0% /run/lock none 1007M 72K 1007M 1% /run/shm /dev/mapper/vg_ftp-ftp 9.9G 5.3G 4.2G 56% /mnt/local/ftp
Inodes seem to be ok as well
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 1168128 227009 941119 20% / udev 215094 442 214652 1% /dev tmpfs 219463 384 219079 1% /run none 219463 6 219457 1% /run/lock none 219463 3 219460 1% /run/shm /dev/mapper/vg_ftp-ftp 655360 1553 653807 1% /mnt/local/ftp
After investigating with du and ncdu we still couldn't find where the space has vanished. Later we thought it were open file handlers which were still there so we did the lsof | grep deleted which gave us a few files, including the /sbin/mountall which is a bit scary.
mountall 287 root txt REG 8,1 120404 9334 /sbin/mountall (deleted) mysqld 615 mysql 4u REG 8,1 0 416 /tmp/iba1fEnV (deleted) mysqld 615 mysql 5u REG 8,1 0 899 /tmp/ibcxXlJG (deleted) mysqld 615 mysql 6u REG 8,1 0 900 /tmp/ibdby44r (deleted) mysqld 615 mysql 7u REG 8,1 0 901 /tmp/ibYaPXvd (deleted) mysqld 615 mysql 11u REG 8,1 0 986 /tmp/ibS9XlvZ (deleted) ntpd 1231 ntp 3r REG 8,1 481 1427 /etc/network/interfaces~ (deleted)
We are trying to avoid a reboot. Does anyone have an idea where this could come from or what else we could check to find the missing space?