I have a server, where I deleted a large file. Let's call it
~/tempfile.txt. Somehow, the deletion didn't work properly, and the disk got corrupted -- meaning,
du -hs * doesn't show that file existing. But
df -h shows that the root partition is full.
Turns out this is a known problem and could happen due to the "deleted" file being accessed by a running process. A stackoverflow answer suggested running
lsof +L1 to get a list of such files. Well, running it gives the following entry:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME none 2241 root txt REG 0,5 8560 0 52453 / (deleted)
A couple of weird things:
- "Name" should be "~/tempfile.txt", but it isn't.
- The inode number - 52453 points to some other file:
- The process with PID 2241 doesn't exist.
The standard procedure is to kill PID accessing this file, but well, there's no such process.
I tried rebooting, but that didn't bring back the free space (which it should if a process is accessing the file). Instead, running
lsof +L1 again gave an entry, but this time the inode number and pid were different, but the "Name" field was the same (
I thought of running fsck now. First ran it directly in dry-run mode using
fsck.ext4 -nvf /dev/sda1 and the output said there are some issues: "Free blocks count wrong, Free inodes count wrong, Block bitmap differences" etc. So I thought let me reboot the system and mount the root partition in read-only mode, and run fsck on that.
fsck.ext4 -nvf /dev/sda1 showed there's nothing wrong! So I tried running
lsof +L1, and surprise, surprise. No ghost files! Is the disk space freed now?
df -h -- still 100% disk usage.
I tried rebooting with rw-mount of root partition, and
lsof +L1 complained again.
This keeps happening deterministically - read-only mode shows no error, whereas read-write mount shows disk errors.
I have no clue what's going on. Does anyone have a reasonable guess on what could be the issue?
I have data backed up, so I can just spin up another server with the data, but this is super weird and I'd like to understand what's going on.