Very strange disk corruption error

Question

I have a server, where I deleted a large file. Let's call it ~/tempfile.txt. Somehow, the deletion didn't work properly, and the disk got corrupted -- meaning, du -hs * doesn't show that file existing. But df -h shows that the root partition is full.

Turns out this is a known problem and could happen due to the "deleted" file being accessed by a running process. A stackoverflow answer suggested running lsof +L1 to get a list of such files. Well, running it gives the following entry:

COMMAND  PID USER  FD   TYPE DEVICE SIZE/OFF NLINK  NODE NAME
none    2241 root txt    REG    0,5     8560     0 52453 / (deleted)

A couple of weird things:

1. "Name" should be "~/tempfile.txt", but it isn't.
2. The inode number - 52453 points to some other file: /usr/src/linux-azure-headers-5.0.0-1025/include/uapi/video/sisfb.h.
3. The process with PID 2241 doesn't exist.

The standard procedure is to kill PID accessing this file, but well, there's no such process.

I tried rebooting, but that didn't bring back the free space (which it should if a process is accessing the file). Instead, running lsof +L1 again gave an entry, but this time the inode number and pid were different, but the "Name" field was the same (/).

I thought of running fsck now. First ran it directly in dry-run mode using fsck.ext4 -nvf /dev/sda1 and the output said there are some issues: "Free blocks count wrong, Free inodes count wrong, Block bitmap differences" etc. So I thought let me reboot the system and mount the root partition in read-only mode, and run fsck on that.

Well. fsck.ext4 -nvf /dev/sda1 showed there's nothing wrong! So I tried running lsof +L1, and surprise, surprise. No ghost files! Is the disk space freed now? df -h -- still 100% disk usage.

I tried rebooting with rw-mount of root partition, and fsck & lsof +L1 complained again.

This keeps happening deterministically - read-only mode shows no error, whereas read-write mount shows disk errors.

I have no clue what's going on. Does anyone have a reasonable guess on what could be the issue?

I have data backed up, so I can just spin up another server with the data, but this is super weird and I'd like to understand what's going on.

Answer 1

Image the disk to have a copy to work on.

Skim the disk usage of the top of the tree to confirm where most of the storage is being used. (Or not, if the problem file cannot be enumerated by dentries.) ncdu is a program that visualizes space utilization of a tree.

fsck the block device while read write and unmounted (and save the output). This sounds like a rootfs, in which case consider attaching the disk to a different instance and fsck there. Otherwise, configure the early boot environment to always fsck, then reboot. How to do this depends on distro, for example systemd does the early fsck differently than previous init scripts. Either way, check for things like orphaned inodes that may gain a reference in lost+found.

lsof has limited use if the file isn't open at the time. Possible you are seeing unrelated open unlinked files.

---

With the backup in hand, a restore or a new instance is a rapid recovery. How far you want to investigate depends on your curiosity, and if this a problem that might reoccur.