How do I figure out what is filling up my AWS EC2 file system? [duplicate]

Question

Elastic Beanstalk is telling me that my root file system is 96% full. When I SSH into the EC2 instance and run df, I can see that my root file system is 7.95GB/8GB used:

Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs         1980524       0   1980524   0% /dev
tmpfs            1988500       0   1988500   0% /dev/shm
tmpfs            1988500   33180   1955320   2% /run
tmpfs            1988500       0   1988500   0% /sys/fs/cgroup
/dev/nvme0n1p1   8376300 7959264    417036  96% /
tmpfs             397700       0    397700   0% /run/user/1000

However, if a run a du command, such as sudo du -h -x -d 1 /, I can only see a total of 2.4GB being used:

35M     /etc
0       /local
26M     /tmp
465M    /var
1.3G    /usr
32M     /boot
608K    /home
0       /media
0       /mnt
560M    /opt
17M     /root
0       /srv
2.4G    /

None of those directories look excessively large for what's running. I thought my problem was going to be growing log files, but /var/log is only taking up 36MB.

I don't understand where the other 5.5GB is being used from my root file system since it isn't showing up using du.

Answer 1

It turns out that a large number of files were deleted but are still opened by a running process.

Running sudo lsof -nP | grep '(deleted)' listed hundreds of large log files that were deleted by someone else but were still opened by my Node application. Restarting the app cleared them up and freed up the space.