8

So im a very new sysadmin, just got out of school and doing my internship. Only problem is that im the only sysadmin in the place and no one to show me the job. Anyway, it's a very small company, one CentOs server with that configuration :

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda3             184G  140G   35G  81% /

tmpfs                 2.3G     0  2.3G   0% /lib/init/rw

udev                  2.3G  212K  2.3G   1% /dev

tmpfs                 2.3G     0  2.3G   0% /dev/shm

/dev/sda1             4.6G  156M  4.2G   4% /boot

/dev/sda4              33G  176M   31G   1% /tmp

/dev/sdb1             1.8T  1.8T     0 100% /media/backupInterne

/dev/sdd1             917G  470G  401G  54% /media/Data

I got here only a few days ago, and noticed the full disk right away and im working on fixing that problem. My other problem here is sda3 now at 81%. 4 days ago, it was at 79%.

I ran the du -ah | sort -rh command on the / root directory, nothing stands out. Did it with a few days appart since the sda3 partition is filling up quickly, no major differences that could explain why its growing.

Thanks a lot

8
  • 4
    My guess would be log growth because sdb1 is full, but we'll see how big /var is.. what do you get from du -sh /*? Dec 6, 2013 at 16:55
  • 1
    If you want to see which files have chaged, then you can use find with -mtime n[smhdw]. I suspect that Shane is right. Part of it may be the log files complaining about the full sdb1 volume. The command could look like this: find / -type f -mtime 1d -print If your find supports --exclude-dir= then you mioght want to exclude /dev and /proc.
    – Hennes
    Dec 6, 2013 at 17:20
  • /var is 1.7G and the size of it barely moved in the past few days, it was my first idea to check that. I always run the du command with an --exclude='media' since there's nothing in that directory then mounted directories Dec 6, 2013 at 17:26
  • Since you wrote that you are a new admin I would to point to one of the most common reasons of growing disk usage. Log files. If you open a file (e.g. the log from a webserver) and later delete that file then the file will still use disk space until the program closes its handle to that file. This last is sometimes solved by sending a signup (kill -1 PID -> reread config file and restart for many deamons) or by the blunt axe of rebooting.
    – Hennes
    Dec 6, 2013 at 17:26
  • In case it is log growth, it may settle in a few days when weekly log rotation kicks in.
    – ptman
    Dec 6, 2013 at 17:57

3 Answers 3

6

Here is what I use in trying to figure out problems like this.

du -s `ls -a | egrep -v '\.\.'` | sort -nr | head

It will show you the usage per directory/file in the current directory. From there you step down into sub dirs until you find something obvious.

Having everything in one big partition can make diagnosing problems like this difficult. Another approach to try is using

lsof 

to see what files are open by the various process and see if you can find some clues. This is very hit or miss though.

2
  • 1
    +1 for mentioning lsof. That tool will come in handy for a new admin (even if it might be touch and go for this problem).
    – Hennes
    Dec 6, 2013 at 17:23
  • the usage per directory/files give me 5 results, none of them over 600k. Dec 6, 2013 at 18:05
3

It sounds a lot like a similar issue I have all the time with deleted files (but the reference is still there).

If we're talking a Linux system, run:

lsof +L1

This will be a list of deletes files, but are still open and being used by something. The key is to get whatever has the filehandle open to release it.

3
  • Sadly didn't give any files that could explain the loss of space that is happening everyday. thanks Dec 6, 2013 at 21:32
  • Is it possible that something is writing to a directory that is then getting mounted over with a filesystem? I had gigabytes of unexplained hair-pulling over that once. While this should never be possible, I was able to prove it can happen.
    – Eirik Toft
    Dec 6, 2013 at 22:32
  • Im thinking of something like that too, but how could that happen? And how do i verify it without umounting everything? Dec 9, 2013 at 18:05
2

I finally figured out what was going on. One of the mounted point wasnt mounting correctly and therefor was doing the backup directly on sda3.

Thanks everyone for the help

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .