Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Woke up this morning to find out MongoDB grew to all available disk space for some reason. I cannot repair the db to compact it, as I have no space left. I can't even start postgres to remove an old database which would free up a few gigs more. I literally have zero free disk space.

I did delete a 9GB file but the space is not showing up at all when I "df -h".

I have to clear up space on the drive asap as its a production server and we're down. What to do?

share|improve this question

2 Answers 2

the space will not show up until all processes release the file. Try using lsof to see what's using the file.

share|improve this answer
    
The file i deleted was an old db sql dump. It was not used by anything. –  Harel Mar 21 '11 at 11:44
    
To make things worst, i droped the bad mongodb collection but i cannot run repairDatabase which would reclaim the space, because I don't have any space left! Its a catch22 –  Harel Mar 21 '11 at 11:45
    
Did you double check lsof to see if there are any open file handles? –  Bart Silverstrim Mar 21 '11 at 12:22
    
Yes. I also stopped all the services I could stop. No matter what I deleted, df -h shows 0 bytes left on drive (even after i run 'sync' or reboot). –  Harel Mar 21 '11 at 12:24
up vote 3 down vote accepted

Although the answer provided is valid in many cases, in my scenario I ended up falling into the ext3 5% reserved root space. I could have recovered this space had I known about it at the time. I ended up using a USB drive as a repair path to mongodb reclaiming back 850GB.

This is how to recover that ext3 5% should anyone be interested:

http://www.andremiller.net/content/recovering-reserved-space-ext2-and-ext3-filesystems

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.