Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This morning I noticed a php error whining about inaccessible device. After further investigation I found that this is due to fact that 100% of virtual memory is used. After a reboot (I thought I'll clean it up for the time) a new problem arrived. The memory is now at 0% however no process can write to it.

Nothing was changed for quite some time thus I'm not sure what could be the problem. No logs indicate what could be the cause though some complain about being unable to write to the virtual memory (which is an effect and not the cause, obviously)

Any ideas where to look for the culprit?

Some of the software running on the server and their status after reboot

opendkim - failed to start
postfix - failed to start
courrier - running
clamav - running
svnserve - running
mongodb - running (failing to read/write)
mysqld - failed to start
vsftpd - running
apache2 - success
share|improve this question
    
How much RAM does the server have? Could you post my.cnf, php.ini? –  Chida Aug 13 '12 at 11:01
    
some general details: imageshack.us/f/209/srv.png , php.ini : pastebin.com/DyHLAX3P , my.cnf : pastebin.com/E13WeZvC –  karka91 Aug 13 '12 at 11:18
    
Can you provide the output when you attempt to start these processes that fail to start? We can't do anything without an error message. –  Patrick Aug 13 '12 at 12:05
    
postfix says ok but it actually spawns no process. Mysql - start: Job failed to start. No entries are made at their logs. The problem isn't with these processes alone as all of the processes are denied access to the memory. Running or new. –  karka91 Aug 13 '12 at 12:14
    
You using a crappy tool that confuses virtual memory with swap space. This is leading you to make the same confusion. I'd strongly suggest dumping it and getting a tool that gives you meaningful numbers that are accurately described. –  David Schwartz Aug 13 '12 at 12:36

2 Answers 2

up vote 3 down vote accepted

Thanks for posting the required information. You have 2GB RAM and it is low. You should upgrade it since you have numerous services running on it.

However, your problem seem to occur because of the following,

Filesystem            Size  Used Avail Use% Mounted on
rootfs                 10G  9.6G     0 100% /
/dev/root              10G  9.6G     0 100% /

Cleanup some files in your root partition / to free space and restart your services. Looks like some of your services weren't able to write to the / file system and hence the errors.

You should also look at adding space to your / partition in the long run since you may run into this issue repeatedly. Your /home partition has 850GB with only 3% usage :)

share|improve this answer
    
I can't believe I actually missed this. Thank you –  karka91 Aug 13 '12 at 16:19

The memory is now at 0% however no process can write to it.

This statement makes no sense. If your virtual memory was not writeable then the system would not boot up, and you certainly wouldn't have sufficient access to get this far in diagnosis.

Do you mean that the swap space is at 0%?

Looking at the images you've posted, I see you're using software which uses wrong terminology. (hint: virtual memory is the entire addressable memory available on the system - for a PC type machine this usually means RAM and swap: real memory IS virtual memory - so is swap).

I think your first step should be to read up a bit on memory at least so you the hang of the terminology.

There's something very wrong with a Linux server which gets into this state - the OOM killer should have been clearing out processes. (and writing logs of what it's doing).

That services are failing to start implies that the machine is still broken after the reboot - and the problems go way beyond a simple memory leak. If your swap space is not writeable and processes aren't starting, then the most likely cause is that the disk(s) are flipping to read-only (if it were just the filesystem then it wouldn't affect swap unless you use swap files instead of swap partitions - rarely a good idea). See the man page for hdparm for more info.

There is no direct link between memory filling up and a disk switching to read-only. Implying 2 distinct issues although I'd recommend a full integrity check of your installation using a host based IDS - of course of you don't already have a fingerprint database then your only option is to use a rootkit detector.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.