-1

This morning I noticed a php error whining about inaccessible device. After further investigation I found that this is due to fact that 100% of virtual memory is used. After a reboot (I thought I'll clean it up for the time) a new problem arrived. The memory is now at 0% however no process can write to it.

Nothing was changed for quite some time thus I'm not sure what could be the problem. No logs indicate what could be the cause though some complain about being unable to write to the virtual memory (which is an effect and not the cause, obviously)

Any ideas where to look for the culprit?

Some of the software running on the server and their status after reboot

opendkim - failed to start
postfix - failed to start
courrier - running
clamav - running
svnserve - running
mongodb - running (failing to read/write)
mysqld - failed to start
vsftpd - running
apache2 - success
7
  • How much RAM does the server have? Could you post my.cnf, php.ini?
    – Chida
    Aug 13, 2012 at 11:01
  • some general details: imageshack.us/f/209/srv.png , php.ini : pastebin.com/DyHLAX3P , my.cnf : pastebin.com/E13WeZvC
    – karka91
    Aug 13, 2012 at 11:18
  • Can you provide the output when you attempt to start these processes that fail to start? We can't do anything without an error message.
    – phemmer
    Aug 13, 2012 at 12:05
  • postfix says ok but it actually spawns no process. Mysql - start: Job failed to start. No entries are made at their logs. The problem isn't with these processes alone as all of the processes are denied access to the memory. Running or new.
    – karka91
    Aug 13, 2012 at 12:14
  • You using a crappy tool that confuses virtual memory with swap space. This is leading you to make the same confusion. I'd strongly suggest dumping it and getting a tool that gives you meaningful numbers that are accurately described. Aug 13, 2012 at 12:36

2 Answers 2

3

Thanks for posting the required information. You have 2GB RAM and it is low. You should upgrade it since you have numerous services running on it.

However, your problem seem to occur because of the following,

Filesystem            Size  Used Avail Use% Mounted on
rootfs                 10G  9.6G     0 100% /
/dev/root              10G  9.6G     0 100% /

Cleanup some files in your root partition / to free space and restart your services. Looks like some of your services weren't able to write to the / file system and hence the errors.

You should also look at adding space to your / partition in the long run since you may run into this issue repeatedly. Your /home partition has 850GB with only 3% usage :)

0
2

The memory is now at 0% however no process can write to it.

This statement makes no sense. If your virtual memory was not writeable then the system would not boot up, and you certainly wouldn't have sufficient access to get this far in diagnosis.

Do you mean that the swap space is at 0%?

Looking at the images you've posted, I see you're using software which uses wrong terminology. (hint: virtual memory is the entire addressable memory available on the system - for a PC type machine this usually means RAM and swap: real memory IS virtual memory - so is swap).

I think your first step should be to read up a bit on memory at least so you the hang of the terminology.

There's something very wrong with a Linux server which gets into this state - the OOM killer should have been clearing out processes. (and writing logs of what it's doing).

That services are failing to start implies that the machine is still broken after the reboot - and the problems go way beyond a simple memory leak. If your swap space is not writeable and processes aren't starting, then the most likely cause is that the disk(s) are flipping to read-only (if it were just the filesystem then it wouldn't affect swap unless you use swap files instead of swap partitions - rarely a good idea). See the man page for hdparm for more info.

There is no direct link between memory filling up and a disk switching to read-only. Implying 2 distinct issues although I'd recommend a full integrity check of your installation using a host based IDS - of course of you don't already have a fingerprint database then your only option is to use a rootkit detector.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .