PostgreSQL won't start because it "Cannot allocate memory"

Question

I'm running PostgreSQL 8.4.5 on Ubuntu 10.04. I run an EC2 instance with a few sites running on nginx. Most of these sites run on Django and connect to this Postgres instance.

For some reason, at 8:45pm tonight, Postgres went down. I logged into the machine, and I see this error message:

 * Starting PostgreSQL 8.4 database server
 * The PostgreSQL server failed to start. Please check the log output:
2011-04-17 04:46:49 UTC FATAL:  could not create shared memory segment: Cannot allocate memory
2011-04-17 04:46:49 UTC DETAIL:  Failed system call was shmget(key=5432001, size=16211968, 03600).
2011-04-17 04:46:49 UTC HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space. To reduce the request size (currently 16211968 bytes), reduce PostgreSQL's shared_buffers parameter (currently 1792) and/or its max_connections parameter (currently 53).
        The PostgreSQL documentation contains more information about shared memory configuration.
   ...fail!

The first thing I did was change the Linux shared memory allocation.

sysctl -w kernel.shmmax=367108864>
sysctl -p /etc/sysctl.conf

That didn't do it. So then I edited /etc/postgresql/8.4/main/postgresql.conf and lowered the max_connections value. That worked...for about 10 minutes.

Now I'm getting the same error, no matter how low max_connections is. I need to set it to at least 9 (since that's the number of Django projects that need access to this Postgres server).

Any thoughts on how I might go about fixing this?

Answer 1

You can use the 'ipcs' command to list all the SHM segments. If a program crashes without deleting them, they may be lying around consuming memory; you can remove these manually with the 'ipcrm' command.

Answer 2

Some googling indicates that you should reduce the postgres shared_buffers and max_connections numbers. have you done anything with shared_buffers yet?

Have you reviewed the postgresql shared memory documentation? Lots of details in there about how to tune the shared memory parameters on a linux machine.

Answer 3

Can you show more verbose cluster log ? Is postmaster crashing arbitrarily ? Which code it returns to OS ? How many PostgreSQL clusters you have, only 8.4/main ?

What about your kernel.shmall and kernel.shmmni ? Try to use ipcs -ml, ipcs -m and check your memory (free, top, System Monitor) usage . Try to calm down OOM killer:

vm.overcommit_memory = 2
vm.overcommit_ratio = 50

AFAIK sysctl -w does not change params permanently (only until next OS reboot) and you need to add kernel.shmmax=367108864 to /etc/sysctl.conf.

If possible upgrade your PostgreSQL to 8.4.8 as suggested in Versioning policy:

We always recommend that all users run the latest available minor release for whatever major version is in use.

I think this is probably OOM killer issue, because it kills postmaster with SIGKILL signal and without releasing shared memory. Look at documentation:

Important: It is best not to use SIGKILL to shut down the server. Doing so will prevent the server from releasing shared memory and semaphores, which might then have to be done manually before a new server can be started. Furthermore, SIGKILL kills the postgres process without letting it relay the signal to its subprocesses, so it will be necessary to kill the individual subprocesses by hand as well.

and here:

n Linux 2.4 and later, the default virtual memory behavior is not optimal for PostgreSQL. Because of the way that the kernel implements memory overcommit, the kernel might terminate the PostgreSQL server (the master server process) if the memory demands of another process cause the system to run out of virtual memory.

BTW max_connections are "cheap". Rather reduce shared_buffers.