Shared memory: shmget fails: No space left on device -- how to increase limits?

Question

I call shmget many times in my program, to get shm of of average size 85840 bytes. I obtain about 32771 shm ok, then shmget doesn't return a shm, but the error: "No space left on device".

I have increased the kernel limits to:

$ sysctl -A|grep shm
kernel.shmmax = 33554432
kernel.shmall = 1677721600
kernel.shmmni = 409600

But still get the issue. Why?

Do I have to put something into /etc/security/limits.conf too? I only have "user - nofile 1000000" because the program also opens about as many files as shms.

This is the output of free

$ free
          total       used       free     shared    buffers     cached
Mem:       8150236    7261676     888560          0     488100    3270792
-/+ buffers/cache:    3502784    4647452 
Swap:     12287992     554692   11733300

And ipcs

$ ipcs -lm                                                                         

------ Shared Memory Limits --------
max number of segments = 409600
max seg size (kbytes) = 1638400
max total shared memory (kbytes) = 6710886400
min seg size (bytes) = 1

Since I assume shm is capable of being swapped out, there should be enough space.

Answer 1

It turns out shmmni is limited to 32768 in the kernel:

#define IPCMNI 32768  /* <= MAX_INT limit for ipc arrays (including sysctl changes) */

in the file ...version.../include/linux/ipc.h.

So short of recompiling the kernel, that is the hard limit on the number of shared memory segments.

Answer 2

Use ipcs -l to check the limits actually in force, and ipcs -a and ipcs -m to see what is in use, so you can compare the output. Look at the nattch column: are there segments with no processes attached that were not removed when processes exited (which normally means the program crashed)? ipcrm can clear them, although if this is a test machine, a reboot is quicker (and will make sure your changes to limits are picked up).

Your kernel parameters seem odd. In particular, shmall is a count of pages, not bytes, and 4kB is the default page size (run getconf PAGESIZE to check what you are using). How many terabytes of RAM do you have?

Now, you say you get about 32771 shared memory segments, which is also about 32768 (or 2 to the 15) which suggests a signed 16-bit int is the limiting factor. And what kernel are you running (as this will have its own limits)? The two may be related.

Answer 3

As shmget() allocates a new shared memory segment, and you seem to use a lot of them (considering your limits.conf), isn't possible you are using too many shared memory segments? I do not have a lot of experience with calling shmget(), but it seems to me the number of possible open files (1000000) is larger than the amount of allowed shared memory segments (SHMMNI), which is 409600.

Answer 4

Just do:

sysctl -a | grep shm

and find your kernel.shmmni entry. Double that entry value in your /etc/sysctl.conf.