4

Every time NGINX is restarted, two semaphore arrays are left behind. Given semaphores are a limited, shared resource--the server is constantly careening, albeit slowly, towards a quota limit failure.

The semaphores left behind on restart are owned by "4294967295" (-1) with permissions of 600. Short of a reboot, they do not want to be removed by traditional means, e.g.:

$ sudo ipcrm -s 5111851  
cannot remove id 5111851 (Operation not permitted)

Has anyone dealt with this issue before? Obviously, it would be fantastic to find a way to stop orphaning the arrays through an NGINX configuration, but short of that I'm all for modifying the init.d script to clean up the semaphore arrays left behind, I just can't seem to remove them with the standart ipc* utilities.

Is there a way, short of a reboot, to remove these semaphore arrays assigned to a non-existent user?

$ sudo ipcs -st
------ Semaphore Operation/Change Times --------
semid    owner       last-op   last-changed
...
9568370  4294967295  Not set   Fri Feb 13 03:38:02 2015 <-- logrotate
9601139  4294967295  Not set   Fri Feb 13 03:38:02 2015
9633908  4294967295  Not set   Sat Feb 14 03:31:06 2015
9666677  4294967295  Not set   Sat Feb 14 03:31:06 2015
9764982  4294967295  Not set   Sun Feb 15 03:47:06 2015
9797751  4294967295  Not set   Sun Feb 15 03:47:06 2015
9830520  4294967295  Not set   Mon Feb 16 03:37:02 2015
9863289  4294967295  Not set   Mon Feb 16 03:37:02 2015
9994362  4294967295  Not set   Mon Feb 16 11:34:08 2015 <-- manual restart
10027131 4294967295  Not set   Mon Feb 16 11:34:08 2015
10125436 4294967295  Not set   Mon Feb 16 13:47:25 2015
10158205 4294967295  Not set   Mon Feb 16 13:47:25 2015
10256510 4294967295  Not set   Mon Feb 16 13:52:47 2015
10289279 4294967295  Not set   Mon Feb 16 13:52:47 2015


$ nginx -V
nginx version: nginx/1.6.2
TLS SNI support enabled
configure arguments: 
--prefix=/usr/share/nginx 
--sbin-path=/usr/sbin/nginx 
--conf-path=/etc/nginx/nginx.conf 
--error-log-path=/var/log/nginx/error.log 
--http-log-path=/var/log/nginx/access.log 
--http-client-body-temp-path=/var/lib/nginx/tmp/client_body 
--http-proxy-temp-path=/var/lib/nginx/tmp/proxy 
--http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi 
--http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi 
--http-scgi-temp-path=/var/lib/nginx/tmp/scgi 
--pid-path=/var/run/nginx.pid 
--lock-path=/var/lock/subsys/nginx 
--user=nginx 
--group=nginx 
--with-file-aio 
--with-ipv6 
--with-http_ssl_module 
--with-http_spdy_module 
--with-http_realip_module 
--with-http_addition_module 
--with-http_xslt_module 
--with-http_image_filter_module 
--with-http_geoip_module 
--with-http_sub_module 
--with-http_dav_module 
--with-http_flv_module 
--with-http_mp4_module 
--with-http_gunzip_module 
--with-http_gzip_static_module 
--with-http_random_index_module 
--with-http_secure_link_module 
--with-http_degradation_module 
--with-http_stub_status_module 
--with-http_perl_module 
--with-mail 
--with-mail_ssl_module 
--with-pcre --with-debug 
--add-module=/builddir/build/BUILD/nginx-1.6.2/modsecurity-2.8.0/nginx/modsecurity 
--add-module=/builddir/build/BUILD/nginx-1.6.2/ngx_cache_purge-2.1 
--with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector 
--param=ssp-buffer-size=4 -m64 -mtune=generic' 
--with-ld-opt=-Wl,-E

CentOS 6, NGINX 1.6.2.

2

You might want to upgrade to nginx 1.8 - this fixed the leaked semaphores issue for me (at least at first glance)

| improve this answer | |
1

I didn't find the source of the problem, but here is a workaround.

In /etc/init.d/nginx, locate the stop() function and add the following line before return $retval:

ipcs -s | grep "4294967295" | awk ' { print $2 } ' | xargs -I  % ipcrm -s %

nginx script change

With this change, sudo service nginx restart should now clear the old semaphore arrays.

What about not being able to delete the semaphores in the first place? This had to do with getting confused by sudo privileges not cascading across the entirety of the command. Switching to root encountered no such problems.

[user@example ~]$ sudo ipcs -s | grep "4294967295" | awk ' { print $2 } ' | xargs -I  % ipcrm -s %
ipcrm: permission denied for id (819214)
ipcrm: permission denied for id (851983)
ipcrm: permission denied for id (1114130)
[user@example ~]$ sudo -i
[root@example ~]# ipcs -s | grep "4294967295" | awk ' { print $2 } ' | xargs -I  % ipcrm -s %
[root@example ~]# 
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.