CentOS 6.6 with Nginx 1.6.2 - Suddenly can't restart nginx - nginx: [emerg] open() “/usr/share/nginx/on” failed (13: Permission denied)

Question

This is a new install in which nginx previously started and stopped normally. I believe this error arose after enabling server blocks which tested (nginx -t) successfully. I then tried to restart nginx and received this error:

nginx: [emerg] open() "/usr/share/nginx/on" failed (13: Permission denied)

The file "on" did not exist prior to the attempted restart. It was just created and is empty. When I restart php-fmp (successfully) and then try to restart nginx again, the error changes to:

nginx: [emerg] open() "/var/run/nginx.pid" failed (13: Permission denied)
nginx: configuration file /etc/nginx/nginx.conf test failed

But again, when I run nginx -t, the test is successful:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

I thought it might be a user issue but everything seems fine:

# ps -elf | grep nginx
5 S nginx     2774  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2775  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2776  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2777  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2778  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
0 R root      2940  2472  0  80   0 - 25811 -      22:18 pts/0    00:00:00 grep nginx

Even though Nginx is not running, the nginx.pid file remained so I deleted it. Doing so simply changed the error message back to:

nginx: [emerg] open() "/usr/share/nginx/on" failed (13: Permission denied) nginx: configuration file /etc/nginx/nginx.conf test failed.

This error was received regardless of how I tried to restart the system including $ sudo /etc/init.d/nginx restart and $ sudo /etc/init.d/nginx reload. I removed the empty "on" file which made no difference either. When I use $ getenforce, it returns Enforcing.

In response to @ǝɲǝɲbρɯͽ:

sudo grep -vR '^$\|^\s*\#' /etc/nginx/conf.d/* | grep -v ";" and sudo grep -vR '^$\|^\s*\#' /etc/nginx/nginx.conf* | grep -v ";" produced no missing semicolons.

The command sudo grep -ER "on|/usr/share" /etc/nginx/* printed out nearly every line of every file in nginx so I'm not sure what I would learn from that information. Incidentally, /usr/shar/nginx only contains that empty on file, nothing else.

sudo ausearch -m avc -ts recent -c nginx returned <no matches>

Needless to say, I am a newbie regarding server problems but I thought the command service --status-all (below) may produce useful information. Of course, we already know that a pid file exists even though nginx is dead, but what is the master (pid 1924) is running? Also, is there anything in the iptables that could be responsible for preventing nginx restart?

atd (pid  1995) is running...
auditd (pid  1405) is running...
crond (pid  1982) is running...
htcacheclean is stopped

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all      ::/0                 ::/0                state RELATED,ESTABLISHED
2    ACCEPT     icmpv6    ::/0                 ::/0
3    ACCEPT     all      ::/0                 ::/0
4    ACCEPT     udp      ::/0                 fe80::/64           state NEW udp dpt:546
5    ACCEPT     tcp      ::/0                 ::/0                state NEW tcp dpt:22
6    REJECT     all      ::/0                 ::/0                reject-with icmp6-adm-prohibited

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all      ::/0                 ::/0                reject-with icmp6-adm-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
6    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited
7    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3306
8    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

iscsi is stopped
iscsid is stopped
lvmetad is stopped
mdmonitor (pid  1445) is running...
multipathd is stopped
mysqld is stopped
netconsole module not loaded

Configured devices:
lo bond0 em1 em2

Currently active devices:
lo em1 em2 bond0

nginx dead but pid file exists
php-fpm is stopped
master (pid  1924) is running...
rdisc is stopped
restorecond is stopped
rsyslogd (pid  1425) is running...
sandbox is stopped
saslauthd is stopped
snmpd is stopped
snmptrapd is stopped
openssh-daemon (pid  1486) is running...
svnserve is stopped

Answer 1

/usr/share/nginx/... is where the default webroot is; you may have set something to be "on" and accidentally touched that file. Since your test works, it may also have been resolved, but when nginx crashes (or things like kill -9) it doesn't get rid of its pid file.

I don't have experience with php-fpm, but it appears your nginx master process isn't running. You can check for sure by:

$ ps axu | grep `cat /var/run/nginx.pid`

Those are backticks (`) and not apostrophes ('). If it isn't running, remove the pid file:

$ sudo rm /var/run/nginx.pid

and restart nginx. On many systems, that's:

$ sudo /etc/init.d/nginx restart

You don't want to do that under normal circumstances on a live site. There are better ways, including:

• $ sudo /etc/init.d/nginx reload
• graceful control options

Once you restart nginx, you should see something like this:

$ ps axu | grep nginx

... worker threads
... 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf

You can also:

$ /etc/init.d/nginx status

nginx (pid  1111) is running...

**Edit: Diagnostics from comments**

1. Configuration

• Look very carefully at your changes. nginx is fragile when it comes to missing semicolons. Almost every line must end with ;
  • If you missed some, nginx will process additional lines as if they're part of the same directive. For example (and it can be worse than this) these two lines are processed as the same directive: root /var/nginx/..... # no semicolon sendfile on;
  • It's not nginx's responsibility to be a really good syntax checker for you, so there's no guarantee that nginx will detect this error before attempting to start workers, where SELinux (correctly) stomps on the anomalous process action.

To make it easier to find those lines¹:

$ sudo grep -vR '^$\|^\s*\#' /etc/nginx/conf.d/* | grep -v ";"

Or lines with "on" or "/usr/share":

$ sudo grep -ER "on|/usr/share" /etc/nginx/conf.d/*

You probably need to fix one of those (don't add semicolons to {} lines), then:

$ sudo rm /usr/share/nginx/on /var/run/nginx.pid
$ sudo /etc/init.d/nginx restart

2. Metadata: Prove SELinux involvement (or not)

This is only for information. It crops up all the time in edits, and is a pivot for us / future visitors. Assuming nginx is still broken (you haven't fixed the configuration):

$ sudo /etc/init.d/nginx start
$ sudo ausearch -m avc -ts recent -c nginx

Though we're filtering on nginx (there may be other denials), SELinux is probably not indicated if you see:

<no matches>

SELinux is indicated with AVCs similar to:

type=AVC msg=audit(timestamp:123): avc:  denied  { getattr } for  pid=1234 comm="nginx" path="/usr/share/nginx/on" dev=sda ino=123456 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file

The problem is that at some point during startup, a file is created in a different context than nginx or its workers (which switch security contexts) are rightly not allowed to touch. When intended, contexts are visible by adding -Z to many commands, fixed with chcon (change context), and questions around this are easily found.

¹ grep credit

Answer 2

You have a property that is expecting a path, but is provided a boolean value, e.g.

access_log on;

access_log is expecting a path, but it is provided a boolean value on. You can confirm this by attentively reading the error:

nginx: [emerg] open() "/usr/share/nginx/on" failed (13: Permission denied)

Notice the on relative to your nginx base path.