2

This is a new install in which nginx previously started and stopped normally. I believe this error arose after enabling server blocks which tested (nginx -t) successfully. I then tried to restart nginx and received this error:

nginx: [emerg] open() "/usr/share/nginx/on" failed (13: Permission denied)

The file "on" did not exist prior to the attempted restart. It was just created and is empty. When I restart php-fmp (successfully) and then try to restart nginx again, the error changes to:

nginx: [emerg] open() "/var/run/nginx.pid" failed (13: Permission denied)
nginx: configuration file /etc/nginx/nginx.conf test failed

But again, when I run nginx -t, the test is successful:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

I thought it might be a user issue but everything seems fine:

# ps -elf | grep nginx
5 S nginx     2774  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2775  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2776  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2777  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
5 S nginx     2778  2773  0  80   0 - 234152 skb_re 22:07 ?       00:00:00 php-fpm: pool www
0 R root      2940  2472  0  80   0 - 25811 -      22:18 pts/0    00:00:00 grep nginx

Even though Nginx is not running, the nginx.pid file remained so I deleted it. Doing so simply changed the error message back to:

nginx: [emerg] open() "/usr/share/nginx/on" failed (13: Permission denied) nginx: configuration file /etc/nginx/nginx.conf test failed.

This error was received regardless of how I tried to restart the system including $ sudo /etc/init.d/nginx restart and $ sudo /etc/init.d/nginx reload. I removed the empty "on" file which made no difference either. When I use $ getenforce, it returns Enforcing.

In response to @ǝɲǝɲbρɯͽ:

sudo grep -vR '^$\|^\s*\#' /etc/nginx/conf.d/* | grep -v ";" and sudo grep -vR '^$\|^\s*\#' /etc/nginx/nginx.conf* | grep -v ";" produced no missing semicolons.

The command sudo grep -ER "on|/usr/share" /etc/nginx/* printed out nearly every line of every file in nginx so I'm not sure what I would learn from that information. Incidentally, /usr/shar/nginx only contains that empty on file, nothing else.

sudo ausearch -m avc -ts recent -c nginx returned <no matches>

Needless to say, I am a newbie regarding server problems but I thought the command service --status-all (below) may produce useful information. Of course, we already know that a pid file exists even though nginx is dead, but what is the master (pid 1924) is running? Also, is there anything in the iptables that could be responsible for preventing nginx restart?

atd (pid  1995) is running...
auditd (pid  1405) is running...
crond (pid  1982) is running...
htcacheclean is stopped

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all      ::/0                 ::/0                state RELATED,ESTABLISHED
2    ACCEPT     icmpv6    ::/0                 ::/0
3    ACCEPT     all      ::/0                 ::/0
4    ACCEPT     udp      ::/0                 fe80::/64           state NEW udp dpt:546
5    ACCEPT     tcp      ::/0                 ::/0                state NEW tcp dpt:22
6    REJECT     all      ::/0                 ::/0                reject-with icmp6-adm-prohibited

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all      ::/0                 ::/0                reject-with icmp6-adm-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
6    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited
7    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3306
8    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

iscsi is stopped
iscsid is stopped
lvmetad is stopped
mdmonitor (pid  1445) is running...
multipathd is stopped
mysqld is stopped
netconsole module not loaded

Configured devices:
lo bond0 em1 em2

Currently active devices:
lo em1 em2 bond0

nginx dead but pid file exists
php-fpm is stopped
master (pid  1924) is running...
rdisc is stopped
restorecond is stopped
rsyslogd (pid  1425) is running...
sandbox is stopped
saslauthd is stopped
snmpd is stopped
snmptrapd is stopped
openssh-daemon (pid  1486) is running...
svnserve is stopped
9
  • 1
    Did you start nginx as root ? – user9517 Mar 22 '15 at 6:03
  • @Iain looks like John added some 'sudo' checks to the question. Glad you asked that. – ǝɲǝɲbρɯͽ Mar 22 '15 at 21:16
  • What happens if the configuration file you were editing is renamed? Does nginx start? – ǝɲǝɲbρɯͽ Mar 23 '15 at 3:15
  • Have you manually combed the configuration files for missing ; in the area you were changing? – ǝɲǝɲbρɯͽ Mar 23 '15 at 3:28
  • I have reviewed every file I have touched can find no errors and have removed every conf file - no change. I'm thinking that bricking the entire installation and starting over may be the only option. – John Andersen Mar 23 '15 at 14:21
5

/usr/share/nginx/... is where the default webroot is; you may have set something to be "on" and accidentally touched that file. Since your test works, it may also have been resolved, but when nginx crashes (or things like kill -9) it doesn't get rid of its pid file.

I don't have experience with php-fpm, but it appears your nginx master process isn't running. You can check for sure by:

$ ps axu | grep `cat /var/run/nginx.pid`

Those are backticks (`) and not apostrophes ('). If it isn't running, remove the pid file:

$ sudo rm /var/run/nginx.pid

and restart nginx. On many systems, that's:

$ sudo /etc/init.d/nginx restart

You don't want to do that under normal circumstances on a live site. There are better ways, including:

Once you restart nginx, you should see something like this:

$ ps axu | grep nginx

... worker threads
... 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf

You can also:

$ /etc/init.d/nginx status

nginx (pid  1111) is running...

**Edit: Diagnostics from comments**

1. Configuration

  • Look very carefully at your changes. nginx is fragile when it comes to missing semicolons. Almost every line must end with ;
    • If you missed some, nginx will process additional lines as if they're part of the same directive. For example (and it can be worse than this) these two lines are processed as the same directive: root /var/nginx/..... # no semicolon sendfile on;
    • It's not nginx's responsibility to be a really good syntax checker for you, so there's no guarantee that nginx will detect this error before attempting to start workers, where SELinux (correctly) stomps on the anomalous process action.

To make it easier to find those lines¹:

$ sudo grep -vR '^$\|^\s*\#' /etc/nginx/conf.d/* | grep -v ";"

Or lines with "on" or "/usr/share":

$ sudo grep -ER "on|/usr/share" /etc/nginx/conf.d/*

You probably need to fix one of those (don't add semicolons to {} lines), then:

$ sudo rm /usr/share/nginx/on /var/run/nginx.pid
$ sudo /etc/init.d/nginx restart

2. Metadata: Prove SELinux involvement (or not)

This is only for information. It crops up all the time in edits, and is a pivot for us / future visitors. Assuming nginx is still broken (you haven't fixed the configuration):

$ sudo /etc/init.d/nginx start
$ sudo ausearch -m avc -ts recent -c nginx

Though we're filtering on nginx (there may be other denials), SELinux is probably not indicated if you see:

<no matches>

SELinux is indicated with AVCs similar to:

type=AVC msg=audit(timestamp:123): avc:  denied  { getattr } for  pid=1234 comm="nginx" path="/usr/share/nginx/on" dev=sda ino=123456 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file

The problem is that at some point during startup, a file is created in a different context than nginx or its workers (which switch security contexts) are rightly not allowed to touch. When intended, contexts are visible by adding -Z to many commands, fixed with chcon (change context), and questions around this are easily found.

¹ grep credit

3
  • ps axu | grep cat /var/run/nginx.pid was not recognized. I did remove the pid file which simply changed the error message back to: nginx: [emerg] open() "/usr/share/nginx/on" failed (13: Permission denied) nginx: configuration file /etc/nginx/nginx.conf test failed. This error was received regardless of how I tried to restart the system including your suggestions. I removed the empty "on" which made no difference either. When I entered getenforce, it did return "Enforcing" – John Andersen Mar 22 '15 at 14:12
  • SELinux may be stomping on your restart, combined with either a bug/syntax error or unexpected side-effect of a desirable addition to your server configuration block(s). SELinux would actually be protecting you in the way it's designed to, implying it's the wrong approach to "setenforce permissive", start nginx, perform the requisite SELinux commands, and return to "enforcing" without identifying the potential bug. I'll add to the answer with diagnostic steps. – ǝɲǝɲbρɯͽ Mar 22 '15 at 19:10
  • Response to @ǝɲǝɲbρɯͽ is too long for comment submission - please see edit to original question. – John Andersen Mar 23 '15 at 2:29
4

You have a property that is expecting a path, but is provided a boolean value, e.g.

access_log on;

access_log is expecting a path, but it is provided a boolean value on. You can confirm this by attentively reading the error:

nginx: [emerg] open() "/usr/share/nginx/on" failed (13: Permission denied)

Notice the on relative to your nginx base path.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.