Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am debugging a problem where MoinMoin on CentOS is throwing a permissions error, but I can't track down where the problematic file / directory is.

I ran strace -vp <pid> on the apache pid; when I have the problem I see this:

epoll_wait(10, {{EPOLLIN, {u32=3487534344, u64=140367313734920}}}, 2, 10000) = 1
accept4(6, {sa_family=AF_INET6, sin6_port=htons(52621), inet_pton
    (AF_INET6, "::ffff:105.193.30.91", &sin6_addr), sin6_flowinfo=0, 
    sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 11
## Later on...
read(7, 0x7fffa658ad7f, 1)              = -1 EAGAIN (Resource temporarily 
    unavailable)

However, since apache is already running, I see no corresponding open() on the file referred to as 7; thus I see the permissions problem, but I still don't know which file is the problem.

I know I could try to catch all the file opens when I respawn apache, but I'm hoping there is a way to map file 7 to a real filename... is there a way to do this?

EDIT 1:

Using @lain's guidance, I ran lsof | grep 266474069, but the results are unclear...

[root@lnxlmf moin]# ls -la /proc/9707/fd/7
lr-x------. 1 root root 64 Aug 28 15:39 /proc/9707/fd/7 -> pipe:[266474069]
[root@lnxlmf moin]#

[root@lnxlmf moin]# lsof | grep 266474069
httpd      9703      root    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9703      root    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9705    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9705    apache    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9706    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9706    apache    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9707    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9707    apache    8w     FIFO                0,8          0t0  266474069 pipe
httpd      9733    apache    7r     FIFO                0,8          0t0  266474069 pipe
httpd      9733    apache    8w     FIFO                0,8          0t0  266474069 pipe
[root@lnxlmf moin]#

I see that this is a FIFO pipe, but what does this mean for my system configuration? How can I trace the root cause for the EAGAIN issue?

EDIT 2:

Thanks to @Alan Curry, running strace -fp <pid_of_wsgi_proc> seems to get me a bit farther...

[pid  9731] stat("/opt/moin/share/moin/wikiconfig.py", {st_mode=S_IFREG|0770, st_size=6463, ...}) = 0
[pid  9731] stat("/opt/moin/share/moin/data/pages", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
[pid  9731] access("/opt/moin/share/moin/data/pages", R_OK|W_OK|X_OK) = -1 EACCES (Permission denied)

However, both apache and wsgi run as the apache user...

[root@lnxlmf moin]# ps auxw | grep -E "apache|wsgi"
apache   10187  0.0  0.1 373488  5884 ?        Sl   17:18   0:00 moin_http_wsgi
apache   10188  0.0  0.1 373488  5884 ?        Sl   17:18   0:00 moin_https_wsgi
apache   10189  0.0  0.1 185096  5824 ?        S    17:18   0:00 /usr/sbin/httpd
root     10243  0.0  0.0 103240   848 pts/1    S+   17:21   0:00 grep -E apache|wsgi
[root@lnxlmf moin]#

Yet, when I run the following commands and restart apache, I still can't fix the problem...

[root@lnxlmf moin]# pwd
/opt/moin/share/moin
[root@lnxlmf moin]# chown -R apache:apache data/
[root@lnxlmf moin]# sudo chmod -R ug+rwx data/
[root@lnxlmf moin]# sudo chmod -R o-rwx data/

I am using this in my wiki http config:

<VirtualHost *:443>
  ServerName netwiki.foo.com
  DocumentRoot /opt/moin/share/moin
  WSGIScriptAlias / /opt/moin/share/moin/server/moin.wsgi
  WSGIDaemonProcess moin_https display-name=moin_https_wsgi \
      user=apache group=apache \
      processes=1 threads=10 maximum-requests=1000 umask=0007
  WSGIProcessGroup moin_https
  WSGIApplicationGroup %{GLOBAL}

  # Generate with...
  # openssl req -new -x509 -days 365 -nodes -out netwiki.pem -keyout netwiki.key
  SSLEngine on
  SSLCertificateFile /etc/httpd/ssl/netwiki.pem
  SSLCertificateKeyFile /etc/httpd/ssl/netwiki.key
</VirtualHost>
share|improve this question
1  
It should be noted that EAGAIN is not a permissions issue. It's just a "buffer is empty right now" indicator. It appears quite often in the strace output of programs that are working perfectly. How sure are you that this particular EAGAIN is connected to your problem? –  Alan Curry Aug 28 '12 at 22:08
    
Well, you're probably right... bad call on my part. I will run strace again and see if I can nail this down. –  Mike Pennington Aug 28 '12 at 22:12
    
Since you're looking at an access() failure, I wonder if the real uid doesn't match the effective uid. Look at the Uid: line in the process's /proc/PID/status and if it has different numbers in the different fields, that'll be a clue. –  Alan Curry Aug 28 '12 at 22:59
    
Well, everything is showing up as euid of 48... and grep 48 /etc/passwd yields apache:x:48:48:Apache:/var/www:/sbin/nologin... so I think I'm ok there... the apache user and group have rwx on the whole directory tree –  Mike Pennington Aug 28 '12 at 23:08
add comment

3 Answers

Take a look in /proc/<PID>/fd which should list all of the open files that PID has open.


On my CentOS system fd 7 is

lrwx------. 1 root root 64 Aug 28 22:01 7 -> socket:[1872522]

I can then use netstat -ane | grep 1872522 to get

tcp    0  0 :::443              :::*               LISTEN      0          1872522

You can use

lsof | grep 266474069

to get information on the pipe.

share|improve this answer
    
Symlinked to a pipe??? /proc/9707/fd/7 -> pipe:[266474069] –  Mike Pennington Aug 28 '12 at 20:40
    
Uh, lain was faster, sorry. Yes, a file descriptor can also point to a pipe. –  Valentin Aug 28 '12 at 20:41
    
@MikePennington: on my system it's a socket lrwx------. 1 root root 64 Aug 28 21:34 7 -> socket:[1869276] –  Iain Aug 28 '12 at 20:42
    
+1, good info... please see my update –  Mike Pennington Aug 28 '12 at 22:05
add comment

Looking at my little VPS, I can determine the fd number the following way:

 ll /proc/17684/fd/ |colrm 1 46

0 -> /dev/null
1 -> /dev/null
10 -> /var/www/vhosts/censored.xenuser.org/statistics/logs/error_log
11 -> /var/www/vhosts/censored.de/statistics/logs/error_log
12 -> /var/www/vhosts/censored.org/statistics/logs/error_log 
13 -> /var/www/vhosts/xenuser.org/statistics/logs/error_log
14 -> /var/log/apache2/access.log

[and so on, where 17684 is the PID of the process I straced earlier]

share|improve this answer
    
+1, please see my update –  Mike Pennington Aug 28 '12 at 22:06
add comment
up vote 0 down vote accepted

The problem was that I had SELINUX=enforcing in /etc/selinux/config.

After I set SELINUX=permissive, SELINUXTYPE=targeted, and rebooted wsgi can access all files correctly.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.