Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have this architecture on Amazon EC2, one NFS server and one NFS client, on the client I am serving PHP and Django websites (nginx, uwsgi, php-fpm) and they work perfectly.

I am experiencing an issue when I spin up another NFS client instance base on the image of the first NFS, when I load a PHP site(wordpress), I start to get timeouts on the browser. And when I turn off one of the NFS client instance and things start to work again. I am suspecting there is a file lock problem, I have tried all night, been searching on google and tried the nolock option but I just couldn't solve it.

What I saw was, the NFS mounted folders seemed fine and showing all the files, but when I attach the second EC2 instance, the NFS server and both clients started to get high load average, with very low CPU usage.

Here is the content from /etc/export on the NFS server

/export/www 172.0.0.0/8(rw,async,no_subtree_check)
/export/config/nginx/sites-available 172.0.0.0/8(rw,async,no_subtree_check)
/export/config/nginx/sites-enabled 172.0.0.0/8(rw,async,no_subtree_check)
/export/config/uwsgi/apps-available 172.0.0.0/8(rw,async,no_subtree_check)
/export/config/uwsgi/apps-enabled 172.0.0.0/8(rw,async,no_subtree_check)

And here is the content from /etc/fstab on the NFS clients

LABEL=cloudimg-rootfs   /        ext4   defaults        0 0
/dev/xvdb       /mnt    auto    defaults,nobootwait,comment=cloudconfig 0       2
#172.31.0.62:/export/www        /var/www        nfs     auto    0 0
172.31.0.62:/export/www /var/www        nfs4    rw,noatime,nodev,async,hard,intr,rsize=32768,wsize=32768 0 2
172.31.0.62:/export/config/nginx/sites-available /etc/nginx/sites-available     nfs4    rw,noatime,nodev,async,hard,intr,rsize=32768,wsize=32768 0 2
172.31.0.62:/export/config/nginx/sites-enabled  /etc/nginx/sites-enabled        nfs4    rw,noatime,nodev,async,hard,intr,rsize=32768,wsize=32768 0 2
172.31.0.62:/export/config/uwsgi/apps-available /etc/uwsgi/apps-available       nfs4    rw,noatime,nodev,async,hard,intr,rsize=32768,wsize=32768 0 2
172.31.0.62:/export/config/uwsgi/apps-enabled /etc/uwsgi/apps-enabled   nfs4    rw,noatime,nodev,async,hard,intr,rsize=32768,wsize=32768 0 2

Thanks heaps.

UPDATE:

Looks like it's not only related to PHP FPM, I can even replicate this by refreshing a static html page. Whenever the server starting to get stuck, running nfsstat shows calls and authrefrsh goes up very quickly.

share|improve this question
1  
NFS is a horrible way to solve the root problem that you're dealing with. Much better to use build automation to solve this problem, perhaps using Git. –  Magellan Nov 17 '13 at 0:22
    
while I can use git, I still need to sync the user files, I am in no position to modify all the PHP sites –  James Lin Nov 17 '13 at 0:34
add comment

2 Answers 2

"Timeouts on the browser" in my opinion do not really prove that the NFS is the cause of the problem. You didn't explained your debugging process in enough detail. Try to access the files (you are requesting from the browser) directly from the NFS clients command line. In this kind of situation I would be more suspicious of nginx misconfiguration or some networking misconfiguration that eventually causes "Timeouts on the browser".
If eventually you will find out that the problem is really caused by NFS and you will not come up with the solution in reasonable time, I would suggest switching to GlusterFS.

Update 1: Check out I/O statistics on both client and server "iostat -xm 20". Pay attention to CPU iowait and NFS partiton MB read/write statistics. You want to find out if NFS is slow because of r/w load or for some other reason. If there is a lot of I/O, find out who generates it by executing "iotop" command.

share|improve this answer
    
Hi I have updated my question. Thanks for answering. –  James Lin Nov 17 '13 at 6:47
add comment

There was a problem with NFSv4 on Amazon EC2, I don't know why, but the sys admin I hired told me he has heard NFS problems on EC2 as well. What he discovered was NFS concurrent read speed was very very slow, something like 20MB over 150 seconds, while write speed was pretty ok @ 7mb/s

So the real fix was to drop back to NFSv3 and everything started to work fine again.

Hope this will help someone who is having similar issue.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.