Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a setup where a Drupal 7 site with the following setup - a VMware ESXi 4.1 host server running a web vm and an NFS VM. The web VM is using Apache and mod_php. The site is still in development thus we have to turn off all forms of caching due to the frequently-updated files.

Each page request takes around 15-20 seconds to complete. Profiling the PHP code shows that the vast majority of time (normally over 90%) is taking by all the is_dir(), is_file() function calls that load up the modules.

I've increased PHP's realpath cache size to several megs and an strace shows that the lstat calls then drop from over 200 to around 6 and stat() decreases a bit (around 600 calls). However, while this has shaved off quite a bit of time, I am simply unable to break past the 10 second per request barrier for the most demanding page.

Is there a way to get better performance out of this setup that doesn't involve caching?

Edit: MySQL is not the issue, query caching means that requests take a second at most to complete.

Configs and stats:

VM Host: single quad-core Xeon CPU

VMs:

web - Centos 6 64bt, 2.5GB RAM, normal CPU/HD prioritisation (2 cores) nfs - Centos 6 64bt, 2GB RAM, normal CPU priority (4 cores), high HD priority

PHP: 32M realpath cache size (it's this high for testing purposes)

NFS:

~]# egrep -v '#|^$' /etc/nfsmount.conf 
[ NFSMount_Global_Options ]
 Defaultvers=4
 Ac=False
 Rsize=32k
 Wsize=32k
 Bsize=32k

Reading speeds via NFS are not an issue a dd of a 100M test file using 32k blocks returns:

3200+0 records in
3200+0 records out
104857600 bytes (105 MB) copied, 1.84984 s, 56.7 MB/s

real    0m1.857s
user    0m0.007s
sys 0m0.330s 

Strace on Apache process with empty realpath cache:

    % time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 50.78    1.157452         337      3434        28 stat
 32.58    0.742656         628      1182       425 open
  9.29    0.211788         762       278         1 lstat
  3.17    0.072322           0    237865           write
  2.45    0.055839         490       114        13 access
  0.45    0.010262          43       237           brk
  0.34    0.007725          10       811        74 read
  0.28    0.006340           9       679           fstat
  0.22    0.005069          18       281           poll
  0.20    0.004533           6       698           getdents
  0.09    0.001960          10       190           mmap
  0.05    0.001065          14        74           accept4
  0.04    0.001000         333         3           chdir
  0.03    0.000750           4       190           munmap
  0.01    0.000339           0       836           close
  0.01    0.000247           3        75           writev
  0.00    0.000068           0       611           fcntl
  0.00    0.000063           1        77           shutdown
  0.00    0.000000           0         1           lseek
  0.00    0.000000           0         5           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         3           setitimer
  0.00    0.000000           0         5           socket
  0.00    0.000000           0         5         5 connect
  0.00    0.000000           0        74           getsockname
  0.00    0.000000           0        15           setsockopt
  0.00    0.000000           0         5           getcwd
  0.00    0.000000           0         1           futex
------ ----------- ----------- --------- --------- ----------------

Strace after realpaths are cached

    % time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 60.14    1.371006         484      2831        28 stat
 31.79    0.724705         627      1155       425 open
  3.53    0.080354           0    237865           write
  2.65    0.060433         530       114        13 access
  0.43    0.009913          99       100           brk
  0.38    0.008730          11       804        74 read
  0.35    0.007910          12       675           fstat
  0.30    0.006775          10       654           getdents
  0.13    0.003065          11       281           poll
  0.09    0.002000         333         6         1 lstat
  0.07    0.001545           2       807           close
  0.05    0.001063          14        74           accept4
  0.04    0.001000           6       179           mmap
  0.02    0.000404           2       179           munmap
  0.01    0.000271           4        75           writev
  0.01    0.000212           0       611           fcntl
  0.01    0.000129           2        77           shutdown
  0.00    0.000022           0        74           getsockname
  0.00    0.000000           0         1           lseek
  0.00    0.000000           0         5           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         3           setitimer
  0.00    0.000000           0         3           socket
  0.00    0.000000           0         3         3 connect
  0.00    0.000000           0        15           setsockopt
  0.00    0.000000           0         5           getcwd
  0.00    0.000000           0         3           chdir
------ ----------- ----------- --------- --------- ----------------

Mount:

nfs.xxx.xxx.xxx:/path/to/website/files on /path/to/website/files type nfs (rw,hard,intr,noac,vers=4,addr=xx.xx.xx.xx,clientaddr=xx.xx.xx.xx)

Any help is, naturally, appreciated.

share|improve this question
    
What's MySQL saying (assuming you're using MySQL)? Can you try enabling the slow queries log? –  gravyface Sep 3 '12 at 20:52
    
Also, what's the purpose of having a virtual NFS server? i.e. why not just allocate another virtual disk(s) to the web VM? –  gravyface Sep 3 '12 at 20:54
    
@gravyface MySQL was tested and found to not be the issue, query caching means that requests take a second at most to complete. In addition, PHP profiling showed that the bottleneck was not MySQL. We'll be adding more web servers and a load balancer later on, hence the NFS share. –  Marcus Sep 3 '12 at 21:38
    
For implementing NFS for multiserver Drupal Setup, see this [link][1] [1]: stackoverflow.com/questions/16035307/nfs-drupal-implementation –  Ksindev Apr 22 '13 at 14:10
    
For implementing NFS for multiserver Drupal Setup, see this [link][1] [1]: stackoverflow.com/questions/16035307/nfs-drupal-implementation –  Ksindev Apr 22 '13 at 14:11
add comment

2 Answers 2

To be frank, Drupal on NFS is a real pig. At most, you want to share the "files" director{y,ies} via NFS or something like gluster. The problem with running the DocRoot on NFS is that, added up, all of the lstat(2) and access(2) calls are killer, let alone the getdents(2)s and friends that you'll see in module directories. Something like APC will significantly help the actual read(2) times, as well as removing the compilation delays, but PHP will still do lstat(2) and access(2) on every file. To further speed things up, you can set apc.stat=0, but that won't help you if, as you said, you're constantly changing PHP files, unless you're willing to restart Apache (or manually clear the APC cache via apc.php) every time you make such a change.

Best practices recommend storing the DocRoot on either a dedicated, optimized device (such as a SAN), or separately on each webhead. The "files" directory should generally be shared via gluster/nfs/etc., but as an alternative, you can also periodically rsync it between servers, depending on the use case and whether the LB in front supports sticky sessions. You can also eliminate the files directory altogether by use of a CDN or a service like Amazon's S3 or BlackMesh's Swift.

A hosting provider with detailed, specialized knowledge of Drupal can help you with some of these architectural concerns; you may wish to contact Acquia or BlackMesh (I work for the latter). I don't know if Acquia does, but I know BlackMesh also offers off-site assistance where we work with your existing hosting provider or on-site hosting to optimize the solution for Drupal.

Best of luck with your site!

share|improve this answer
    
We moved to a GFS2 cluster, solved the I/O issues nicely :) –  Marcus Oct 26 '12 at 16:11
add comment

Mounting with noac in general is a performance killer.

Why are you using it?

EDIT: I see above in a comment that you say that you are adding more web servers and a load balancer later. As BMDan suggested you should look into a cluster file system (e.g. OCFS2) if your application requires that you use noac.

Cheers

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.