I/O Speed doubts in apache2

Question

Here is my story:

Im working on improve develutils.com, and i develop in a Virtualbox with Ubuntu install, and 4Gb RAM and 2 Cores of 4. The web use a homemade MVC PHP Framework.

I have a local virtualhost point to the VBox, and launch tests to check the speed using apache test:

ab -n 2700 -c 90 -k -H "Accept-Encoding: gzip, deflate" http://develutils.ben/

This give me 38 Req/sec

Ok... this is to low Req/s, and during the test, the VBox HDD indicador get On always, and the 2 Cores are not in the 100%...

First i try to move the machine to another disk, and SSD, check the ab test and get 360 Req/s, ok, this number like me, but this give me a more questions...

I try to determine what its the problem with the I/O excess, and use the fantastic command:

inotifywait -mr /var/www/Develutils/

this spit me:

/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/system/temp/cache/ OPEN LANGUAGE_DICTIONARY_es.tmp
/var/www/WEBS/Develutils/system/temp/cache/ ACCESS LANGUAGE_DICTIONARY_es.tmp
/var/www/WEBS/Develutils/system/temp/cache/ CLOSE_NOWRITE,CLOSE LANGUAGE_DICTIONARY_es.tmp
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/templates/bts/css/ OPEN bootstrap-theme.min.css
/var/www/WEBS/Develutils/webroot/js/ OPEN jquery.min.js
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/templates/bts/css/ OPEN bootstrap.min.css
/var/www/WEBS/Develutils/webroot/templates/bts/css/ OPEN bootstrap-datetimepicker.css
/var/www/WEBS/Develutils/webroot/templates/bts/css/ CLOSE_NOWRITE,CLOSE bootstrap-theme.min.css
/var/www/WEBS/Develutils/webroot/templates/bts/css/ OPEN styles.css
/var/www/WEBS/Develutils/webroot/js/ CLOSE_NOWRITE,CLOSE jquery.min.js
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/templates/bts/css/ CLOSE_NOWRITE,CLOSE bootstrap.min.css
/var/www/WEBS/Develutils/webroot/templates/bts/css/ CLOSE_NOWRITE,CLOSE styles.css
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/templates/bts/css/ CLOSE_NOWRITE,CLOSE bootstrap-datetimepicker.css
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/js/ OPEN moment.js
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/js/ CLOSE_NOWRITE,CLOSE moment.js
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/js/ OPEN bootstrap.min.js
/var/www/WEBS/Develutils/webroot/js/ CLOSE_NOWRITE,CLOSE bootstrap.min.js
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/js/ OPEN handlebars.js
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/js/ CLOSE_NOWRITE,CLOSE handlebars.js
/var/www/WEBS/Develutils/webroot/js/ OPEN bootstrap-datetimepicker.js
/var/www/WEBS/Develutils/webroot/js/ OPEN web.js
/var/www/WEBS/Develutils/webroot/js/ CLOSE_NOWRITE,CLOSE web.js
/var/www/WEBS/Develutils/webroot/js/ CLOSE_NOWRITE,CLOSE bootstrap-datetimepicker.js
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/ OPEN .htaccess
/var/www/WEBS/Develutils/ ACCESS .htaccess
/var/www/WEBS/Develutils/ CLOSE_NOWRITE,CLOSE .htaccess
/var/www/WEBS/Develutils/webroot/ OPEN .htaccess
/var/www/WEBS/Develutils/webroot/ ACCESS .htaccess
/var/www/WEBS/Develutils/webroot/ CLOSE_NOWRITE,CLOSE .htaccess

I see this, and dont understand why the system its reading the .httaccess so many times, tooo many times, if the file its an existing file (.css, .js, images, etc) the .htaccess need to be avoided, but the apache call continuously

I have a base directory with this .htaccess :

<IfModule mod_rewrite.c>
    RewriteEngine on
    RewriteRule    ^$    webroot/    [L]
    RewriteRule    (.*) webroot/$1    [L]
</IfModule>

and in the webroot dir, i have this:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteRule ^(.*)$ index.php?url=$1 [L,QSA]
</IfModule>

I have many questions:

First, its normal this .htaccess accesses or i have any configured wrong?

Second, if this its "normal", apache dont have any way to cache this? If this its a "normal" behavior, i think in use an ramdisk, copy the CMS files (only files of CMS, and temp directories, not images or any big files) at the startup of the system, and then, launch apache using this directories to speed up the Req/s to insane numbers, but i dont know if this its a madness.

Any help and comment about this Req/s speed and I/O numers are wellcome.

Answer 1

that s why i put all in my apache site/mod/port conf, it is loaded when apache start then kept in memory, so no extra disk access.

.htacces will be read for each files accessed through apache, so even images and extra stuff for your web site.

By the way .htaccess are mostly use for shared server to prevent abusive user configuration, but let them have more possibility than default config.

So if you want to add .htaccess to you website configuration you can use the instruction in your /etc/apache2/site-enabled/{yoursiteconfig}

<Directory /var/www/WEBS/Develutils/>
RewriteEngine on
RewriteRule    ^$    webroot/    [L]
RewriteRule    (.*) webroot/$1    [L]
</Directory>

<Directory /var/www/WEBS/Develutils/webrrot/>
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.php?url=$1 [L,QSA]
</Directory>

and you have to do this for each .htaccess

If you want to reduce even more hard disk access, you can add some cache and remove some useless logs in your configuration

for exemple remove images, css and js from logs

SetEnvIf Request_URI "\.jpg$|\.jpeg$|\.gif$|\.png$|\.ico|\.icon|\.css$|\.js$" dontlog

<virtualhost {IP}:{PORT}>
...
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog
...
</virtualhost>

cache file to prevent access file

#Remove ETags
Header unset ETag
FileETag none

#MOD_DEFLATE COMPRESSION
SetOutputFilter DEFLATE
AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/x-javascript application/x-httpd-php

#DEFLATE NOT COMPATIBLE BROWERS
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

#NOT CHACHING IF ALREADY CACHED
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip

#PROXY CHECK CONTENT
Header append Vary User-Agent env=!dont-vary

#CACHE-CONTROL HEADERS
<IfModule mod_headers.c>
<FilesMatch "\.(ico|jpe?g|png|gif|swf|gz|ttf)$">
Header set Cache-Control "max-age=2797200, public"
</FilesMatch>
<FilesMatch "\.(css)$">
Header set Cache-Control "max-age=2797200, public"
</FilesMatch>
<FilesMatch "\.(js)$">
Header set Cache-Control "max-age=2797200, private"
</FilesMatch>
<filesMatch "\.(html|htm)$">
Header set Cache-Control "max-age=86400, public"
</filesMatch>
#Disable caching for scripts and other dynamic files
<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
Header unset Cache-Control
</FilesMatch>
</IfModule>

#EXPIRES HEADERS
<IfModule mod_expires.c>
ExpiresActive On
#Images
ExpiresByType image/jpg "access plus 1 year"
ExpiresByType image/jpeg "access plus 1 year"
ExpiresByType image/gif "access plus 1 year"
ExpiresByType image/png "access plus 1 year"
AddType image/x-icon .ico
ExpiresByType image/ico "access plus 1 year"
ExpiresByType image/icon "access plus 1 year"
ExpiresByType image/x-icon "access plus 1 year"
#Elements
ExpiresByType application/xhtml+xml "access plus 1 week"
ExpiresByType text/html "access plus 1 week"
ExpiresByType text/css "access plus 1 week"
ExpiresByType application/javascript "access plus 1 week"
ExpiresByType text/x-javascript "access plus 1 week"
#Others
ExpiresDefault "access plus 1 month"
</IfModule>

Answer 2

Unless you absolutely have to (e.g. you are on shared hosting and this is not the site for you) then you should move everything out of the .htaccess and put it in the server config files.

If you have enabled .htaccess files using AllowOverride then Apache httpd has to check every directory in the path to a resource for .htaccess files for every request. The .htaccess files found are opened and read for every request.

It can't be said often enough - don't use .htaccess files unless there is no other way of doing what you want. Normally you can do everything you need to in a server config file.