I have a problem with high server load (load average from 10 to 100+) when there is a lot of web traffic. There are also noticable delays when reading/writing files. Machine is a Apache web server. Web page is using PHP and is accessing/opening some small (max 30kB) files (config, blacklist, file from cache directory).
Im trying to trace the couse of this issue for couple of days now. This is what I found out:
- CPU load is below 10%
- load is coused by long disk response.
%waparameter (iowait) at 20-100%
- iostat: pastebin
- after some testing it appears that the problem is one, specific fopen() function in my PHP script, accessing cache files
- if I delete the line with this specific fopen(), server load is going down fast, to 0.00-0.50 values, iowait is 0%
Some informations about server:
- xeon e3-1245v2, 32 gb ram, 2x 2TB SATA3 drives with software RAID 1 (mirroring), filesystem ext4 (with
- cache directory size is about 500GB
- it has about 16-17 milion files, each 20-30KB
- filenames are MD5 hashes
- directory structure is (example):
- so there is 256 directories in
cacheDir/(1st level) and 256 2nd level directories (in each 1st level directory)
- so, on average, there is about 200-300 files per 2nd level directory
- PHP is
fopen'ing cache files with absolute paths
- disks were checked: no errors/warnings
e2fsck -Df(optimize directories in filesystem) result is here
e2fsck -Dfthere was no difference in server load
I thought, that such cache directory organisation would be good for optimal disk performance. Apparently I was wrong..
What would you recomend to fix this high load/slow disk reponse issue?