Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a problem with high server load (load average from 10 to 100+) when there is a lot of web traffic. There are also noticable delays when reading/writing files. Machine is a Apache web server. Web page is using PHP and is accessing/opening some small (max 30kB) files (config, blacklist, file from cache directory).

Im trying to trace the couse of this issue for couple of days now. This is what I found out:

  • CPU load is below 10%
  • load is coused by long disk response. top shows %wa parameter (iowait) at 20-100%
  • iostat: pastebin
  • after some testing it appears that the problem is one, specific fopen() function in my PHP script, accessing cache files
  • if I delete the line with this specific fopen(), server load is going down fast, to 0.00-0.50 values, iowait is 0%

Some informations about server:

  • xeon e3-1245v2, 32 gb ram, 2x 2TB SATA3 drives with software RAID 1 (mirroring), filesystem ext4 (with dir-index enabled and noatime in fstab)
  • cache directory size is about 500GB
    • it has about 16-17 milion files, each 20-30KB
    • filenames are MD5 hashes
    • directory structure is (example): cacheDir/d4/1d/d41d8cd98f00b204e9800998ecf8427e.dat
    • so there is 256 directories in cacheDir/ (1st level) and 256 2nd level directories (in each 1st level directory)
    • so, on average, there is about 200-300 files per 2nd level directory
  • PHP is fopen'ing cache files with absolute paths
  • disks were checked: no errors/warnings
  • e2fsck -Df (optimize directories in filesystem) result is here
  • after e2fsck -Df there was no difference in server load

I thought, that such cache directory organisation would be good for optimal disk performance. Apparently I was wrong..

What would you recomend to fix this high load/slow disk reponse issue?

share|improve this question
1  
Buy faster disks. Look at alternate architectures. Is implementing a cache within PHP really the right thing? Why not setup a caching server specifically designed for caching. Maybe use memcache instead of the filesystem. Stop caching, do you really need to cache, are you sure you were not prematurely optimizing? –  Zoredache May 29 '13 at 1:32
    
600GB of memcache? Seriously? –  gibson May 29 '13 at 6:40
    
This cache is a result of other program many months work (getting some online informations). If it wasnt cached, I would have to download it every time I run the script. –  gibson May 29 '13 at 6:47
    
Note how that was the third item in a list? Is your cache usage random, or could you simply put the most frequently used 10% in memory, and hit disk for the less common stuff? My first suggestion still stands buy faster storage. Do you really think you are going to sustain high I/O on a two SATA drives in a RAID1? –  Zoredache May 29 '13 at 6:49
    
Cache usage is totally random, so memcache is not possible. The problem is that one, specific directory (cache directory). If I replace this fopen with 20 fopens but accessing different files (not the ones in cache directory), then server load is normal. Accessing cache directory makes disks respond longer than accessing any other files... –  gibson May 29 '13 at 6:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.