Apache is using 100% CPU even with one user
Apache (or should I say PHP through Apache?) is using 100% CPU for few seconds every time somebody search bugs at our bug tracker or just reload the search result page or just sort results.
That causes a very slow page loading. It occurs even if I access our bug tracker from server machine itself.
My question is what seems to be the bottleneck here (apache config, php config, mysql config, cache, php modules or something else?) and what should I try to do to fix it?
Software we are using (downloaded few months ago and not updated after this):
Kubuntu: 14.04.3 LTS
PHP: 5.5.9 (I guess running as apache mod? CGI? Not seems to be fastCGI)
apache2：2.4.7 (PREFORK)
mysql： 5.5.44
The Bug Genie (bug tracker software): 4.1.0
Here is image of Firebug.
“POST paginated” is the heavy one, it takes 1s to receive 4.0M but it have to wait 7s before it can receive.
Here is an image of header of “POST paginated”.
And as about server it is doing like this.
It seems that request of “paginated” required 7-8 seconds to process.
First wave is when I search and display about 250 issues (“POST paginated”) and second one is when I take a small action like going to one issue page. You can see that even small action takes much CPU.
I don't know why so much memory are used (1.6G of 2.0G) even if I don`t access the bug tracker, but anyway free memory seems to be enough.
It does not seem that there are I/O problems ("%iowait" is low, and "await" can be bigger than "svctm" sometimes but only sometimes)
And it does not seem that there are much swapping going around.
Here is our settings (the part that may matter for my guess).
Apache2.conf (not using httpd.conf)
ServerName localhost
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
HostnameLookups Off
<Directory /var/www/html/bugs2/>
Options Indexes MultiViews
AllowOverride All
Order allow,deny
Allow from all
</Directory>
AccessFileName .htaccess
Changing ServerName to 127.0.0.1 did not help.
I also try to add below text to apache.conf but it would not change anything. I guess that is because there is already mpm_prefork.conf which is quite close (only MaxRequestWorkers is 150).
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 20
MaxRequestsPerChild 0
</IfModule>
.htaccess
<IfModule mod_php5.c>
php_flag magic_quotes_gpc off
php_flag register_globals off
</IfModule>
Options +FollowSymlinks
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /bugs2/public/
RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^(.*)$ index.php?url=$1 [NC,QSA,L]
RewriteCond %{REQUEST_URI} \..+$/
RewriteCond %{REQUEST_URI} !\.(html|wsdl|json|xml)$
RewriteRule .* - [L]
RewriteRule ^(.*)$ index.php?url=$1 [NC,QSA,L]
RedirectMatch 403 ^/\.svn(/|$)
</IfModule>
Few days ago I also putted below mod_expires, mod_headers, mod_deflate, mod_gzip into .htaccess, but I couldn’t see any difference in CPU usage or loading speed before/after.
<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 24 hours"
ExpiresByType application/json "access plus 1 months"
ExpiresByType image/jpg "access plus 1 months"
ExpiresByType image/gif "access plus 1 months"
ExpiresByType image/jpeg "access plus 1 months"
ExpiresByType image/png "access plus 1 months"
ExpiresByType text/css "access plus 1 months"
ExpiresByType text/javascript "access plus 1 months"
ExpiresByType application/javascript "access plus 1 months"
ExpiresByType application/x-shockwave-flash "access plus 1 months"
ExpiresByType application/x-javascript "access plus 1 months"
ExpiresByType application/pdf "access plus 1 months"
ExpiresByType image/x-icon "access plus 1 year"
</IfModule>
<ifModule mod_headers.c>
<filesMatch "\\.(ico|pdf|flv|jpg|jpeg|png|gif|swf)$">
Header set Cache-Control "max-age=2592000, public, no-transform"
</filesMatch>
<filesMatch "\\.(css)$">
Header set Cache-Control "max-age=604800, public"
</filesMatch>
<filesMatch "\\.(js)$">
Header set Cache-Control "max-age=216000, private"
</filesMatch>
<filesMatch "\\.(xml|txt)$">
Header set Cache-Control "max-age=216000, public, must-revalidate"
</filesMatch>
<filesMatch "\\.(html|htm|php)$">
Header set Cache-Control "max-age=1, private, must-revalidate"
</filesMatch>
Header unset ETag
Header unset Last-Modified
</ifModule>
<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|ico)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI _\.utxt$ no-gzip
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/atom_xml
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/x-httpd-php
AddOutputFilterByType DEFLATE application/json
</IfModule>
<ifModule mod_gzip.c>
mod_gzip_on Yes
mod_gzip_dechunk Yes
mod_gzip_item_include file .(html?|txt|css|js|php|pl|json)$
mod_gzip_item_include handler ^cgi-script$
mod_gzip_item_include mime ^text/.*
mod_gzip_item_include mime ^application/x-javascript.*
mod_gzip_item_exclude mime ^image/.*
mod_gzip_item_exclude rspheader ^Content-Encoding:.*gzip.*
</ifModule>
I also tried using filesMatch instead of ExpiresByType at mod_expires’s part and it didn’t help too.
Php.ini
engine = On
output_buffering = 4096
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
By the way I can see other memory limits when run phpinfo()
But probably it does not matter.
my.cnf
bind-address = localhost
skip-external-locking
skip-name-resolve
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
query_cache_limit = 1M
query_cache_size = 16M
Switching bind-address to 127.0.0.1 did not help.
Php modules installed
Bcmath
Bz2
Calendar
Core
Ctype
Curl
Date
Dba
Dom
Ereg
Exif
Fileinfo
Filter
ftp
gd
gettext
hash
iconv
json
libxml
mbstring
mhash
mysql
mysqli
openssl
pcntl
pcre
PDO
Pdo_mysql
Phar
Posix
Readline
Reflection
Session
Shmop
SimpleXML
Soap
Sockets
SPL
Standard
Sysvmsg
Sysvsem
Sysvshm
Tokenizer
Wddx
Xml
Xmlreader
Xmlwriter
Zend OPcache
Zip
Zlib
I have learned that sometimes php modules can cause this kind of things, but I did not try to stop them one by one yet.
Also, I am not much sure but it seems that OPcache is working.
Apache modules installed
Core_module (static)
So_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
alias_module (shared)
auth_basic_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
filter_module (shared)
mime_module (shared)
mpm_prefork_module (shared)
negotiation_module (shared)
php5_module (shared)
rewrite_module (shared)
setenvif_module (shared)
status_module (shared)
The access and error logs don’t show anything special or suspicious.
