1

I'm running apache2 on a EC2 micro instance with ~600 mb RAM. The instance was running for almost a year without problems, but in the last weeks it just keeps crashing, because the server reached MaxClients.

The server basically runs few websites, one wordpress blog(not often used), company website(most used) and 2 small sites, which are just internal.

The database for the blog runs on RDS, so there's no Mysql running on this web server.

When I came to the company, the server already was setup and is running apache + mod_php + prefork. We want to migrate that in the future to a nginx + php-fpm, but it still needs further testing. So for now I have to stick with the old setup.

I also use CloudFlare DDOS protection in front of the server, because it was attacked a couple of the times in the last weeks.

My company don't want to pay money for a better web server at this point, so I have to stick with the micro instance also. Additionally the code for the website we run is really bad and slow and sometimes a single page load can take up to 15 seconds. The whole website is dynamic and written in PHP, so caching isn't really an option here. It's a customized search for users.

I've already turned off KeepAlive, which improved the performance a little bit.

My prefork config looks like the following:


 StartServers 2
 MinSpareServers 2
 MaxSpareServers 5
 ServerLimit 10
 MaxClients 10
 MaxRequestsPerChild 100

The server just becomes unresponsive after a while running and I've run the following command to see how many connections there are: netstat | grep http | wc -l 75

Trying to restart apache helps for a short moment, but after that a while the apache process(es) become unresponsive again.

I've the following modules enabled(output of apache2ctl -M)

Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 alias_module (shared)
 authz_host_module (shared)
 deflate_module (shared)
 dir_module (shared)
 expires_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 ssl_module (shared)
 status_module (shared)
Syntax OK

apache2.conf

# Security
ServerTokens OS
ServerSignature On
TraceEnable On
ServerName "web.example.com"
ServerRoot "/etc/apache2"
PidFile ${APACHE_PID_FILE}
Timeout 30
KeepAlive off
User www-data
Group www-data

AccessFileName .htaccess
<Files ~ "^\.ht">
  Order allow,deny
  Deny from all
  Satisfy all
</Files>

<Directory />
  Options FollowSymLinks
  AllowOverride None
</Directory>

DefaultType none
HostnameLookups Off
ErrorLog /var/log/apache2/error.log
LogLevel warn
EnableSendfile On

#Listen 80


Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Include /etc/apache2/ports.conf

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
Include /etc/apache2/conf.d/*.conf
Include /etc/apache2/sites-enabled/*.conf

Vhost of main site

<VirtualHost *:80>
  ServerName www.example.com
  ## Vhost docroot
   DocumentRoot /srv/www/jenkins/Web
  ## Directories, there should at least be a declaration for /srv/www/jenkins/Web

  <Directory /srv/www/jenkins/Web>
    AllowOverride All
    Order allow,deny
    Allow from all
  </Directory>

  ## Load additional static includes

  ## Logging
  ErrorLog /var/log/apache2/www.example.com.error.log
  LogLevel warn
  ServerSignature Off
  CustomLog /var/log/apache2/www.example.com.access.log combined

  ## Rewrite rules
  RewriteEngine On
  RewriteCond %{HTTP_HOST} !^www.example.com$
  RewriteRule ^.*$ http://www.example.com%{REQUEST_URI} [R=301,L]

  ## Server aliases
  ServerAlias www.example.invalid
  ServerAlias example.com

  ## Custom fragment
<Location /srv/www/jenkins/Web/library>
 Order Deny,Allow
 Deny from all
</Location>
<Files ~ "^\.(.+)">
  Order deny,allow
  deny from all
</Files>

</VirtualHost>
  • I see that you configure Jenkins in your apache; if Jenkins is running on the same machine, I can imagine that you run out of memory very quickly on a micro instance. Am I correct with this assumption? – Skiaddict Aug 19 '14 at 11:36
  • Nope, you're not. Jenkins just deploys the code to the web server. Jenkins is based in my office, while the micro instance is on AWS. – Shiyu Sekam Aug 19 '14 at 11:38
  • In that case I cannot see anything odd in your config... – Skiaddict Aug 19 '14 at 11:42
6

Upgrade your server

I know you don't want to hear it. I know you client doesn't want to hear it. But it's what you need to hear! If the client is running 4 websites (production no less) they should fork out some cash for a better server.

But why?

micro instances are NOT designed for production use (AWS says so themselves), they suffer from contention, don't expect anything great or predicable out of the CPU, Network or Disk I/O.

If you must use mirco, consider t2 instead of t1

T1 = http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html

The t1.micro is a previous generation instance and it has been replaced by the t2.micro, which has a much better performance profile. We recommend using the t2.micro instance type instead of the t1.micro. For more information, see T2 Instances.

T2 = http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html

They are intended for workloads that don't use the full CPU often or consistently, but occasionally need to burst. T2 instances are well suited for general purpose workloads, such as web servers, developer environments, and small databases.

A quick hack

Sometimes you can solve contention issues (CPU steal, bad neighbour syndrome etc) with your host by stopping the EC2 and starting it again. This will cause a change to the underlying host that your EC2 is sitting on.

Answers to some of your other questions...

The instance was running for almost a year without problems, but in the last weeks it just keeps crashing, because the server reached MaxClients.

Why do you think the server has reached MaxClients? More traffic? You could increase this number, however I suspect a micro instance couldn't handle too many concurrent connections.

The database for the blog runs on RDS, so there's no Mysql running on this web server.

I love RDS, and recommend people use it. But why is your company forking out cash for RDS but not willing to upgrade their EC2 instance? Why not just drop the RDS and upgrade the EC2? (In an ideal world I'd say keep the RDS and upgrade the EC2).

The whole website is dynamic and written in PHP, so caching isn't really an option here.

Are you sure APC wouldn't help in this situation? It can be hard to determine exactly how much APC can help until you try. http://php.net/manual/en/intro.apc.php

Trying to restart apache helps for a short moment, but after that a while the apache process(es) become unresponsive again.

This sounds like you're (maybe) getting too much traffic?

  • Thanks, that was helpful. I'll try to convince the company to pay a bit more and get a better server. – Shiyu Sekam Aug 19 '14 at 12:47
0

Its late but another solution is decrease workload on your ec2. Use a cloudfront distribution in front of this instance. Will decrease load on your server if the caching policies are correct and AFAIK with the right caching policies your server can scale to thousands of readers (not editor, admins etc)

Tutorial to setup CDN with wordpress

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.