32

We are using Nginx as a reverse proxy to Apache in a service that gives anyone their own website. At account creation, the system creates a new nginx conf file for the domain with two entries, one for port 80, the other for 443. We are noticing that at every 30 or so domains, we get the error:

Restarting nginx: nginx: [emerg] could not build the server_names_hash, 
you should increase either server_names_hash_max_size: 256 
or server_names_hash_bucket_size: 64.

With around 200 domains and growing we have had to up the server_names_hash_max size to 4112 and are concerned this is not going to scale well. I'm looking to understand how these configurations work and what the optimal settings would be to ensure we can grow to thousands of domains using this method.

Also, at that hash size nginx is starting to take seconds to reload which is causing the system to be unavailable while it restarts.

Here are the overall settings (running on Ubuntu server 10.10 nginx/1.0.4):

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
    worker_connections 4096;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 300;
    types_hash_max_size 2048;
    # server_tokens off;

    server_names_hash_bucket_size 64;
    # server_name_in_redirect off;
    # server_names_hash_max_size 2056;
    server_names_hash_max_size 4112;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;

ssl_session_cache shared:SSL:10m;
ssl_ciphers ALL:!kEDH:-ADH:+HIGH:+MEDIUM:-LOW:+SSLv2:-EXP;
}

(Below the ciphers are couple main site configs and a catch all):

include /etc/user-nginx-confs/*;

server {
listen 80;
server_name .domain.com;
location / {
proxy_pass http://127.0.0.1:8011;
proxy_set_header host $http_host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-SystemUse-Header 111;
}
}

server {
listen 443 ssl;
server_name .suredone.com;
ssl_certificate /etc/apache2/sddbx/sdssl/suredone_chained.crt;
ssl_certificate_key /etc/apache2/sddbx/sdssl/suredone.key;
location / {
proxy_pass http://127.0.0.1:44311;
proxy_set_header host $http_host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-SystemUse-Header 111;
}
}

server {
listen 80 default_server;
listen 443 default_server ssl;
server_name _;
ssl_certificate /ssl/site_chained.crt;
ssl_certificate_key /ssl/site.key;
return 444;
}

(And a sample user conf file)

server {
listen 80;
server_name username.domain.com;
location / {
proxy_pass http://127.0.0.1:8011;
proxy_set_header host $http_host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-SystemUse-Header 1111;
}
}

server {
listen 443 ssl;
server_name username.domain.com;
ssl_certificate /ssl/site_chained.crt;
ssl_certificate_key /ssl/site.key;
location / {
proxy_pass http://127.0.0.1:44311;
proxy_set_header host $http_host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-SystemUse-Header 1111;
}
}

Any help and direction is greatly appreciated!!

6 Answers 6

19

The list of server names that nginx serves is stored in a hash table for fast lookup. As you increase the number of entries, you have to increase the size of the hash table and/or the number of hash buckets in the table.

Given the nature of your setup, I can't think of any way for you to easily reduce the number of server names you're storing in the table. I will suggest, though, that you not "restart" nginx, but rather simply have it reload its configuration. For instance:

service nginx reload
2
  • That is great for the latter part of the question, thanks. So should I be concerned that the server_names_hash_max_size will be somewhere around 20000 to get to 10000 domains? Aug 22, 2012 at 14:49
  • It's only an issue when you restart nginx. As I said, reload instead whenever possible to avoid the problem. Aug 26, 2012 at 1:59
40

Just some technical details that I dug form the source code:

  • The general recommendation would be to keep both values as small as possible.
  • If nginx complains increase max_size first as long as it complains. If the number exceeds some big number (32769 for instance), increase bucket_size to multiple of default value on your platform as long as it complains. If it does not complain anymore, decrease max_size back as long as it does not complain. Now you have the best setup for your set of server names (each set of server_names may need different setup).
  • Bigger max_size means more memory consumed (once per worker or server, please comment if you know).
  • Bigger bucket_size means more CPU cycles (for every domain name lookup) and more transfers from main memory to cache.
  • max_size is not related to number of server_names directly, if number of servers doubles, you may need to increase max_size 10 times or even more to avoid collisions. If you cannot avoid them, you have to increase bucket_size.
  • bucket_size is said to be increased to the next power of two, from the source code I would judge it should be enough to make it multiple of default value, this should keep transfers to cache optimal.
  • Average domain name should fit into 32 bytes even with hash array overhead. If you increase bucket_size to 512 bytes, it would accommodate 16 domain names with colliding hash key. This is not something that you want, if collision happens it searches linearly. You want to have as less collisions as possible.
  • If you have max_size less than 10000 and small bucket_size, you can come across long loading time because nginx would try to find optimal hash size in a loop.
  • If you have max_size bigger than 10000, there will be "only" 1000 loops performed before it would complain.
3
  • This is great info; thanks for the research and write-up.
    – womble
    Jan 13, 2017 at 0:11
  • @brablc I am curious how you got to 32769 for instance. Where can one see whats the current heap size? Mar 2, 2018 at 21:14
  • 1
    The occupied memory would be max_size*bucket_size (but I do not know if it is shared or per worker). I had 8000 server names and 32769 felt already too high. But if you have a lot of memory you may want to go higher.
    – brablc
    Mar 3, 2018 at 23:18
5

Increase the "server_names_hash_bucket_size" configuration inside your nginx.conf.

I had it 64 and changed to 128.

Problem solved.

3

I'm not 100% sure in your case, but I was getting the same warning because I was calling proxy_set_header for X-Forwarded-Proto twice:

proxy_set_header X-Forwarded-Proto ...;

This was happening because I was including proxy_params, and it contains this line among others:

proxy_set_header X-Forwarded-Proto $scheme;

Removing that line from my site's configuration made the warning go away.

0
2

@Michael Hampton is absolutely right with his answer. This hash table is constructed and compiled during restart or reload and afterwards it's running very fast. I guess this hash table could grow a lot more without degrading the performance noticeable. But I'd suggest using a size which is power of two, like 4096, due to the nature of C code.

2
  • Power of two with what base, Is it correct to grow in multiples of the default of 512? Aug 22, 2012 at 14:52
  • Yes, absolutely. Aug 23, 2012 at 5:18
-2

Change

proxy_set_header X-Forwarded-For $remote_addr;

to

proxy_set_header X-Real-IP $remote_addr;

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .