3

I have one master nginx.conf where I include the rest of my servers (sever blocks) with the include directive:

include myservers/*.conf;

My problem is when I have a new configuration file in myservers/ I need to reload the nginx with nginx -s reload

The problem takes long time to reload the server takes 1 minute and this is going to grow, because I will have more and upstream servers.

Do you see any technique to improve this?

The only solution I have found for now is the paid version of Nginx Nginx Plus API https://docs.nginx.com/nginx/admin-guide/load-balancer/dynamic-configuration-api/ where you can add new upstream severs dynamically with a REST API without any reload.

Also I was thinking to have a kind sharding technique with one master wirth hash keys to slave servers (like elasticsearch with the RAFT algorithm to keep the consensus state) so when you need to update you have only reload one slave server with less upstream servers.

8
  • 4
    Since a reload is graceful and happens without downtime it shouldn't matter that the reload command takes a minute or longer to complete. During that time all your existing sites will remain online.
    – HBruijn
    Jan 31, 2019 at 10:35
  • @HBruijn yes it's important. Until the new config files are reload I will not able to manage the requests to the new severs configurations because are not ready until the reload is done. Jan 31, 2019 at 17:06
  • Why is it critical to include a new server in less than a minute? How often do you add servers?
    – Tim
    Jan 31, 2019 at 18:49
  • 1
    Waiting a minute before you can access your new server is quite normal and tollerable, in my opinion. If you want it faster, perhaps split the load over multiple instances of nginx, each running on its own IP address. I would prefer that anyway so as not to have too many websites on the same IP address.
    – Tommiie
    Feb 3, 2019 at 11:48
  • 2
    I suggest that providing more context / information will get you a better outcome than just offering a bounty, given the limited information you've provided. I asked a few questions above I suggest you edit your question to address. There are a lot of very smart, experienced people on SF who charge large amounts of money to give people advice offering their help, but they can't help if they don't have the full picture. Answering your question might help solve your immediate problem, but maybe you need to address a different issue than the obvious one.
    – Tim
    Feb 4, 2019 at 5:30

4 Answers 4

3

How many files, and what sort of configuration, do you have that nginx -s reload is taking a whole one minute?!

Identify the Source.

I think you have to figure out why it's taking so long in the first place, before being capable of coming up with a solution to address it.

Filesystem issues?

  • Is it a ridiculous number of individual files that slow down the process?

    E.g., does doing a cat myservers/*.conf | md5 take a whole minute by itself?

    If so, you might want to look into using a ramdisk for your configuration; or into keeping the individual configurations in a database, and having a single nginx.conf for reload purposes.

Configuration directive issues?

  • Is it the contents of the configuration files that take a really long time to reload?

    There could be more than one way this could be a problem.

    For example, maybe one of your configurations is using a domain name that takes a long time to resolve (through a timeout), slowing down the whole reload. This is potentially a security vulnerability in your setup, as a single user might be able to slow down your whole reload sequence given the "right" input.

    This could also be another issue with the configuration, maybe when too many individual log files have to be closed/opened. You can look more into this with tools like lsof and/or fstat, to see the number of open files that your applications take.

Is this even a real problem?

  • As others pointed out, even as-is, this is already not a huge problem, because nginx -s reload is a graceful reload of the configuration, where nginx should still remain fully functional, even when you're reloading its configuration.

    I would say it should be totally reasonable to architect the reloads into batches, and perform the reloads once every 5 to 15 minutes. If you're dealing with new domain names, you probably already have to wait until the configuration starts working on the DNS level. A delay of up to 1 minute is not at all unreasonable, and is very often implemented in production services of various cloud providers to this day. In fact, DNS root zones are often updated in batch mode as well, often on a schedule much less frequent than once every 15 minutes, especially given the sheer volume of data that's involved; for example, .ru gets refreshed only 4 times per day, as it has 5 million records, and is mirrored by several separate providers for redundancy, with each update taking up to 30 minutes, hence they have to be spaced apart to ensure a reasonable level of consistency, and to ensure that the separate updates don't run into each other.

    If you require to have the changes have immediate effect, then perhaps a different architecture is required; maybe one where a separate staging area is provided for the testing of configuration, or a multilayer approach, or a commercial version of nginx, and/or third-party plugins.


Come up with a Solution.

Depending on the source of the issue, the solution would be to re-architect the way you're doing the configuration.

Without knowing the source of the issue, the question is just too broad to offer any specific advice.

2
+25

I fired up a fresh virtual machine (with SSD backed storage) and installed nginx on it. And then I wrote a script to generate a huge number of files, each containing a single server block. They looked a lot like this:

[root@localhost ~]# cat /etc/nginx/sites/server047393.conf 
server {
    listen 80;
    listen [::]:80;
    server_name server047393;
}

At first I made 50,000 of them, but that only took 9 seconds to reload nginx, so then I moved up to 100,000. With that, it was consistently taking 20 seconds to reload nginx. About the first half of that time was disk I/O wait, the latter half was CPU. With this number of server blocks, nginx is using nearly 1GiB of RAM.

This really doesn't look like a problem, unless you have the nginx configuration on a really slow disk. It does get re-read in its entirety when you reload or restart nginx. With a rotational disk, this easily could take a couple of minutes to reload. Use an SSD or even a RAM disk to store the nginx configuration.

Indeed, nginx's own optimization advice for server names hardly mentions configuration parsing time. It's not really something you should care about very much. What it does talk about a lot is the amount of time it takes to locate the correct server block to process an incoming request. By default nginx tries to optimize this to minimize CPU cache line misses.

To optimize this for your larger number of server names, you might need to do nothing, but you probably do need to adjust the server_names_hash_max_size directive. Run nginx -t. If you see a message like this:

nginx: [warn] could not build optimal server_names_hash, you should increase either server_names_hash_max_size: 512 or server_names_hash_bucket_size: 64; ignoring server_names_hash_bucket_size

Then you should tune server_names_hash_max_size. Start by setting this to a power of two larger than the number of server_names you are creating. If you have 30,000 server names, start with server_names_hash_max_size 32768.

The optimization document does mention that:

if nginx’s start time is unacceptably long, try to increase server_names_hash_bucket_size.

I found in testing that this didn't really help, but if you want to try it, increase it by powers of two each time. This value must be a power of two, or nginx will not start. This value is set by default according to the CPU cache line size, so if you're on a virtual machine and the CPU properties are not correctly exposed to the VM, you might be able to safely double this number (or nginx refused to start in the first place, but that's a slightly different error message, could not build the server_names_hash). Don't go overboard with it, or your incoming requests will be slowed down by CPU cache misses.

2
  • An empty configuration block is not representative of the time it takes to parse a real configuration. Feb 6, 2019 at 22:20
  • @user5994461 It doesn't have to be a "real" configuration to demonstrate where the bottlenecks are. It certainly could have been, but the end result would be the same, with perhaps a longer parsing time. And besides, the configuration is perfectly "real" anyway. It is perfectly suitable for serving a static web site, images, CSS, etc. Feb 6, 2019 at 23:30
0

The obvious (naive) approach is to use a ramdisk, then set up regular backups. This can be scheduled or triggered by some kind of hook, e.g. when the user makes a change in cPanel or something, it triggers a backup.

A variation on this is forcing the whole directory to be treated as cache, e.g. vmtouch.

There's also the possibility to move certain parts of the configuration to a database with something like ngx_postgres. It'd be much better if nginx supported loading the full config from DB, but I'm not aware of such a thing.

0

How about this very dynamic approach, which does take a direction of reducing the amount of config files. Depending on your requirements it could be a way forward:

https://stackoverflow.com/a/14464835

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.