I have one dedicated server for Varnish to serve tens of thousands of static files. This server has 32GB of RAM. These files take up almost all the RAM, the vast majority of them are also used regularly, so I stand now before the problem - what to do. The server dealing effortlessly with the handling of requests, but the number of served files is increasing all the time and at any moment I will face the problem of RAM. Adding additional RAM is not an option, as well as saving part of the data on the hard drive.
I decided to put an additional server with the same parameters. I would like to spread the data between the two servers. Both servers will work with HAProxy.
As the amount of data stored in the cache grows very rapidly, and their rotation, I need a solution that will allow me to automatically distribute the data on both servers. For example:
Varnish1 handles site S1, S3, S5
Varnish2 handles site S2, S4, S6
At a time when it appears page S7, should be automatically appended to the least loaded server. Likewise, when the site will disappear, eg S2, the data should be re-distributed the servers to achieve the most balanced load. The same should also happen during the normal operation of servers, when one site, such as S1 will suddenly consume much more resources than normal, the server should automatically transfer part of the supported data to other servers. I accept the partial replication, ie, when the number of requests such as S1 increases to such an extent that will clog up one server, data should be immediately replicated to another server.
I predict that in the future will increase the amount of data served several times as well as the number of servers, Varnish, so I can not rely on hand-held solutions.