I have a couple of questions about caching:
Basically I have a server which serves images/video for a website hosted elsewhere. It runs Caddy (but that is not important).
I have a 8TB ZFS (HDD) pool which at the time of writing has about 1TB of stuff on it, these files tend to be good targets for caching as the access patterns are not random (a particular post will become popular for a few days and get a number of hundreds of thousands of hits, then slow down).
I also have a 200GB SSD which I intend to use for some kind of caching. What I would like to get some feedback on is what kind of caching would be most useful. FWIW my server has 32GB RAM.
My main question: is (in memory) Varnish useful if I have (way) more content than memory? I have done some googling about what happens when Varnish runs out of memory and what I've read suggests "bad things", but this seems odd to me. I would have imagined that if a few items are being hit with high frequency in a short period, Varnish would perform well, even if there is not enough memory to cache everything.
The next question is about bcache, and this largely hinges on the answer about the viability of Varnish for my use case. I have this 200GB SSD, should I use it for bcache (and have Varnish in memory on top of that), or should I use Varnish on disk (on SSD, still a lot faster than the big HDD pool) and have a significantly larger portion of the content be cached?