Load Balance (http) nodes slower when Redis is non-local

Question

We run a load balance with 3 Windows 2019 servers. One of them is the "master" one, serving SMB share to the other nodes, and it's also the server that Redis runs on (Redis used for sessions and cache). But this master srever is also a HTTP node.

Our problem is that on the master server, our website runs considerably faster than on the other nodes, it's 1 second of difference, pretty big.

In fact, the other (non-master) nodes have faster CPU, and on a PHP benchmark they perform better:

Master server (Redis, SMB, IIS):

PHP version : 7.4.16
Platform : WINNT
--------------------------------------
test_math                 : 0.396 sec.
test_stringmanipulation   : 2.818 sec.
test_loops                : 0.304 sec.
test_ifelse               : 0.374 sec.
--------------------------------------
Total time:               : 3.892 sec.

The other nodes:

PHP version : 7.4.16
Platform : WINNT
--------------------------------------
test_math                 : 0.311 sec.
test_stringmanipulation   : 2.334 sec.
test_loops                : 0.234 sec.
test_ifelse               : 0.191 sec.
--------------------------------------
Total time:               : 3.07 sec.

So it's definitely not a CPU issue, because the master node is slower. So it's either network, SMB or Redis issue.

I'm leaning towards Redis, because we use Wincache and Opcache for the nodes, so basically all .php files are cached locally for each node (SMB used only for revalidation every few minutes).

I do not know however how to find out where the bottleneck is exactly, and how to fix it.

Can anyone please help?

Thanks Alex

Answer 1

I actually got it all wrong, it's the SMB share the creates the problem. I verified this by copying files to the slave nodes, and the performance same as the master server, or even slighty better (due to faster CPU).

So this is indeed an SMB issue. Probably SMB still sends data back and forth.

The SMB share is on very fast SSD storage, I suspect it's either simply because SMB is not fast enough, or a configuration issue.

If anyone has any tips on SMB for this case, please do tell :)

Thanks Alex