I am currently dealing with a networking problem on high latency (100-400ms) Internet links. I run a Minecraft network, and I recently moved it to a separate datacenter to get a server with a better CPU and more ram. The users of this server are spread all over the world. Before the switch, the server was in Montreal and the users in Europe got ~100-200ms latency, and those in Australia got ~200-300ms latency. After the switch, the server is in Germany, the users in North America get ~100-200ms latency, and the users in South America and Australia get 200-400ms latency. Overall, the latency is quite similar, but who gets great latency and who gets tolerable latency varies (note that Minecraft isn't very latency sensitive in general, especially compared to most video games). There also isn't any significant packet loss, as measured by the MTR and ping tools. Furthermore, the software on both servers is almost identical. Both servers run Debian 10, and I tar'd all software + configuration that's not in APT repositories to send that over, while reinstalling the exact same packages via apt. As such, the software configuration should be essentially identical.
Yet, many users are having connection trouble. It seems to only occur around 6:00 PM (± a few hours) in the eastern United States. The connection trouble specifically takes the nature of the throughput of all TCP connections being atrociously low. With an SSH+SOCKS proxy, it took minutes to load a normal web page (Gmail), and in-game, it often takes minutes for even a simple chat message to get through if a few MBs of world data are currently being transferred. The effective latency of a TCP connection (e.g. the time it takes a chat message to get through it) increases unreasonably massively when any data is put over that TCP connection. A normal SSH session with just a terminal is basically fine, and the game is mostly fine if not much is going on, but as soon as anything of significant size gets sent over TCP and it is during the aforementioned time, then the throughput breaks down and latency over TCP (but not via ping) becomes unreasonable, even to multiple minutes in the worst cases. When this issue first took place, there was significant packet loss (~25%) which I thought was to blame, but, that packet loss is no longer occurring (according to ping, etc.) yet the issue remains. The packet loss, but not the problematic symptoms, disappeared after I made a general report to the new host about packet loss, but before I could give them more detailed data with MTR as they requested in their response to that report. My impression from the host is that they did not change anything, but, who knows really.
As such, at this point, I suspect that the relevant difference between the servers is that the old host (OVH) does some sort of tuning of their OS images (something that I know to be the case), and that the new host (Hetnzer) does not.
I suspect that this tuning has something to do with TCP window size, but when I attempted to manipulate those settings to make changes, the settings didn't seem to do what they're supposed to. Specifically, when I set the various
net.core.mem settings that I find listed on the Internet via sysctl, the window size that
iperf selects (or the maximum that it is allowed to select, when using the
-w option) seems to take a random value with no discernible-to-me relation to the values that I set via sysctl, as opposed to behaving in the way that I expect, where it's maximum value is simply whatever I set via sysctl. Note that
iperf -s misbehaves even before a client connects to it, so failing to make the same changes on the client is not a plausible explanation.
As such, I'm wondering 2 things:
1) How can I fix my server, and allow the latency in TCP connections to be similar to the actual latency on the link, even at peak times and under moderate load (a few mbps)?
2) How can I reliably and predictably change the TCP window size of all applications? (or, equivalently, what's going on with the sysctl settings applying in seemingly random ways? / what is the pattern that I'm missing?)