0

For our game we host its static assets on a VM that has just httpd installed and running on it (of course along some native linux things), in order to serve web content. MPM configured is worker with MaxClients of 6400, ServerLimit 100 and ThreadsPerChild 64. Memory is 4 GB. With the above configuration, served static content is of total size about 20 MB, and it is served in my country (Bulgaria), as well in different other countries. It is checked and confirmed that national and international bandwidth speeds do not differ. However, at peak moments, when bandwidth is maxed out, we start to receive mass complaints from distant users (i.e. from Russia) that the game is downloaded fully for 2-3 minutes. Anytime we check loading the game with disabled cache from here, it took around 10 secs, every time we tried, from any computer. We added 2 more VMs from the original VM's image (same config and content), and did the fastest load balancing - DNS round robin to total of three IPs. Complaints shrinked down, but loading time for Russian users kept on being 1 minute+. When we again tried multiple times downloading the game from here it was still 10s, no difference for us. What may be the possible reasons given that static content servers have equal national and international peering, and when load is low, all Russian users are able to download for 10s too, but not on peak times? Shouldn't it be the same for all users?

P.S. At all times the static servers had plenty of memory, and spawned httpd processes never got above 50, with set limit of 100

EDIT: Short summary of the question - on low load, all clients (local and distant) download the client for equal time (for instance 15 sec). When load is high, local clients load it for 15 secs again, while distant do for 2-3 minutes. What are the possible reasons?

  • 1
    I've noticed some services to BG are very slow from abroad, but this sounds more like congestion on the route to Russia during peak hours. – Recct Sep 28 '15 at 14:41
0

As per the clarification that this was only happening when the bandwidth was maxed out, that might sound like a totally normal behaviour, then — when you max out your available bandwidth (to line rate), you may start losing packets and have TCP window for long-bound clients never scaled up for optimum speeds; Bandwidth Delay Product grows, time to download the same file over the same pipe increases; you'd have to do some traffic shaping (in other words, packet queueing and prioritisation) if you want to make it more even for everyone during the periods of overload. – cnst Oct 6 at 4:56

3

The answer depends on a lot of things. You can't just say your international speeds are constant. Distant users will always have lower performance, depending on the network between you and them, and how heavy it's loaded.

BTW, you did say your bandwidth is maxed out. Bandwidth of your server's network connection? Then you really need a CDN or caching reverse proxies.

I can offer some quick improvements:

  • Use Nginx; it can serve static content much more efficiently.
  • Use a CDN like Cloudflare, or if that's too elaborate, you could rent a VM in Russia and install a caching reverse proxy on it, make your DNS geo IP aware and have Russian users redirect there. Cloudflare might actually be easier :)
  • I edited the question to reflect more on what I asked. Thank you – Martin Asenov Oct 2 '15 at 14:53
3
+50

You can't really say that peering is the same for national and international traffic.

It might have changed in the last couple of years, but traditionally in Russia, most providers have never paid for any local peering, getting it directly from MSK-IX, with the rest of the traffic being handled by transit providers.

The links in various directions almost always have varying capacities, and very often certain links are kept saturated every once in a while (either through unexpected traffic spikes, or because someone is too lazy to update their links, or pay more for more traffic etc), and this may especially happen more often during peak hours.

Often, at the peering or transit points, providers pay a fixed rate for an unlimited 100Mbps, 1Gbps or 10Gbps. What happens when the traffic outgrows what's been paid for? Some packets are dropped, some are slowed down, and it generally only happens during the peak hours, and sometimes only in one direction (but even if it happens in one direction, traffic still slows down in both, since latency gets increased, and some ACK congestion control packets get lost as well).

I would troubleshoot the issue by running mtr towards one of the hosts in Russia that's experiencing the issue, as well as from one of the hosts in Russia towards your server in Bulgaria. I find it most useful to run each instance for 30 seconds to 15 minutes (mtr will aggregate the statistics for the whole duration of such run), and then run it again for another 5 to 15 minutes immediately after the prior run completes. This way, you'll be able to see exactly during which time the issues strike.

Otherwise, it might also be an issue with Apache, perhaps related to a higher latency of the hosts in Russia -- nginx is generally more efficient in serving all sorts of content than Apache, so, perhaps it's a good opportunity to try out nginx instead?

  • I edited the question to reflect more on what I asked. Thank you – Martin Asenov Oct 2 '15 at 14:53
  • Your edit appears to be just the summary of what was already said by you earlier. It's just not enough information to make a more informed decision at this point, due to what is already outlined in the answers. I mean -- you seem to present it as a fact that the issue is due to the high load locally within your server/Apache, but you never provide any good evidence or plausible explanations of why that's true. An example of such evidence might be mtr monitoring, or, since you run 3 identical servers all with Apache, replacing one or two (or even all three!) of them with nginx. – cnst Oct 3 '15 at 9:30
  • Since we got those problems we switched to Russian static server because we were forced to. Unfortunately we are not experiencing the same load again, so that I can test freely, furthermore the exact IPs of users who experienced the problem are unknown to me. It is proven that we had this load, because it was a planned release, and we were expecting high load + I was watching all the time the page views on Google Analytics, where thousands of users were coming in. – Martin Asenov Oct 4 '15 at 13:42
  • @MartinAsenov, so, are you saying it was only some users in Russia? That'd be even more evidence of some transit links being overloaded somewhere, which just happens to have happened during the time you've had high load. And we're not disputing you've had the load! Just not enough controlled variables presented. BTW, it appears that hetzner is really popular in Russia, perhaps you might want to check out if that's an option for better connectivity. – cnst Oct 4 '15 at 22:09
  • actually all Russian users had the problem, including testers who previously had downloaded for 15 seconds. And it is not ISP issue nor internet route issue because it was only happening when our available bandwidth was maxed out. At that point I would think all clients should download slow. The case was local clients were still downloading fast, while distant - very slow – Martin Asenov Oct 5 '15 at 8:57
0

While the speed of light is constant, and data travels through long distance fiber cables at the same speed, the further one is away from the source, the more "hops" it has to go through.

If you run a traceroute to a server 100 miles away, and then compare that traceroute to a server that is half way across the globe, the one half way across the globe will likely go through many more hops.

Latency is the amount of time it takes data to go through each router (hop) along the way before it reaches its destination, and latency is the issue here.

  • I edited the question to reflect more on what I asked. Thank you – Martin Asenov Oct 2 '15 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.