9

It has been a few days so far that I am trying to figure how to solve this problem. First of all, I am running a website with an average daily page view of 350,000. Previously, all ads management (tracking click and impression that each ads has served) and content were served in a single server with the following spec:

Server 1
OS: Windows 2008 R2 64-Bit
CPU: Intel® Core™ i5 - 4 cores
RAM: 8 GB
Storage: 2 x 1 TB hard drives
Bandwidth: 10 TB per month

To improve our website speed, I decided to separate the ads management script to another dedicated server because we have more than 15 advertisers to 30 advertisers per each page.

Server 2
OS: Windows 2008 R2 64-Bit
CPU: Intel® Core™ i5 - 4 cores
RAM: 4 GB
Storage: 2 x 300 GB hard drives
Bandwidth: 10 TB per month

The Problem
The problem is that Server 1 can handle both content and ads system. Now, that I take away the ads system and put it at Server 2. Server 2 can barely serve only ads system.

Test

  • First of all, I moved 75% of the ads to Server 2. And then, perform a ping to server: ping -t xxxxx. [I did the ping for 10 minutes and its following similar pattern as below]
Reply from xxxxx bytes=32 time=290ms TTL=116
Reply from xxxxx bytes=32 time=289ms TTL=116
Reply from xxxxx bytes=32 time=320ms TTL=116
Reply from xxxxx bytes=32 time=286ms TTL=116
Reply from xxxxx bytes=32 time=286ms TTL=116
Reply from xxxxx bytes=32 time=348ms TTL=116
Reply from xxxxx bytes=32 time=284ms TTL=116
  • Then, I moved 100% of the ads to Server 2. Then, perform a ping to server again. [I did the ping for 10 minutes and its following similar pattern as below]
Reply from xxxxx bytes=32 time=290ms TTL=116
Request timed out
Reply from xxxxx bytes=32 time=320ms TTL=116
Reply from xxxxx bytes=32 time=286ms TTL=116
Request timed out
Request timed out
Reply from xxxxx bytes=32 time=284ms TTL=116

Attempts

  1. Increase MaxUserPort and TcpNumConnection
  2. Restart the server
  3. Increase IIS Max Instances and Instance MaxRequests

Server Resource

  • Only 10%-15% of the network connection is used
  • Only 10%-15% of the CPU is used
  • Only 25% of the memory is used
5
  • 3
    This is a well-formed question in my opinion and worthy of upvotes, but I still feel like we're missing some clues to help us solve the riddle.
    – Ryan Ries
    Oct 30, 2013 at 3:35
  • @RyanRies, please give pointer which missing information that is needed to solve this riddle. Right now, I have 75% of ads in Server 2 and 25% of ads in Server 1. It performs pretty good now, but I cannot completely move to Server 2. Oct 30, 2013 at 3:40
  • 1
    I agree with @RyanRies: we're missing something. I would look into the way the ad serving system is built: it could be that there is a dependency between the content and ad serving services that causes the excessive delay.
    – Stephane
    Oct 30, 2013 at 10:23
  • @Stephane no, that should and would no cause lost pings. Pings are not handled by the ad serving code but by the TCP stack directly. Which basically means they should work unless the Server is CRAZILY overloaded. THis seriously Looks - as per my answer - like the bandwidth just is not there. Before the Server.
    – TomTom
    Oct 30, 2013 at 13:04
  • Some things that are missing here are 1) what can you change/not change in your environment? 2) how are the web servers configured by disk? 3) are the network times internal to where you're at or is this a remote server? If it's a remote server, how far away is it? 4) if you need to improve your sever, what's your budget freedom to add hardware to the mix across the board 5) are the event logs throwing any errors? 6) I don't see results from perfmon captures at the height of the traffic 7) what are your unique visitor numbers? 8) have you filtered out the junk traffic?
    – Techie Joe
    Nov 1, 2013 at 23:06

2 Answers 2

4

Well, let's start. This is longer.

You totally misjudged the facts here it looks. Windows - even the outdated 2008 R2 which you should update ASAP - is completely capable of handling a volume my mobile phone has no problem handling.

So, that leaves 3 possible areas of issues:

  • Installation. Your drivers may be crappy. Given you run an outdated operating system - how good are your drivers? Update them - this CAN cause all kinds of issues.

  • Network. This seriously looks like "My car is too slow, please help me make it faster" when the problem is you spend most time in a traffic jam and complaint about the traffic not moving. Not a car tuning problem. 10tb traffic say nothing about the network congestion. Watch your network traffic statistics on your NIC and then react accordingly - if they re not topped out at the speed they should be.... your provider has oversold. Simple like that.

  • Code. Could be you need more RAM (computer is busy swapping out to RAM instead of processing) or crappy coding is using all your CPU to a degree that makes the kernel level TCP stack not react properly (yes, ICMP replies are that low). This would be brutal - but it is another avenue to check. It could also be that you overload the discs by accessing them too often instead of caching in RAM, but I somehow fail to see that leading to lost pings. Any issue here is not something an admin can handle, though - you have to throw hardware at it, or take a stick and hit the programmer with it until he fixes it (if it is a "stupid" level mistake that eats the performance - if it is not, then it is a lot harder to make serious gains and it may just be your need beefier hardware).

It definitely requires no tuning of windows - a well configured windows can deliver a LOT more than that. My file servers regularly ß over longer periods of time - deliver 4-6gigabit from a relatively stock setup.

Now, all the numbers you give say nothing Seriously.

  • 10-15% CPU is used COULD mean swapping.
  • 25% memory is used likely is a good indicator now swapping happens, but it could still mean the CPU is waiting for IO.
  • 10%-15% network is used means - absolutely nothing because it is only YOUR Side of the network. What about upstream? What if the provider is putting 20 servers with 1 gigabit on a 1 gigabit uplink from the rack and that is overflowing like hell?

The last point is quite likely - dropped packets are a good indicator of that. And this will not be visible for you.

My advice.... turn off anything on a machine for a moment, make a speed test from external with a large static file. I would bet you run into congestion higher up.

Anything you did so far - maxuserport, tcpnumconnection, restarting the server, playing around with IIS settings - is totally off and do nothing in the best place. Banging a hammer on a slow car never fixes anything - especially if the car is slow because it stands in a traffic jam. I would undo all the changes and start analyzing the problem, not only your server. I would bet on network congestion at the moment.

9
  • 3
    even the outdated 2008 R2 which you should update ASAP - C'mon, TomTom, get a grip. 2008 R2 is still well within mainstream support and a LONG way off going out of Extended Support which is in 2020 I believe...
    – Dan
    Oct 30, 2013 at 10:18
  • 4
    "outdated 2008 R2 which you should update ASAP": that is both incorrect (2008R2 is not "outdated" and shouldn't be updated "just because") and making no sense in the context of the answer. The rest does little to improve your answer since it provide no useful step for identifying the issue, much less fixing it, it just provide generic (and only mildly on topic) advises about performance.
    – Stephane
    Oct 30, 2013 at 10:21
  • Actually it is because guess what was seriously changed later ß the IP stack. Scalability being a Point. So, yes, in the given context it is. Updates should happen asap - because guess what, YOU may not make your stuff better over time, most companies do. 2012 R2 is better than 2008 R2.
    – TomTom
    Oct 30, 2013 at 13:03
  • Using speedtest.net, the uplink and downlink is around 100mbps. Oct 30, 2013 at 15:45
  • There are two dependency software for ads system: memcached and MySQL. It keep track all the impression and click in the memcached and update to MySQL once every 10 minutes. Oct 30, 2013 at 15:47
0

Are the pings external to the two servers? If so, try the ping from server one to server two. If the results are totally different from when pinging externally then that shows the issue may be not due to server two. Also, have a look in resource monitor for the disk queue length for the drives and the active time - its some more information which may highlight a problem.

2
  • I have already tried that. I used Server 1 to ping Server 2 and the ping still drop if I put more ads. Oct 30, 2013 at 15:44
  • 1
    The connection between server 1 and server 2, is it via your own hardware? Or is it the datacentre switch? Just trying to determine if when you pinged you were going through DC hardware which could have been causing congestion. Have you got anything in IIS / Windows which is dynamically blocking requests? No dos protection which is blocking legitimate requests? The system, is it ASP.NET and the db is MySQL? Oct 30, 2013 at 16:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .