We are using Amazon.com Product Ads and have around 150,000 products. Unfortunately, every time we upload our feed Amazon's system hits our server with 150,000 requests. There probably isn't anything wrong with the 150,000 requests except that their bot sends them too fast and our IIS7.5 webserver eventually cannot respond to anything until it is recycled.
While I have had to block their IP address to prevent losing business, I am looking for another solution to throttle the requests so other requests take priority over Amazon.
I have tried working this out with Amazon's engineers, but all they could come up with was to split the file into multiple files. That really didn't help all that much as their bot still sends the requests at too high of a rate for our server to respond.
So, is it possible to lower the priority of these requests so they yield to real traffic?
I am working this from the other end as well and discovered what could be the cause to my ASP.NET application woes. This thread on StackOverflow sounds similar to what I am seeing - the server still has plenty of memory (> 2GB) and the CPU is around 70%. However, the application slows down and eventually quits responding after a sustained period of high requests.
That said, it may be tricky to track this down being that I am using practically everything in the list in that post and then some. It would still be helpful to find a way to configure the server to lower the priority of these requests in the meantime if there is such a thing.