Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are using Amazon.com Product Ads and have around 150,000 products. Unfortunately, every time we upload our feed Amazon's system hits our server with 150,000 requests. There probably isn't anything wrong with the 150,000 requests except that their bot sends them too fast and our IIS7.5 webserver eventually cannot respond to anything until it is recycled.

While I have had to block their IP address to prevent losing business, I am looking for another solution to throttle the requests so other requests take priority over Amazon.

I have tried working this out with Amazon's engineers, but all they could come up with was to split the file into multiple files. That really didn't help all that much as their bot still sends the requests at too high of a rate for our server to respond.

So, is it possible to lower the priority of these requests so they yield to real traffic?

Update

I am working this from the other end as well and discovered what could be the cause to my ASP.NET application woes. This thread on StackOverflow sounds similar to what I am seeing - the server still has plenty of memory (> 2GB) and the CPU is around 70%. However, the application slows down and eventually quits responding after a sustained period of high requests.

That said, it may be tricky to track this down being that I am using practically everything in the list in that post and then some. It would still be helpful to find a way to configure the server to lower the priority of these requests in the meantime if there is such a thing.

share|improve this question
    
Are you sure there isn't a setting in Amazon for how quickly they pole your server? I believe Google has a set up for that, I would expect Amazon to be similar. –  Jack M. May 21 '12 at 21:10
    
Nope, Amazon has no such setting. Even worse, their system seems to always start during peak hours regardless of what time I upload the file(s). –  NightOwl888 May 22 '12 at 8:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.