Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm considering writing an application that aggregates information from a fairly popular website. This application would request information from this website at a set interval. I know this is a really hard question to even "ballpark" an answer to, but what might be a good safe interval to stay mostly "under the radar"? I'm a programmer first, a human being second, and a server admin a distant third, so my knowledge of what a server software like Apache can handle as far as server load with dynamic content is pretty basic.

Again, I know that this question is EXTREMELY open ended and the answer depends on many many many variables, but if anyone has some related experiential knowledge they'd like to share, it would be very much appreciated.

share|improve this question
No "right" answer == community wiki. –  womble Oct 27 '09 at 21:06

3 Answers 3

up vote 2 down vote accepted

First, second and third, I would see if the site has an API. Fourth, I would see if the site has a Terms of Use Policy. Lastly, random numbers are your friend.

share|improve this answer
Servers with APIs usually also have TOS which prevent you from re-using the content. –  Jay Stevens Oct 27 '09 at 22:08

If it is measured in seconds and a high traffic site, it shouldn't be that much of an impact. More important than your second+ interval is probably to ensure you are accepting compressed responses etc properly.

Although if you're really trying to be polite you should ask them for permission or for a copy of the data you want.

share|improve this answer

My advice is to have a look at similar services. Services with open APIs usually publish their rate limits. For example Twitter.

While accessing an API is something different than what you are up to and this does certainly not guarantee that you "stay under the radar" but it might give you an idea.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.