Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm considering writing an application that aggregates information from a fairly popular website. This application would request information from this website at a set interval. I know this is a really hard question to even "ballpark" an answer to, but what might be a good safe interval to stay mostly "under the radar"? I'm a programmer first, a human being second, and a server admin a distant third, so my knowledge of what a server software like Apache can handle as far as server load with dynamic content is pretty basic.

I know this question is EXTREMELY open ended and the answer depends on many variables, but any related experiential knowledge being shared would be very much appreciated.

share|improve this question

closed as primarily opinion-based by masegaloeh, Jenny D, Andrew Schulman, mdpc, HBruijn May 21 '15 at 6:53

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

No "right" answer == community wiki. – womble Oct 27 '09 at 21:06
up vote 2 down vote accepted

First, second and third, I would see if the site has an API. Fourth, I would see if the site has a Terms of Use Policy. Lastly, random numbers are your friend.

share|improve this answer
Servers with APIs usually also have TOS which prevent you from re-using the content. – Jay Stevens Oct 27 '09 at 22:08

If it is measured in seconds and a high traffic site, it shouldn't be that much of an impact. More important than your second+ interval is probably to ensure you are accepting compressed responses etc properly.

Although if you're really trying to be polite you should ask them for permission or for a copy of the data you want.

share|improve this answer

My advice is to have a look at similar services. Services with open APIs usually publish their rate limits. For example Twitter.

While accessing an API is something different than what you are up to and this does certainly not guarantee that you "stay under the radar" but it might give you an idea.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.