up vote 0 down vote favorite
share [g+] share [fb]

If a server is monitoring thousands of endpoints (http, https, ping, etc) and these checks are done every minute, what o/s issues should I be researching?

1K+ requests per minute is what I want to build for.

I will have to make sure file handles are tweaked.

What about open sockets? waiting sockets?

For example, I may make a http request to check uptime for a website, and it may take 50-100ms to respond, so if I have 1000 checks going on for this minute, things can pile up etc.

link|improve this question
50% accept rate
1  
Why would you want or need to check your monitors every minute? Are you going to respond within a minute of a failure? My suggestion would be to set a more reasonable and realistic time interval, based on the service or component you're monitoring. For instance, I might ping a router every 5 minutes, check the availability of a service every 15 minutes, and check disk space every 60 minutes. – joeqwerty Feb 2 at 19:19
well its up you really, but I see other monitoring services giving a 1 minute check interval so it isn't uncommon. – codecompleting Feb 2 at 21:48
1  
It's important to distinguish the check frequency from alerting thresholds. Doing checks every minute gives very valuable information on the status of your systems, but not necessarily something you need to immediately act on. We've had customers that switch to 60-second checks only to discover that they've been having lots of short outages that they didn't even know about. Not something you want to get paged about every time, but having that aggregate information lets you fix underlying problems that your customers are definitely noticing. – Jason Abate Feb 5 at 17:46
feedback

1 Answer

up vote 1 down vote

1000 checks per minute on a modern server shouldn't be a problem as long as your code that's performing the checks is setup to time out connections to machines that aren't responding. For example, if you set your connections to time out after 20 seconds then at most you'll have 333 open connections at any one time. This is well below any socket or filesystem limits for a default Linux install. If you get to a higher volume, maybe 20-30x that rate then you'll need to start worrying about tuning the system configuration.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.