We've been attempting to debug load programs on our servers, and along the way we've been looking at some graphs, like this one:

Memory graph

(The white spaces are when the server got so hosed we couldn't collect Munin stats.) I was curious to know why committed memory persistently drops down at approximately 4AM every night. Is there something obvious I'm missing? It's not like we reboot all of our user applications during that time...

Extra. This server is a shared web host. We handle about two thousand active users, and offer them CGI, FastCGI, SSH in any language they could possibly want. This server is running Fedora 13, and part of an active cluster of five servers. We use the networked filesystem AFS. We support cron but no user cronjobs run from this server (we give those to a different server.)

  • Are there any services being started or reloaded at that point in time? – Zoredache Oct 21 '11 at 18:54
  • Not obviously. Our networked filesystem runs backups at 2AM, we rotate logs at around 3AM, if it's a user-service, they would have all needed to have decided to reload their applications simultaneously, which is pretty implausible. – Edward Z. Yang Oct 21 '11 at 18:58
  • Correction: our AFS backups are staggered over many hours. They also have no meaningful impact on client performance. – Edward Z. Yang Oct 21 '11 at 19:05

logrotate calls service httpd graceful, which kills any running FastCGI processes. Thanks for playing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.