2

If you're looking for a challenge... you've found it.

Every day at between 07:01:30 - 07:03:00 my apache installation crashes. Just at that time. It runs fine any other time of the day. I've had this problem for about a year now, Googled on this for hours and tried a lot of things. My guess is something is scheduled around this time that causes it. So far I have not been able to find it.

Now you're going to say, oh, look at this link you lazy person! Just up the maxclients setting! Trust me, I have, but unfortunately, none of the solutions on this site or any other site have worked. Now to add to this, to me, mystery, the server runs absolutely fine all day. Just at ~07:01 it crashes with below error.

The error I get is: [Sun Jul 28 07:01:30 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting [Mon Jul 29 07:03:09 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting [Tue Jul 30 07:01:54 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting

Notice monday is slightly off. This is because, at 7:01:00 I did a service apache2 restart. It just made this crash come a little bit later. Huh? Maybe a fluke... I don't understand this at all.

I've used apachetop to check traffic. Just before getting this error, there aren't many users on the site, perhaps 10-20. During evenings, however, we can get hunderds of visitors at the same time. I've seen Apache handle up to 100 requests/sec! The problem is not at all that the server reached MaxClients because 7am is a really quiet time for the site. Access logs show nothing special. A few clicks here and there.

So in my eyes, it can't be a keepalive issue. This wouldn't cause a crash at nearly exactly the same time each day. It can't be an attack, because why would this be at 7am every day for a year? Also, the logs don't show any kind of traffic really.

This is what I've tried and checked: - Upping the maxclients setting (as high as 512. Yes, also ServerLimit). Making it much lower. Nothing works. - Reduced keepalive time in Apache config. - Moved all my resources (js, css, images) to another server with nginx using haproxy. It's not the amount of requests! - Upgraded Ubuntu to 12.04. - Tried every kind of tool to check my apache config - Moved daily cronjob an hour earlier in /etc/crontab. Still ~7:00. - Checked other users for crontabs. Nothing. - mod_status is not showing an abnormal amount of connections.

There is a haproxy installation before Apache. But I have tried without this, and Apache still crashes. Haproxy logs state that the server went down (I'm using httpcheck). I tried switching off this check, this just causes the site to not load at all (it times out).

Version info: I'm running Ubuntu 12.04 but also had this problem running 11.10. I'm using prefork. Server version: Apache/2.2.22 (Ubuntu) Server built: Jul 12 2013 13:37:10

No crontabs except for root: for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done1

Server has plenty of resources, with mostly gigabytes of RAM free.

Obviously I'm missing something here. Where else can Ubuntu schedule stuff? Is Apache doing something special around 7am that is not in the crontab?

Thanks!

~~~~ I can't answer my own question but it's solved! ~~~~

Reply: Aaaaaah I found it. Janne Pikkarainen got it right the first time. Apparently I was not using my eyes. Probably not the correct command with netstat. Because I just saw loads of connections... (sure, it's a busy time now, but that doens't explain 10 times more connections suddenly).

Without further ado, it was the database server. It's running a mysqldump every day at 7am. Because it's a different server it didn't even come up to me that I should check that one too.

I just tried to run that backup and soon enough requests start piling up. And bam! Apache throws error and stops responding. Stupid developer (me) is stupid :) So the fix was never service apache2 restart, the backup just ended and the server responded again. So actually, it did not crash at all.

Thanks everyone for all the help. Not such a mystery... I knew I missed something somewhere. I hope this answer helps someone, some day :)

7
  • Does your web application use some kind of big database? Perhaps that's being backed up (or otherwise maintained) every day around 07:00 and is inaccessible during that period. Yeah, would be weird that Apache2 would hit max processes if it's the silent time of day with only couple of clicks here and there, but perhaps the app performs some internal http requests or something ... Jul 30, 2013 at 5:41
  • 1
    What's in the access logs at that time? Have you looked at mod_status when it's happening? What time zone is the server in? Jul 30, 2013 at 5:43
  • Perhaps you have a log rotation script that's not playing nicely - check /etc/logrotate.d/ for any apache or http files, also take a look at /etc/logrotate.conf Jul 30, 2013 at 6:07
  • Don't forget to check /etc/crontab, /etc/cron.d for cron jobs.
    – user9517
    Jul 30, 2013 at 7:05
  • We do have a fairly big database running on a seperate phyiscal server, and an external site we do HTTP requests to. A queue is handled by a cronjob every 2 minutes. This queue is usually empty or near-empty in the morning, however. It was this morning as well. Curl is set not to reuse its connections. Because no jobs were running that were using the external site I think we can rule that out. The database is interesting. I will check the database server for cronjobs. The server timezone is in CEST (GMT+2 with DST). I double checked the cronjobs, I really can't spot any running.
    – Bart
    Jul 30, 2013 at 10:47

1 Answer 1

2

Really this should be a comment, but it's too long.

my apache installation crashes

Are you sure? Reaching maxclients implies that the program is running, "crashing" means that it has stopped (and should have logged this and dumped a core file). They are very different things. If it has stopped, where is the core file? Have you tried getting a stacktrace from it?

there aren't many users on the site, perhaps 10-20

Meaningless metric. How many established connections are there are at any one time? When it "crashes"? What's the server-status before a crash? Does this tally with netstat?

What do your system logs show?

So in my eyes, it can't be a keepalive issue.

I see nothing in your post to rule out keepalives as a contributory factor.

2
  • Apache stops responding. No clue if it crashes? No clue where to find a core dump. The only thing I see in the log is maxclients reached. I checked with netstat and 28 connections currently open. I will check again tomorrow morning but I have checked before and it was below 30 at that time too. Why it's extremely unlikely to be a keepalive issue in my opinion: This would not reoccur in the same timeframe of 1-2 minutes every day for a year. Also the connection amount is very low during the alleged "crash". And the keepalive is very short.
    – Bart
    Jul 30, 2013 at 10:41
  • I've never seen netstat report a connection as "open" - how many connections are there to ports opened by httpd and what state are they in? If you are serving your static content off a seperate server then you should already have switched off keepalives - doing so would go give you a far better indicator of whether it is a contributory factor. The log entries you have cited I would expect to see in the httpd error log, not in the system log.
    – symcbean
    Jul 30, 2013 at 11:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .