3

We started monitoring our web server using Pingdom and found out that we have a downtime of a few minutes every Sunday at 0:00 UTC.

The test runs every minute and checks if a successful HTTP response (code 200) is returned on port 80. The test fails due to a timeout (no response after 30 seconds).

Here's what we've already checked – without success:

  • Since we run our webserver behind a load balancer, I've set the Pingdom test on the load balancer's public DNS and the webserver's public DNS in order to find out if there's a problem with the AWS load balancer – both tests return the same result

  • We set up Munin on our webserver. Everything looked fine even after the failure. Since the last failure lasted only 2 minutes I suppose Munin couldn't capture a potential problem (it only checks every 5 minutes)

  • I have checked /var/log/apache2/error.log and /var/log/syslog for suspicious entries

  • I have checked /etc/cron.weekly and /etc/crontab for suspicious entries

  • I have searched for files created or last-modified during 0:00 and 0:15 using this method:

    touch -t 201209020000 start
    touch -t 201209020015 end
    find / -newer start -and ! -newer end

    (nothing found)

Has anybody experienced a similar problem? Any proposals on how to find the reason for this behavior?

It's Ubuntu 10.04 LTS running on an AWS m1.large instance.

Thanks!

5
  • 2
    Did you ask AWS support if they could explain it?
    – StarNamer
    Sep 4, 2012 at 0:11
  • Not yet. Since I'm not sure if it's related to AWS, I want to figure that out first. I think I'll set up more alerts in Pingdom (on other ports and other machines) to find out if also other AWS machines experience this issue.
    – aaronk6
    Sep 4, 2012 at 6:35
  • Obviously, my thought is that if the server doesn't seem to be indicating a failure it's likely to be external, i.e. maintenance on the hosting systems or external routers. Does it only affect the web server (port 80) or are other services unavailable, e.g. SSH (22), etc. Does the server continue to respond to ICMP pings?
    – StarNamer
    Sep 5, 2012 at 12:57
  • It this a virtual system? I'm using kvm to save a machine state and take an lvm snapshot for backups... Sep 30, 2012 at 22:03
  • The easiest way to see if it's a network problem or if the machine is actually experiencing the issue is to manually be logged into the machine at that time via SSH and pull up the site. That would narrow it down very quickly. Good 'ol lynx from command-line works if you don't have a GUI when shelling in.
    – Peter
    Oct 13, 2012 at 20:25

1 Answer 1

1

There are some reports out, that the update-apt-xapi process takes lot of cpu usage for couple of minutes. It runs on a weekly schedule. It can take your box down, if the regular load is also high. The command runs update-apt-xapian-index to update the index of software packages.

See few hints for workarounds here: http://empoccz.wordpress.com/2012/01/02/ubuntu-update-apt-xapi-takes-lot-of-cpu-usage-ii/ or https://askubuntu.com/questions/79481/is-100-cpu-usage-harmful-while-update-apt-xapi-runs

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .