1

I'm not sure if this question belongs to ServerFault or StackOverflow, but since I'm guessing that I need to debug this problem serverside, I'm going with ServerFault.

The problem

We're running a shared webhosting server for some client of ours. Everything is running smoothly, except for one clients their website. Around 2 to 3 days a week, our monitor detects a brief downtime because apache is not serving the page within 30 seconds, but instead between 60 to 120 seconds. I checked one time with my own desktop to confirm: the website kept loading for 80 seconds and then suddenly loads. There is no increased load, no more requests than normal and the other websites on the server loads perfectly.

We had issues with a specific plugin earlier: this plugin made contact with the server from the author to confirm the license-key. When this server was not reachable, Wordpress couldn't continue loading and had the same symptoms as now. We noticed this because one day their server was down for a couple of hours and we had time to disable and enable all the plugins, one by one. According to the plugin author, the problems are solved now.

I have the strong feeling that we're looking at the same problem again, maybe with the same plugin and maybe not. But since the downtime is so brief (usually no more than 2 minutes), I have no idea how to debug this timeout error.

Things I've thought of

Normally I would disable the plugins one by one, but before I'm connected to the database to disable the plugins, the website is up again. Since there is no pattern in the downtime, I can't stay standby for when it happens. Apache logs don't show any errors: I can only see the request from users and see that there are no files served for some time.

My second thought was to run a stacktrace on the apache process. I'm pretty sure this would reveal where Apache is waiting on for so long. But since the server is getting more than 30 requests a minute, the logging file would become very large in a couple of hours, which would make it impossible for us to find the right requests.

Relevant server specifications

CentOS Linux release 7.0.1406 (Core)
Kernel 3.10.0-123.el7.x86_64

Apache/2.4.12 with mod_ruid2
PHP 5.4.38 (cli)
mysql Ver 15.1 Distrib 5.5.41-MariaDB, for Linux (x86_64) using readline 5.1

All compiled by DirectAdmin 1.48.3

Ideas?

Who could think of a good way to debug this very specific problem? Any help is greatly appreciated!

EDIT:

  • Slow query log doesn't report any slow queries at during the slow requests.
  • Is there a way to turn on debug logging for this plugin? Did that company write it? – Matt Oct 18 '15 at 23:35
  • No, unfortunately there isn't. Also, I'm not so sure if this plugin is really the cause of the time-out/long wait. It's just that this plugin had problems before, but there are 31 other plugins which, hypothetical, could be faulty as well. – BlueCola Oct 18 '15 at 23:45
  • Use a tool like smokeping to test connectivity with the author's licence server. When the problem occurs have a look back at the smokeping graph and see if the two correlate. That'll rule this out. It wouldn't surprise me if this is the issue. – Matt Oct 19 '15 at 2:11
0

If Apache is still reachable, i would grab first the extended status page to see what requests are being served right now. If there is a long running request you could even strace it, pid should be visible in status ( since you have mod_ruid2 i guess you run mod_php and prefork MPM, so a process would serve only a single request at a time ).

Maybe reconfigure Customlog, and log the time taken to serve the request, so later you can identify the slow requests.

Once you have the slow requests, see if can be reproduced . If yes, then its easier to debug, you can even add xdebug for PHP profiling/debugging .

Also see what MySQL queries are running at the time of hang, maybe its a MySQL slow query/locking problem.

Could also be a net API issue as you said.

And when you run out of all options, maybe just talk with the boss, and kick the user. Depending on how many other sites are on the server, the server health may be more important then the site itself.

  • Apache is still reachable, all other sites on the server are reachable as well, as I mentioned in my post. So we don't need to kick the user for the server health, because other websites aren't effected. When this problem occurs, not a single page of this particular website is reachable, so every single request is slow. Slow query log doesn't show anything unusual. As I also mentioned, the problem is usually not longer present than a few minutes. Before I'm logged in to the server and got the right PID, the problem is already gone. So I need some kind of monitoring that is continuously running – BlueCola Oct 19 '15 at 23:03
  • No single page is reachable, not even static files ? ( like domain.com/test.text ) weird. Well, you could add to cron a little script, which pulls a page from domain with a timeout of ex 10 sec. Then search for a string or something in the content you got, if you can't find the string, pull localhost/server-status, and save to somewhere taged with time ( /place/YEARMONTHDAY/HOURMINSEC/status.html ) . then later you can examine that. Same with other interesting things like mysql process list etc. – Sandor Marton Oct 20 '15 at 1:14
0

As I mentioned, we suspected that one of the plugins was the cause of the issue at hand. Earlier, when their license-server was down, our website was down too. They stated this issue was fixed with one of their last updates, but since we're having so much downtime, I doubted this.

We eventually debugged it the following way:

  • We made a strace of a normal request, see how the page is loaded.
  • If this plugin was the problem, it would probably make contact with the license sever via TCP port 80. We didn't thought of this before, but this did the trick for us: we blocked this port in IP-tables to simulate a time-out on the license server (make sure to whitelist 127.0.0.1 in IP-tables, so it doesn't get a permanent block).
  • We did a strace again and loaded the page: this time, it didn't load and got stuck. We closed the strace after a couple of seconds, and viewed the file.

The last line of the strace was a load of the file: /wp-content/plugins/[plugin-name]/[file-of-plugin].php. Apache couldn't get passed this plugin, until we deblocked port 80 again.

We deleted the plugin and haven't experienced any downtime since then. It's a very rare problem, but I hope that my answer can be helpful if someone else is experiencing the same problem.

Thanks for all the comments and answer. We really appreciate it, it really helped us to think about a solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.