Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running a Ubuntu Linux server with Apache, wsgi, django and mysql on it. Recently something happened and the wsgi processes frooze up. Restarting apache solved the problem. As with many live systems, it's better to get the system back up and running rather than poking around. However we are having trouble diagnosing the problem, since everything looked fine, and we don't know the full state of the problem now.

Is there any tool/command (on linux/debian/ubuntu (or any other *nix flavour, I'm fine with compiling any command)) that, when invoked, will write to a file some details about the state of the system as it is now? If/When this happens again, we can just run this command, then get down to fighting some fires (restarting apache/the server etc.), and then later, we can try to diagnose the problem.

Wishlist of things it would record:

  • CPU status (and various types)
  • Process list & various detailed
  • Details of file system usage
  • List of all open files (and what process has them, etc.)
  • List of all open internet connections
  • (If possible) Details of what our mod_wsgi processes are doing
  • MySQL status: current queries that are being run, etc.
  • (maybe) run strace on apache/mysql/mod_wsgi for a few seconds to collect some data of what they are doing, and save this to a file.
  • Anything else I'm forgetting?

In theory this is a simple set of commands and if no-one else has done this, then we'll just write our own scripts, but it would be better if we can use a proper tool.

share|improve this question

2 Answers 2

Work is being done in mod_wsgi 4.0 to better recover from the issue where all the WSGI request threads block on something, which is ultimately going to be the cause of this. How this then leads onto Apache as a whole blocking and why you may not get any logging out of Apache about it is mostly understood.

As part of the new recovery mechanism which has been implemented, mod_wsgi will prior to restarting the blocked daemon process attempt to log a minimal stack trace of each WSGI request thread so you can see where the code was blocked.

There is also work going on with tracking and reporting thread utilisation so you can know when request threads are starting to block in your code for some reason. This data will be able to be reported into a tool such as New Relic so you can chart it and then analyse it in conjunction with all the other information about web requests that the New Relic Python agent captures about your application.

New Relic also now has server monitoring, so it also can track a reasonable amount of information about the system as a whole, disk activity, network activity, cpu, processes etc etc. So, as a whole New Relic is one possible option for monitoring your system.

Overall, as time allows, a lot of work is being done on trying to make mod_wsgi easier to monitor and better able to automatically recover when your application starts to hang for one reason or another.

You might consider getting onto the mod_wsgi mailing list and watch for posts about this, or ask any specific questions about it you may have on the mailing list.

share|improve this answer
    
That's good to hear –  Rory Dec 23 '11 at 9:52

In general, I would hope that my monitoring system would pick up on the conditions that could lead to or trigger a crash. Having some sort of historical based monitoring solution would help you identify trends. Orca, Cacti, Munin... they all work well for this if it's a standalone system.

On the Red Hat/CentOS/Fedora side, we have the sosreport utility, which collects detailed hardware and process information.

Since you're on Ubuntu, that's not quite available. I think you can use Apport for some bits (if you can tie into your application), but everything else may be a combination of utilities: dmesg, dpkg, lshw, udevadm, dmidecode...

share|improve this answer
    
Yes, we have Munin set up, and that's helpful, but that doesn't help us know what (say) MySQL queries are being run, what open network connections there are, etc. –  Rory Dec 22 '11 at 21:11
    
Although I mentioned Ubuntu Linux, any *nix software will suffice, so long as I can get it running. As I said, it's either this or write our own. –  Rory Dec 22 '11 at 21:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.