0

I have a bunch of websites running on a year-or-so old Fedora installation. Tonight, all the sites stopped responding. I've been trying to log in to investigate, but can't, and I'm clueless about what to do.

Pinging the server works, response times are decent, ~200ms.

When I shh, the "Last login: ..." text appears, but nothing more. However, if I leave it at that, and open another ssh session, the first one actually logs in (I've got it set up to use keys instead of a password), and the new one gets stuck at "Last login: ...". I can repeat this as many times as I've tried - opening a new ssh session always makes the previously opened log in.

When logged in, the only command I can consistently run without the connection(?) freezing, is ls in my home dir. I have been able to sudo su once, but every other time things freeze at that, as does as running any other command (top, killall -9 httpd, ps aux are some I've tried).

Does this pattern sound familiar to anyone? Any tips on how to proceed would be greatly appreciated!

2 Answers 2

1

This sounds like your system is under very heavy load. It's most likely swapping a lot or otherwise using disk I/O.

What I suspect that happens is that for some reason*) your site suddenly receives a flood of http requests to some heavy dynamic page, which makes Apache consume all of your server resources. The real reason could very well also be a heavily loaded and slowly responsing database server, which would ramp up the page load times at your web server, which would make Apache consume more processes, which would lead to server overload, which would lead to situation you are observing.

Are your sites usually having a lot of traffic? A ballpark figure would be nice -- are we talking about thousands of hits a day, tens of thousands hits a day, millions...?

Do you have any resource graphs of your server? Something like snmpd+mrtg or Cacti graphing CPU, memory, disk usage? If, are there any spikes? Or if you have sysstat package installed, then you might have sar snapshots available -- try command sar. It would save various information about your system every 10 minutes or so and through it reports you might see what was going on.

And is this Fedora installation a separate physical server or is it somehow depending on the VMware environment you are talking about? Could it be that some part of your VMware infrastructure gets overloaded every now and then and THAT is the real reason for your problems?

Also, after the dust settles, I would investigate the logs to see if there were any signs of kernel yelling about OOM (Out of Memory) Killer, or if httpd logs would look suspicious.

Usually if I'm unable to see anything suspicious in httpd logs by my own eye, I will let Webalizer or similar analyzer to run through that log and see if any URL's are popping in the output. Or I just pipe the log to apachetop and see what was happening.

*) Reason can be a DoS attack, a badly behaving PHP script, an erroneous .htaccess file making your ErrorDocument page to recursively call itself, Slashdot/reddit effect, or something completely else.

1
  • Thanks for all the insight. I never got a good answer to what caused this from the friend who owns the server, but this incident actually caused me to move my stuff to a different type of server (which I had been postponing for a long time). I'll be playing with some of your suggestions there so I'm well prepared for the next crash ;-). To answer your question - the sites don't get huge amounts of visits, but they use a lot of bandwidth (~40GB/day, files going back and forth), and there are server side processes operating on the files which at times consume lots of CPU. Nov 15, 2010 at 19:52
0

Oh, I got the exactly same problem yesterday morning with my virtual server (CentOS 5.4 run via VMWare). When I try to ssh in verbose mode, it hangs at:

debug1: Entering interactive session.

If I open another ssh session, the first one continues with some debug info and gets stuck in:

debug2: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug2: channel 0: request shell confirm 1
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login:

and after few minutes, it actually gives me a command prompt.

But in the afternoon, I can ssh normally. So strange to me and I cannot find out any info on Google.

PS: I also use public key instead of password.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .