Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I suspect this may be an unanswerable question, with the only solution being to rebuild, but I'll ask anyway.

Last night the some internal processes stopped working on the Smoothwall firewall at the office, exhibiting the following outward signs:

  • Stopped responding to SSH and HTTPS connection attempts from either side. Didn't reject them, just never responded.
  • Hung during an attempt to log on at the console - user name entered but the prompt never returned for the password
  • Client side Internet browsing failed because there was no response from squid

All "pass-through" traffic flowed fine. i.e. Email, DNS, PPTP, etc. traffic flowed uninterrupted in both directions. Inbound HTTPS to other servers also worked fine. From the user perspective only Internet browsing was affected. As this occurred outside office hours nobody was any the wiser. I only learned about it thanks to Nagios alerts.

There is absolutely nothing in the logs to indicate a problem and in fact all logging also ceased at the same time. There is nothing in the logs prior to the failure to indicate there was any kind of external reason for it. Pressing the reset button (the only option, as I was unable to log on) brought the system back up as usual, with no hint of problems in the last 4 hours.

What I'm looking for are theories which might explain why some systems failed without warning with no apparent adverse effect on traffic flowing through. Perhaps someone far more knowledgable them myself can recognise something common to the failed parts.

share|improve this question
    
Smoothwall Express or the full Smoothwall? Smoothwall express ships with almost no binaries that make troubleshooting easy, and it's almost impossible to get anything on there, because it requires MAKE and getting MAKE on there is a mission (can you tell I've done this before) and just not worth the effort. So if it's express, I don't like your chances of being able to do anything :( –  Mark Henderson Aug 28 '09 at 1:15
    
It's Express. I have the dev version running as a virtual machine, so I can build whatever is required and copy it over, but where to start? –  John Gardeniers Aug 28 '09 at 2:12

2 Answers 2

up vote 2 down vote accepted

Does sound vaguely hardware related - is unusual to have a live system (kernel obviously still NATing) but numerous processes dead - eg. SSH and squid. I would point the finger of suspicion at the disk myself - when I have seen smoothies do this kinda trick (and I have seen a few.. been working at SmoothWall for >6 years!) it has 9 times out of 10 been disk related.

share|improve this answer
    
Thanks Tom. I might image it across to another drive and put the current one through the diagnostic testing. I must admit that apart from human error (mine) the only failures I've ever had with Smoothies was the drive. Of course in previous cases it was total failure, not something as odd as what happened this time. –  John Gardeniers Aug 28 '09 at 11:18
    
I hate unexplained failures :( Good luck with your tests - you might want to get memtest86 on the case as well if you can afford a night's downtime. –  Tom Newton Aug 28 '09 at 13:38
    
Swapped out the drive and ran the tests. Just the one bad block but that's one more than I want. Thanks Tom. –  John Gardeniers Aug 31 '09 at 5:59
    
One's enough eh? Storage devices.. who'd trust the darn things? ;) –  Tom Newton Aug 31 '09 at 19:45

I have been having the same problem over the last couple of days. I am running Smoothwall Express 3.0. The thing locks up whenever I am downloading large files via torrent. I've read on the support forums that this may be caused due to too many open connections and therefore filling up all of the RAM. I tried changing one of the firewall settings to close these inactive connections every 2 minutes instead of every 2.5 hours as is the default. This seemed to help for a little while but the problem is reoccurring now. The machine is 500MHz with 192MB of RAM.

No matter what the problem is, the system should fail gracefully - not whatever it's doing currently. Time to try something new I think.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.