2

I suspect this may be an unanswerable question, with the only solution being to rebuild, but I'll ask anyway.

Last night the some internal processes stopped working on the Smoothwall firewall at the office, exhibiting the following outward signs:

  • Stopped responding to SSH and HTTPS connection attempts from either side. Didn't reject them, just never responded.
  • Hung during an attempt to log on at the console - user name entered but the prompt never returned for the password
  • Client side Internet browsing failed because there was no response from squid

All "pass-through" traffic flowed fine. i.e. Email, DNS, PPTP, etc. traffic flowed uninterrupted in both directions. Inbound HTTPS to other servers also worked fine. From the user perspective only Internet browsing was affected. As this occurred outside office hours nobody was any the wiser. I only learned about it thanks to Nagios alerts.

There is absolutely nothing in the logs to indicate a problem and in fact all logging also ceased at the same time. There is nothing in the logs prior to the failure to indicate there was any kind of external reason for it. Pressing the reset button (the only option, as I was unable to log on) brought the system back up as usual, with no hint of problems in the last 4 hours.

What I'm looking for are theories which might explain why some systems failed without warning with no apparent adverse effect on traffic flowing through. Perhaps someone far more knowledgable them myself can recognise something common to the failed parts.

Improve this question
2
  • Smoothwall Express or the full Smoothwall? Smoothwall express ships with almost no binaries that make troubleshooting easy, and it's almost impossible to get anything on there, because it requires MAKE and getting MAKE on there is a mission (can you tell I've done this before) and just not worth the effort. So if it's express, I don't like your chances of being able to do anything :(
    – Mark Henderson
    Aug 28, 2009 at 1:15
  • It's Express. I have the dev version running as a virtual machine, so I can build whatever is required and copy it over, but where to start?
    – John Gardeniers
    Aug 28, 2009 at 2:12

2 Answers 2

Reset to default
2

Does sound vaguely hardware related - is unusual to have a live system (kernel obviously still NATing) but numerous processes dead - eg. SSH and squid. I would point the finger of suspicion at the disk myself - when I have seen smoothies do this kinda trick (and I have seen a few.. been working at SmoothWall for >6 years!) it has 9 times out of 10 been disk related.

Improve this answer
4
  • Thanks Tom. I might image it across to another drive and put the current one through the diagnostic testing. I must admit that apart from human error (mine) the only failures I've ever had with Smoothies was the drive. Of course in previous cases it was total failure, not something as odd as what happened this time.
    – John Gardeniers
    Aug 28, 2009 at 11:18
  • I hate unexplained failures :( Good luck with your tests - you might want to get memtest86 on the case as well if you can afford a night's downtime.
    – Tom Newton
    Aug 28, 2009 at 13:38
  • Swapped out the drive and ran the tests. Just the one bad block but that's one more than I want. Thanks Tom.
    – John Gardeniers
    Aug 31, 2009 at 5:59
  • One's enough eh? Storage devices.. who'd trust the darn things? ;)
    – Tom Newton
    Aug 31, 2009 at 19:45
0

I have been having the same problem over the last couple of days. I am running Smoothwall Express 3.0. The thing locks up whenever I am downloading large files via torrent. I've read on the support forums that this may be caused due to too many open connections and therefore filling up all of the RAM. I tried changing one of the firewall settings to close these inactive connections every 2 minutes instead of every 2.5 hours as is the default. This seemed to help for a little while but the problem is reoccurring now. The machine is 500MHz with 192MB of RAM.

No matter what the problem is, the system should fail gracefully - not whatever it's doing currently. Time to try something new I think.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .