I suspect this may be an unanswerable question, with the only solution being to rebuild, but I'll ask anyway.
Last night the some internal processes stopped working on the Smoothwall firewall at the office, exhibiting the following outward signs:
- Stopped responding to SSH and HTTPS connection attempts from either side. Didn't reject them, just never responded.
- Hung during an attempt to log on at the console - user name entered but the prompt never returned for the password
- Client side Internet browsing failed because there was no response from squid
All "pass-through" traffic flowed fine. i.e. Email, DNS, PPTP, etc. traffic flowed uninterrupted in both directions. Inbound HTTPS to other servers also worked fine. From the user perspective only Internet browsing was affected. As this occurred outside office hours nobody was any the wiser. I only learned about it thanks to Nagios alerts.
There is absolutely nothing in the logs to indicate a problem and in fact all logging also ceased at the same time. There is nothing in the logs prior to the failure to indicate there was any kind of external reason for it. Pressing the reset button (the only option, as I was unable to log on) brought the system back up as usual, with no hint of problems in the last 4 hours.
What I'm looking for are theories which might explain why some systems failed without warning with no apparent adverse effect on traffic flowing through. Perhaps someone far more knowledgable them myself can recognise something common to the failed parts.