Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a dozen or so single board computers on a network running Debian (squeeze) and access them via ssh (ssh server is dropbear). To give an idea of the hardware of these computers they're 1.2 GHz x86 processors, 1GB of RAM and 4GB flash drives formatted as ext2 (I avoided ext3 to prevent the added flash write stress from journaling), there is also a swap partition on the drive.

Normally the setup I'm using works great and I can access all the computers. Every once in a while one will prevent access. What happens is I try to connect via ssh (putty) and it gives me the login prompt, I enter the username and password and it responds 'Access Denied' and it will also refuse any public key in ~/.ssh/authorized_keys. The credentials are correct as they worked previously. The computer responds to pings and putty recognizes the server public key, which implies to me the system is still running. Restarting the server fixes the problem and I can log in again. (I tried a temporary fix of putting shutdown -r now in the root crontab but this doesn't seem to reliably be run once the hang happens) Once I restart however there doesn't seem to be any information in any of the system logs to indicate what happened, the logs are simply empty for that time period, as if the system had crashed.

There is some custom software running on the system which appears to stop working (which is why I wanted to ssh to begin with). I'm assuming that this program is the source of the problems but I'm unsure of how it would cause it and how to debug what is happening.

The most likely explanation I can think of is that there is a memory leak in the other program that then prevents dropbear from spawning a new login shell (and crontab from executing shutdown) as there is not enough free memory. But looking at memory usage of the other (working) computers there doesn't seem to be any meaningful increase in memory to indicate a leak (unless it's a very big, fast acting and rare leak). I would think that when the OS ran out of memory it would restart the system or kill processes (the Linux kernel restarts right?). The other thing I wonder about is if the fact that they are running off a flash drive could have some effect, especially the swap partition (which I think I should remove to prevent wear of the flash), but the flash drives are young (~1 month) and I don't think that wear would be a factor yet.

Does anybody have an idea of what could cause these symptoms, if it could be done by a memory leak, or something else I haven't thought of. And does anybody know of a method to try to debug the problem and find out more information about what's going wrong?

share|improve this question
    
Sorry, I don't have an answer for you. But i guess same occurred here: I have had different debian machines (all i386, kernel 2.6.18, minimal system with just some small shellscripts running once in a while). They showed the same hangs without any known reason. No logs, no connection, ... They too ran from flash, some had swap, some not. I couldn't find any hints where to look for. If i find something new about this i will let you know ... –  m.sr Jun 1 '10 at 20:33
    
I/O errors maybe (how's the flash connected? via a disk controller?). Do you have console/serial logging? Remote syslog? Do they have anything useful? If the system is in a bad state, logs might not be making it to the local storage. –  James Jun 1 '10 at 21:53
    
the flash is connected on usb. I added remote syslog and waited for the problem to happen again. You're right, it seems there are I/O errors preventing access to the device, as this is in the syslog: [1177047.908344] EXT3-fs error (device sda1): ext3_find_entry: reading directory #96961 offset 0 –  choobablue Jun 21 '10 at 12:42

1 Answer 1

up vote 2 down vote accepted

It turns out that the problem had something to do with the specific flash drives that I was using. They had this special 'U3' junk on it that apparently can cause problems on linux if it's not completely uninstalled. I decided it would be better instead to switch to a more 'live' type install anyways. Now I transfer the root file system into ram at boot and run from that so the flash drive isn't critical for the system to continue running.

http://live.debian.net/manual/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.