Sorry in advance for the wall of text. I'm having a problem which is leaving me with very few clues as to what might be going on. Also, I apologize if ServerFault isn't the ideal place for this question, but I don't think StackOverflow would be appropriate because it doesn't appear to be a problem with my code per se, so much as me just overwhelming the system somehow with my test setup.
I'm working on a kernel driver for a PCIe board. The machine I'm running on has something like 24 cores and a fairly large amount of RAM. It's running RHEL 5.7 with Linux kernel 2.6.18. Sorry, I don't know the exact specs off-hand because it hasn't mattered up until now, and I didn't think to check before I left work today. It's a development machine and my driver (and associated test utils) is the only thing running on it. I won't go into detail about the device or the driver, because those don't seem to be the cause of the problem, but basically the device has 4 separate RS-232 serial interfaces on it, and my driver creates four character devices, with corresponding device nodes in /dev/. The driver also creates numerous sysfs attributes. Some of the attributes are for showing/setting various registers on the board, and the other attributes are actual telemetry data being parsed out of the data received on one of the serial ports. Edit: Come to think of it, the telemetry data sysfs attributes are being created and managed via the
hwmon kernel API. Not sure if that might have anything to do with it, but at this point, I have no idea which details may turn out to be important.
In an attempt to stress the driver and have everything running at the max, I opened a slew of terminal windows: Four windows running
hexdump on each of the character devices, another four windows continuously dumping the contents of the sysfs attributes representing the device register for each of the serial ports, and another window continuously dumping the telemetry-related sysfs attributes. I also had a window running
top, and another running
watch 'dmesg | tail -40' to view whatever debug info my driver generates (which isn't much - just things like open/close of the chardevs, any errors, etc).
After about 15-20 minutes of running, ALL of my terminal windows just suddenly disappear, all at once. No error popups, and
dmesg shows nothing except my driver reporting that the character devices have been closed. The system remains stable. I am able to open new windows, etc. And my driver seems to just keep humming right along as well as if nothing had happened. From the driver's perspective, it just looks like all four
hexdump processes just exited all at once, and there were no errors reported by my close function.
I've been searching Google, forums, and sites like this, but haven't found anything remotely close to what I'm seeing yet. It's as if some error condition occurred that causes Linux to decide that it needs to kill all my processes at once. I was thinking perhaps some type of resource exhaustion, like memory, file descriptors, etc, but utilities like
iostat (not sure how that would relate to anything but I checked it anyway) don't seem to show anything unusual. The system has something like 20 CPU cores and gobs of RAM, and with all that stuff running, it hardly seems to break a sweat. That is, until all my processes just die outright.
I'm at a loss to think of what to check next. I'm thinking maybe monitoring several dozen sysfs attributes all at once, dumping them all as fast as possible might not be something sysfs was designed for, but the system doesn't appear to be giving me any indication that there is a problem with that, and my driver seems to have no trouble keeping up with the constant onslaught of sysfs "show" calls, while still managing to not skip a beat with the character devices (the data all looks good throughout).
One other thing I should mention (even though I'm not sure how it might be relevant) is that I've tried this in a slightly different configuration where two of the four serial ports had loopback connectors on them, and instead of running
hexdump on those two character devices, I ran my own bit error rate tester utility. In all, I've tried this 4 times, and the result has been the same each time. After about 15 minutes (give or take 5 minutes), all the terminal windows close all at once, the driver keeps running just fine, and absolutely nothing at all unusual in the dmesg output.
Any ideas what might cause something like that?
TL;DR: All my terminal windows spontaneously close, with no apparent indication of what went wrong. System remains stable, and nothing unusual in dmesg output. This is on RHEL 5.7.