0

Sorry in advance for the wall of text. I'm having a problem which is leaving me with very few clues as to what might be going on. Also, I apologize if ServerFault isn't the ideal place for this question, but I don't think StackOverflow would be appropriate because it doesn't appear to be a problem with my code per se, so much as me just overwhelming the system somehow with my test setup.

Background

I'm working on a kernel driver for a PCIe board. The machine I'm running on has something like 24 cores and a fairly large amount of RAM. It's running RHEL 5.7 with Linux kernel 2.6.18. Sorry, I don't know the exact specs off-hand because it hasn't mattered up until now, and I didn't think to check before I left work today. It's a development machine and my driver (and associated test utils) is the only thing running on it. I won't go into detail about the device or the driver, because those don't seem to be the cause of the problem, but basically the device has 4 separate RS-232 serial interfaces on it, and my driver creates four character devices, with corresponding device nodes in /dev/. The driver also creates numerous sysfs attributes. Some of the attributes are for showing/setting various registers on the board, and the other attributes are actual telemetry data being parsed out of the data received on one of the serial ports. Edit: Come to think of it, the telemetry data sysfs attributes are being created and managed via the hwmon kernel API. Not sure if that might have anything to do with it, but at this point, I have no idea which details may turn out to be important.

In an attempt to stress the driver and have everything running at the max, I opened a slew of terminal windows: Four windows running hexdump on each of the character devices, another four windows continuously dumping the contents of the sysfs attributes representing the device register for each of the serial ports, and another window continuously dumping the telemetry-related sysfs attributes. I also had a window running top, and another running watch 'dmesg | tail -40' to view whatever debug info my driver generates (which isn't much - just things like open/close of the chardevs, any errors, etc).

Problem

After about 15-20 minutes of running, ALL of my terminal windows just suddenly disappear, all at once. No error popups, and dmesg shows nothing except my driver reporting that the character devices have been closed. The system remains stable. I am able to open new windows, etc. And my driver seems to just keep humming right along as well as if nothing had happened. From the driver's perspective, it just looks like all four hexdump processes just exited all at once, and there were no errors reported by my close function.

I've been searching Google, forums, and sites like this, but haven't found anything remotely close to what I'm seeing yet. It's as if some error condition occurred that causes Linux to decide that it needs to kill all my processes at once. I was thinking perhaps some type of resource exhaustion, like memory, file descriptors, etc, but utilities like top, vmstat, lsof, iostat (not sure how that would relate to anything but I checked it anyway) don't seem to show anything unusual. The system has something like 20 CPU cores and gobs of RAM, and with all that stuff running, it hardly seems to break a sweat. That is, until all my processes just die outright.

I'm at a loss to think of what to check next. I'm thinking maybe monitoring several dozen sysfs attributes all at once, dumping them all as fast as possible might not be something sysfs was designed for, but the system doesn't appear to be giving me any indication that there is a problem with that, and my driver seems to have no trouble keeping up with the constant onslaught of sysfs "show" calls, while still managing to not skip a beat with the character devices (the data all looks good throughout).

One other thing I should mention (even though I'm not sure how it might be relevant) is that I've tried this in a slightly different configuration where two of the four serial ports had loopback connectors on them, and instead of running hexdump on those two character devices, I ran my own bit error rate tester utility. In all, I've tried this 4 times, and the result has been the same each time. After about 15 minutes (give or take 5 minutes), all the terminal windows close all at once, the driver keeps running just fine, and absolutely nothing at all unusual in the dmesg output.

Any ideas what might cause something like that?

TL;DR: All my terminal windows spontaneously close, with no apparent indication of what went wrong. System remains stable, and nothing unusual in dmesg output. This is on RHEL 5.7.

4
  • Yeah it is TL;DR and the tl;dr doesn't help. There are other logs you should probably read too.
    – user9517
    Jul 18, 2014 at 7:04
  • @Iain Thanks. I'll go read some "other logs".
    – Mike Holt
    Jul 18, 2014 at 8:36
  • That's the spirit.
    – user9517
    Jul 18, 2014 at 8:40
  • what gives this: 'echo $TMOUT' from a terminal window?
    – tonioc
    Jul 18, 2014 at 8:55

2 Answers 2

2

What exact terminal are you using? You know that a few of the fancy GUI terminals included with desktop managers are actually multi-threaded process. So there is a single container app, which contains all the child terminals. This is used to give you features like a window with terminals in tabs.

If there is some bug in a single terminal that crashes the master process, you lose everything. You could try using xterm instead of whatever terminal it was that you were using. Or make sure you invoke your terminal in a way that each one is a separate process, see the man page for your terminal.

1

check /var/log/ , read every logfile there for similar timings to the closing of the terminals.

If you cannot find anything useful or related in there try another terminal solution for example instead of kconsole xterm

As an alternative testing increase the buffer size of the terminal itself. (depending which exactly you use might be a bit tricky, but google should help)

I had a similar looking issue a few years back with kconsole not showing anymore data while having tail -f running. For me it turned out to be a buffer error.

5
  • It did occur to me to read every bit of info in /var/log, and I will definitely do that. The reason I posted here is that I'm hoping to find someone who's actually seen stuff like this happen before. Having 10+ terminal windows simultaneously just go "poof" and disappear with no apparent cause, and no error, and no noticeable problems with the system either before or after, is just not something you see every day. After doing some more reading, I'm beginning to think that the kernel decided to kill my processes for some reason...
    – Mike Holt
    Jul 18, 2014 at 8:41
  • I've read about the OOM Killer bit of the Linux kernel (kills select processes if the system gets too close to running out of virtual memory), but none of my processes do a whole lot of memory allocation (most of them don't do any dynamic allocation whatsoever), and vmstat showed nothing out of the ordinary prior to the killing of the processes. Nor was there any sort of slow-down or unresponsiveness. Just one second everything is humming right along, and not even appearing to stress the system, and the next...poof.
    – Mike Holt
    Jul 18, 2014 at 8:44
  • So I think perhaps the kernel applies similar kill logic for other types of resources besides memory. Most likely something related to the constant accesses to dozens of sysfs attributes. After I check everything in /var/log, I'll run it again but without hammering sysfs, and see if it prevents my processes from being killed.
    – Mike Holt
    Jul 18, 2014 at 8:48
  • it might be kernel related, i cannot comment on that due to lack of experience, however as i said,speculation though, the terminal window buffer might be full and/or breaks for some reason. Jul 18, 2014 at 8:51
  • On 10+ terminal windows, all at exactly the same time? I'll look into that, but it seems unlikely. Still, thanks for the help.
    – Mike Holt
    Jul 18, 2014 at 15:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.