Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I run the perl script in screen (I can log in and check debug output). Nothing in the logic of the script should be capable of killing it quite this dead.

I'm one of only two people with access to the server, and the other guy swears that it isn't him (and we both have quite a bit of money riding on it continuing to run without a hitch). I have no reason to believe that some hacker has managed to get a shell or anything like that. I have very little reason to suspect the admins of the host operation (bandwidth/cpu-wise, this script is pretty lightweight).

Screen continues to run, but at the end of the output of the perl script I see "Killed" and it has dropped back to a prompt. How do I go about testing what is whacking the damn thing?

I've checked crontab, nothing in there that would kill random/non-random processes. Nothing in any of the log files gives any hint. It will run from 2 to 8 hours, it would seem (and on my mac at home, it will run well over 24 hours without a problem). The server is running Ubuntu version something or other, I can look that up if it matters.

share|improve this question
    
Running out of memory? Check the logs for oom killer messages. –  mattdm Feb 8 '11 at 18:13
    
Is it a script that runs in the current PTY, or is it written to daemonize itself? Even if you run some programs that daemonize from a PTY session, they will die when you log off. Try putting nohup before your command to run your script. Alternately, you could add a script in /etc/init.d to launch your script, which you could then start and stop with sudo service _name-of-Perl-script_ [start|stop|restart]. –  nesv Feb 8 '11 at 18:16
    
Not written to daemonize itself. Just outputs some monitoring data via the Curses mod. That's why I'm using screen at the moment, to avoid that. –  John O Feb 8 '11 at 18:19
    
Just checked the logs again. Nothing in messages, debug, dmesg, kern.log, or syslog. Am I missing any, should I be grepping for something specific? I mean, I don't think I'm using that much memory in the script... but there could be some leak I'm not recognizing. –  John O Feb 8 '11 at 18:25
    
Do the hosts have root on the server? Adding the signal traps below would be the best first step, and you should add timestamp logging if you don't have it already to make sure you know exactly how long the script is living and when it is being killed. If you get that far with no answer then more logging, and/or external monitoring will be the next step. –  daveadams Feb 8 '11 at 18:47

7 Answers 7

up vote 2 down vote accepted

Without much in the way of actual knowledge, I'd start looking in dmesg output or assorted syslogs if the OOM killer is running. If so, that's probably it.

share|improve this answer

Put in signal handlers for all the signals (TERM, SEGV, INT, HUP, etc) and have them log out whenever they are hit. It wont tell you what is sending the signal, but it will allow you to see what signal it is and perhaps ignore it.

$SIG{'TERM'} = $SIG{'INT'} = sub { print(STDERR "Caught SIG$_[0]. Ignoring\n"); };

That would print out when it caught a sigterm or sigint and then return control back to the program. Of course with all those signals being ignored, the only way to kill it would be to have the program itself exit, or to send it a SIGKILL which cant be caught.

share|improve this answer
    
I added the signal handling last night, per your suggestions. It's certainly harder to stop it for debugging now... and we can rule out anything quite so simple now. –  John O Feb 9 '11 at 16:06

I realize this isn't exactly an answer to the question you asked, so I apologize if it's somewhat off-topic, but: does your app really need to run continuously, forever? Perl is not the most resource-thrifty environment in the world, and while the overhead of interpreter start-up is not without its drawbacks, extremely long-running scripts can have troubles of their own - memory leaks, often at a level below your control, are the bane of the vanilla-perl developer's existence, which is why folks often mitigate those issues either by running in a more formally resource-conservationist sub-environment like Perl::POE, or by handing over the long-running listener part of the job to a front-end service like xinetd and only executing the perl component when work needs to be done.

I run several perl scripts which run continuously reading and processing the output of our (considerably large) central syslog stream; they suffer from terrible, inexplicable "didn't free up memory despite pruning hash keys" problems at all times, and are on the block to be front-ended by something better suited to continuous high-volume input (an event queue like Gearman, for example), so we can leave perl to the data-munging tasks it does best.

That went on a bit; I do apologize. I hope it's at least somewhat helpful!

share|improve this answer
    
Work needs to be done once a second, by requesting time-sensitive json off a third party webserver. That data is chucked into a database for analysis, and we think it might be worth as much as $5000 a month. We want to keep the opportunity to ourselves, so I can't say much more than that... but if we can get a month or two's worth of data, more or less continuous, we should have what we need. We'll deal with the longer term memory leak problems by periodically restarting it should we need that. –  John O Feb 8 '11 at 22:17

Syslog is the first thing to consult. If it isn't sufficient…

You can't determine who sends a signal to a process. It could be another process, it could be the kernel, etc. Short of involving the very recent perf framework some guesswork is involved.

However, you can set up some better monitoring. The atop package, in debian/ubuntu, sets up a service that will log system load and per-process activity (disk, memory, cpu). You can then consult those logs and get a feel of what was happening at the time the process crashed.

Crash course: sudo atop -r, navigate with t and T, type h to get help about the various visualisations.

Also consider adding a signal handler that dumps pstree to a temporary file.

share|improve this answer

Likely you are running into resource limits. For example CPU time. Try ulimit -a to check. If it's only a soft-limit, set in a login script then you can fix it with, eg, ulimit -t unlimited. If it's a hard limit, as is set for example for regular users on OpenBSD and other OSs, then you'll have to override.

share|improve this answer

Until you nail the issue, running the script with

nohup scriptname

can help. If it still crashes, examine the nohup.out file.

And if nothing mentioned here helps, I'd try to use strace/ltrace to see what system or library calls script was doing before failure, but they generate a LOT of output.

share|improve this answer

In a previous life I found a DEC Ultrix box that had a very clever cron job which looked for all processes with more than 1 CPU hour and killed them. Which was why the nightly batch report job died every night.

Any clever cron jobs/scripts that might be killing it? Or it might be another performance tuning parameter or somethng like the ulimit answer already given.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.