Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a midsized server with 48GB of RAM and are attempting to import a list of around 100,000 opt-in email subscribers to a new list management system written in Perl. From my understanding, Perl doesn't have imposed memory limits like PHP, and yet we are continuously getting internal server errors when attempting to do the import. When investigating the error logs, we see that the script ran out of memory. Since perl doesn't have a setting to limit the memory usage (as far as I can tell) why are we getting these errors? I doubt a small import like this is consuming 48GB of ram.

We have compromised and split the list into chunks of 10,000, but would like to figure out the root cause for future fixes. This is a CentOS machine with Litespeed as the web server.

share|improve this question
2  
Silly question - 64bit OS or PAE? When you run "top" how much RAM does perl consume? –  Tom Newton Feb 29 '12 at 20:45
    
Sorry - new to debugging and wasn't sure what info was relevant. 64 bit OS and perl doesn't even show up in top. mail.cgi (The script being used) does show up and is using 25% CPU and .3% ram before the error occurs. –  Kevin Feb 29 '12 at 21:54
    
What is the error you are getting? How do you know its running out of memory? –  Patrick Mar 22 '12 at 22:44

2 Answers 2

Many factors could apply here (kernel memory parameters, selinux, CGI sanity checks by the web server), but ulimit is probably the most likely culprit; start with /etc/limits.conf. That said, running a batched import script as a CGI is probably not your best choice - even if it doesn't run out of memory, most web servers don't enjoy long-lived CGI scripts, and you may end up terminated anyway. That kind of script will run best from the command line, ideally with checkpointing or at least logs so you can see how far you've made it through the import in the event that the script dies before completion.

share|improve this answer

It's hard to debug without seeing code, but you may be reading the file to the memory, instead of processing it line by line using foreach $line (<FILE>). Also, if you add this to a variable or array, it's the same. Do all processing instead of the foreach. Even so, 100k emails should not be using 48Gb.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.