1

We have a midsized server with 48GB of RAM and are attempting to import a list of around 100,000 opt-in email subscribers to a new list management system written in Perl. From my understanding, Perl doesn't have imposed memory limits like PHP, and yet we are continuously getting internal server errors when attempting to do the import. When investigating the error logs, we see that the script ran out of memory. Since perl doesn't have a setting to limit the memory usage (as far as I can tell) why are we getting these errors? I doubt a small import like this is consuming 48GB of ram.

We have compromised and split the list into chunks of 10,000, but would like to figure out the root cause for future fixes. This is a CentOS machine with Litespeed as the web server.

3
  • 2
    Silly question - 64bit OS or PAE? When you run "top" how much RAM does perl consume?
    – Tom Newton
    Feb 29, 2012 at 20:45
  • Sorry - new to debugging and wasn't sure what info was relevant. 64 bit OS and perl doesn't even show up in top. mail.cgi (The script being used) does show up and is using 25% CPU and .3% ram before the error occurs.
    – Kevin
    Feb 29, 2012 at 21:54
  • What is the error you are getting? How do you know its running out of memory?
    – phemmer
    Mar 22, 2012 at 22:44

2 Answers 2

2

It's hard to debug without seeing code, but you may be reading the file to the memory, instead of processing it line by line using foreach $line (<FILE>). Also, if you add this to a variable or array, it's the same. Do all processing instead of the foreach. Even so, 100k emails should not be using 48Gb.

0

Many factors could apply here (kernel memory parameters, selinux, CGI sanity checks by the web server), but ulimit is probably the most likely culprit; start with /etc/limits.conf. That said, running a batched import script as a CGI is probably not your best choice - even if it doesn't run out of memory, most web servers don't enjoy long-lived CGI scripts, and you may end up terminated anyway. That kind of script will run best from the command line, ideally with checkpointing or at least logs so you can see how far you've made it through the import in the event that the script dies before completion.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .