Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

and thank you beforehand for reading this. I'm pretty new to linux and have just migrated from a shared hosting to a vps for my shop. I set up nginx with fpm and varnish, and some cron jobs to import a product feed from my providers. Up to this point, all is well.

But I enabled the cron jobs too close to each other and now when they ran two nights ago, seems like they're importing 15,000 products non stop. The load has gone past 30 and I only managed to get it down by limiting the child processes in the fpm config. Now the load is down, but it uses up all the RAM and the SWAP (with errors due to lack of RAM and the VPS rebooting, getting me nowhere fast), and I have tried several things: kill pid, killall php-fpm, kill -9 pid, kill -15 pid but none work. As soon as I restart the php-fpm service, the child processes reappear and collapse everything.

I've rebooted, killed them for hours, but nothing. I cannot use the VPS because on every reboot they respawn and pop up again, rendering my shop useless.

Researched and tried all the kill examples I could find, but nothing works. I know tis because of my lack of expertise, but I really need help here diagnosing and fixing this, its been over 48 hours, and I need to kill these processes and not have them reappear.

Again, thanks in advance. Cheers

Output of ps auxf:

php-fpm: master process (/usr/local/php/etc/php-fpm.conf)             
www       8970  3.0 28.8 1435348 550676 ?      R    22:58   0:05  \_ php-fpm: pool www                                                 
www       8976  2.3 18.2 845740 347228 ?       S    22:59   0:02  \_ php-fpm: pool www                                                 
www       8978  2.9 18.2 860492 348092 ?       S    22:59   0:02  \_ php-fpm: pool www                                                 
www       8983  2.4 16.6 494556 317888 ?       D    23:00   0:01  \_ php-fpm: pool www                                                 
www       8986  3.2  8.9 276684 170424 ?       R    23:00   0:00  \_ php-fpm: pool www 

And initial cron jobs (all disabled at my first chance):

# 0 23 * * * wget -q -O /dev/null "http://regalosoutletonline.com/wp-cron.php?import_key=REGALOS&import_id=92&action=trigger" #PerfumeriaStockFEED Trigger once every 24h
# 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59 * * * * wget -q -O /dev/null "http://regalosoutletonline.com/wp-cron.php?import_key=REGALOS&import_id=93&action=processing" #OciostockFEED Processing once every 2min

UPDATE:

I still didn't manage to find the culprit process, even after using pstree -p, which gave me the process tree and I could find the master:

| |-{mysqld}(9595) | -{mysqld}(9620) |-nginx(1779)-+-nginx(1781) |-nginx(1782) |-php-fpm(9588) |-php-fpm(9592) |-php-fpm(9594) |-php-fpm(9619)

But even killing the master did not fix the issue, as they reappeared.

Then I checked that all crons were deleted, not only disabled, and deleted the plugin and cleaned the db that still didn't help. Then edited my hosts file to block all calls to my provider's feed urls (downloading images), and that finally solved the present issue and got the site back up. I then singled out the one url that was the problem and things are back online now.

But the problem remains because as soon as I enable the url in the hosts file, even with the crons deleted and the plugin not present, the child processes shoot up again. Seems like there is some kind of registry for the php-fpm processes in the machine that makes them continue after a reboot or shutdown. So this is temporarily solved, but I cannot import from this one provider, even though everything is deleted related to it.

share|improve this question
    
that is strange, perhaps your cronjobs are being triggered continuously, what's your cronjob command? (or how is its frequency defined). Also look with ps auxf at what's calling the processes. –  LinuxDevOps Apr 4 at 20:57
    
Hello @LinuxDevOps and thanks for answering! The first thing I did after lowering the child processes, and thus the load, was to login via webmin and disable all the cron jobs. So in theory, there are no cron jobs running, just these processes that never die. - please see my output of ps auxf in my question above –  Sam Apr 4 at 21:02
    
I'm going to guess that Webmin didn't delete the cronjob that runs every other minute; ssh into the machine and check the cronjobs directly (crontab -l -u user where user is any of ls /var/spool/cron/crontabs/ and any entries in /etc/cron.d) –  LinuxDevOps Apr 4 at 21:14
    
If you disabled the cron job and still have the problem, then the cron job isn't the source of your problem. It sounds like you've configured php-fpm to use more resources than your VPS has. –  sciurus Apr 4 at 23:43
    
It's plain that your VPS is much too small for the workload that you want from it. Go get a larger one. –  Michael Hampton Apr 5 at 4:21

1 Answer 1

The right way to solve the problems is design your application so it that it doesn't need scheduled maintenance.

Admittedly that's sometimes a big hurdle, so second best solution would be to run the batch processing from a single-threaed daemon. Its not that hard. Since some folks out there don't have shell access and can't run a daemon, the third solution (i.e. the worst solution, and only marginally better than what you are doing just now) is to use lock files. e.g.

 <?php
 $running="/tmp/" . $argv[0] . ".pid";
 if (file_exists($running)) {
     die("already running running with pid " . file_get_contents($running));
 }
 file_put_contents($running, getmypid());
 register_shutdown_function('cleanup');

 function cleanup() 
 {
    global $running;
    unlink($running) || print "failed to unlink $running";
 }
 .....your code goes here....

If you get to the point where you are even trying to kill processes then you have a bug in your code you need to fix. If processes are not getting reaped after a kill -KILL (please don't use literals) then you need to learn a lot more about systems admin.

share|improve this answer
    
Thank you @symcbean for your helpful comment, unfortunately I cannot apply it to my issue at hand because a) I didn't write the code, its an expensive plugin and b)Its not scheduled maintenance, its a scheduled import of the products from a feed like I stated in my original post. But you are right, I really do need to learn a lot, lot more about being a sysadmin. –  Sam Apr 6 at 14:01
    
Even after trying to eradicate the problem by deleting the plugin, it persisted, making me think that if all the sources are gone, the problem has to be in the system itself. But then again, since it only started acting up after triggering the cron, the system couldn't be badly configured if it was working up to that point for 7 months already, all I did was add a new feed. –  Sam Apr 6 at 14:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.