My server is running Ubuntu 12.04 LTS (precise) and I'm having a weird issue. The server is hosting a large website which is used for data collection. The website is in PHP and uses Zend Framework. The data is in a MySQL database. Part of that data (from surveys) is rendered into Excel files (using the PHPExcel library) every hour (cron job). This process (a PHP/Zend Framework script) is done for each client, each with a different data set and can sometimes take a lot of time (30+ minutes).
As soon as it hits the 30 minutes mark the process' state changes from R to D. What's weird is that processes in D state are usually "unkillable" but this one can be killed as any other process. Here is a sample output while the process is running normally:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www 16089 0.5 0.8 311640 31708 ? S 09:34 0:39 /usr/sbin/apache2 -k start
www 17635 0.6 0.6 305020 23396 ? S 10:52 0:18 /usr/sbin/apache2 -k start
www 18520 0.0 0.0 63104 1960 pts/0 S 11:32 0:00 su www
www 18521 0.0 0.1 23236 4516 pts/0 S 11:32 0:00 bash
www 18621 98.8 68.1 2665208 2416568 pts/0 R 11:33 10:48 php run.php -a hourly
www 18659 0.6 0.6 302848 22948 ? S 11:34 0:03 /usr/sbin/apache2 -k start
www 18876 0.0 0.0 18160 1244 pts/0 R+ 11:44 0:00 ps ux
You can see the process is resource-intensive: it has already been trimmed a lot and it's still undergoing surgery but I need it to complete so I can get the resulting Excel files. The process is logging a lot of its activities (~300k log for each Excel file), and the log files always end up abruptly. There is no error logged anywhere that I could find.
Here is the same process list just after the 30 minutes mark:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www 18034 1.0 0.7 312412 26632 ? S 11:13 0:33 /usr/sbin/apache2 -k start
www 18245 2.1 0.4 303656 16912 ? S 11:25 0:49 /usr/sbin/apache2 -k start
www 18520 0.0 0.0 63104 296 pts/0 S 11:32 0:00 su www
www 18521 0.0 0.0 23236 968 pts/0 S 11:32 0:00 bash
www 18621 96.3 85.0 3969192 3012596 pts/0 D 11:33 30:08 php run.php -a hourly
www 18659 0.4 0.5 302856 19136 ? S 11:34 0:08 /usr/sbin/apache2 -k start
www 19431 0.0 0.0 18160 1240 pts/0 R+ 12:04 0:00 ps ux
If left running the process will still consume resources but does nothing. Nothing more is logged.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www 18621 9.1 80.0 4030532 2835032 ? D 11:33 33:19 php run.php -a hourly
Note that the process' state changes after exactly 30 minutes, it will have more or less data processed depending on the server load. Interestingly, this only happens on the production server. My development server, Ubuntu 12.04 as well, does not have this problem.
Is there something, anything that would cause processes to run normally for a maximum of 30 minutes and then change their state to D? What could cause a PHP script to end up in D state?