1

My server is running Ubuntu 12.04 LTS (precise) and I'm having a weird issue. The server is hosting a large website which is used for data collection. The website is in PHP and uses Zend Framework. The data is in a MySQL database. Part of that data (from surveys) is rendered into Excel files (using the PHPExcel library) every hour (cron job). This process (a PHP/Zend Framework script) is done for each client, each with a different data set and can sometimes take a lot of time (30+ minutes).

As soon as it hits the 30 minutes mark the process' state changes from R to D. What's weird is that processes in D state are usually "unkillable" but this one can be killed as any other process. Here is a sample output while the process is running normally:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www      16089  0.5  0.8 311640 31708 ?        S    09:34   0:39 /usr/sbin/apache2 -k start
www      17635  0.6  0.6 305020 23396 ?        S    10:52   0:18 /usr/sbin/apache2 -k start
www      18520  0.0  0.0  63104  1960 pts/0    S    11:32   0:00 su www
www      18521  0.0  0.1  23236  4516 pts/0    S    11:32   0:00 bash
www      18621 98.8 68.1 2665208 2416568 pts/0 R    11:33  10:48 php run.php -a hourly
www      18659  0.6  0.6 302848 22948 ?        S    11:34   0:03 /usr/sbin/apache2 -k start
www      18876  0.0  0.0  18160  1244 pts/0    R+   11:44   0:00 ps ux

You can see the process is resource-intensive: it has already been trimmed a lot and it's still undergoing surgery but I need it to complete so I can get the resulting Excel files. The process is logging a lot of its activities (~300k log for each Excel file), and the log files always end up abruptly. There is no error logged anywhere that I could find.

Here is the same process list just after the 30 minutes mark:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www      18034  1.0  0.7 312412 26632 ?        S    11:13   0:33 /usr/sbin/apache2 -k start
www      18245  2.1  0.4 303656 16912 ?        S    11:25   0:49 /usr/sbin/apache2 -k start
www      18520  0.0  0.0  63104   296 pts/0    S    11:32   0:00 su www
www      18521  0.0  0.0  23236   968 pts/0    S    11:32   0:00 bash
www      18621 96.3 85.0 3969192 3012596 pts/0 D    11:33  30:08 php run.php -a hourly
www      18659  0.4  0.5 302856 19136 ?        S    11:34   0:08 /usr/sbin/apache2 -k start
www      19431  0.0  0.0  18160  1240 pts/0    R+   12:04   0:00 ps ux

If left running the process will still consume resources but does nothing. Nothing more is logged.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www      18621  9.1 80.0 4030532 2835032 ?     D    11:33  33:19 php run.php -a hourly

Note that the process' state changes after exactly 30 minutes, it will have more or less data processed depending on the server load. Interestingly, this only happens on the production server. My development server, Ubuntu 12.04 as well, does not have this problem.

Is there something, anything that would cause processes to run normally for a maximum of 30 minutes and then change their state to D? What could cause a PHP script to end up in D state?

1 Answer 1

1

Firstly, a process in D state is in uninterruptible sleep (usually IO).

Usually what causes a process to be put into this state is execution of a blocking system call.

You can see what the last call your process made using strace:

strace -fp <pid>

Because D state is usually caused by an IO Operation, it's also usefull to examine what files the process has open using lsof:

lsof -p <pid>
1
  • Sadly I cannot test the commands since I skirted the problem by changing my code. I believe the problem was that the script was using too much memory. The moment it would try to access more memory than was available on the system it would fall in D state. This is on a server with "adjustable" memory so I'm guessing the memory limit is somehow hard coded in the kernel or something similar and it is what was causing the process to fall into that state.
    – Technoh
    Oct 15, 2013 at 16:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .