How to find what an Apache httpd process is doing?

Question

I have a web application that is used nearly constantly by 5 - 10 users. Between 1 and 3 times a day it will grind to a halt.

When this happens I can see the following in TOP:

last pid:  1744;  load averages: 15.50, 45.91, 31.79                                                        up 0+01:04:01  14:47:59
357 processes: 1 running, 351 sleeping, 5 waiting
CPU:  0.2% user,  0.0% nice,  2.8% system,  2.4% interrupt, 94.6% idle
Mem: 1612M Active, 180M Inact, 174M Wired, 8012K Cache, 103M Buf, 108K Free
Swap: 4096M Total, 2104M Used, 1992M Free, 51% Inuse, 40M Out

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1384 www           1  44    0   156M  3256K pfault  0   0:36  0.00% httpd
 1477 www           1  44    0   154M  8008K pfault  1   0:35  0.00% httpd
 1053 mysql       161  44    0   199M 54108K umtxn   0   0:27  0.00% mysqld
 1441 www           1  44    0   154M     0K WAIT    0   0:26  0.00% <httpd>
 1521 www           1  44    0   164M     0K sbwait  1   0:20  0.00% <httpd>
 1519 www           1  44    0   156M  7832K pfault  0   0:19  0.00% httpd
 1520 www           1  44    0   138M  8724K select  1   0:19  0.00% httpd
 1527 www           1  46    0   138M     0K accept  0   0:17  0.00% <httpd>
 1422 www           1  44    0   138M  3096K pfault  0   0:16  0.00% httpd
 1290 root          1  44    0   130M  9060K select  1   0:15  0.00% httpd
 1529 www           1  44    0   164M  3740K pfault  1   0:15  0.00% httpd
 1532 www           1  44    0   154M 22704K pfault  0   0:15  0.00% httpd
 1530 www           1  44    0   164M     0K sbwait  0   0:14  0.00% <httpd>
 1525 www           1  44    0   156M     0K sbwait  1   0:14  0.00% <httpd>
 1531 www           1  44    0   138M  8168K pfault  1   0:13  0.00% httpd
 1528 www           1  44    0   164M     0K sbwait  0   0:13  0.00% <httpd>
 1533 www           1  44    0   156M  3252K pfault  1   0:12  0.00% httpd
 1534 www           1  44    0   156M  3592K pfault  1   0:12  0.00% httpd
 1544 www           1  44    0   156M  4620K pfault  1   0:12  0.00% httpd
 1540 www           1  44    0   156M  4340K pfault  1   0:12  0.00% httpd
 1543 www           1  44    0   156M  4744K pfault  0   0:11  0.00% httpd
 1546 www           1  44    0   154M 11764K pfault  1   0:11  0.00% httpd
 1547 www           1  44    0   154M  8176K pfault  0   0:11  0.00% httpd
 1545 www           1  44    0   154M  8660K pfault  1   0:11  0.00% httpd
 1541 www           1  44    0   156M     0K WAIT    1   0:10  0.00% <httpd>
 1558 www           1  44    0   158M 19996K pfault  0   0:10  0.00% httpd
 1559 www           1  44    0   154M 10324K pfault  0   0:10  0.00% httpd
 1555 www           1  44    0   154M 18796K pfault  1   0:09  0.00% httpd
 1569 www           1  44    0   156M  6208K pfault  0   0:09  0.00% httpd
 1566 www           1  44    0   154M 10996K pfault  0   0:09  0.00% httpd
 1561 www           1  44    0   156M 15724K pfault  1   0:09  0.00% httpd
 1562 www           1  44    0   154M  8076K pfault  0   0:09  0.00% httpd
 1572 www           1  44    0   154M 11308K pfault  0   0:09  0.00% httpd
 1571 www           1  44    0   154M     0K sbwait  0   0:08  0.00% <httpd>
 1553 www           1  44    0   154M     0K sbwait  1   0:08  0.00% <httpd>
 1567 www           1  44    0   154M 13108K pfault  1   0:08  0.00% httpd
 1565 www           1  44    0   152M 17284K pfault  0   0:08  0.00% httpd
 1602 www           1  44    0   156M  4232K pfault  1   0:08  0.00% httpd
 1548 www           1  44    0   154M  8948K pfault  0   0:08  0.00% httpd
 1577 www           1  44    0   156M 15732K pfault  0   0:08  0.00% httpd
 1573 www           1  44    0   156M 16244K pfault  0   0:08  0.00% httpd
 1554 www           1  44    0   154M     0K sbwait  0   0:08  0.00% <httpd>
 1585 www           1  44    0   156M     0K WAIT    1   0:08  0.00% <httpd>
 1552 www           1  44    0   154M     0K sbwait  0   0:08  0.00% <httpd>
 1575 www           1  44    0   154M 11956K pfault  0   0:08  0.00% httpd
 1570 www           1  44    0   156M 16260K pfault  1   0:08  0.00% httpd
 1583 www           1  44    0   156M 15368K pfault  1   0:08  0.00% httpd
 1580 www           1  44    0   154M 11112K pfault  0   0:08  0.00% httpd
 1600 www           1  44    0   156M  4248K pfault  1   0:08  0.00% httpd
 1589 www           1  44    0   154M     0K sbwait  1   0:08  0.00% <httpd>
 1578 www           1  44    0   154M     0K sbwait  0   0:08  0.00% <httpd>
 1557 www           1  44    0   154M     0K sbwait  1   0:07  0.00% <httpd>
 1617 www           1  44    0   154M 11244K pfault  1   0:07  0.00% httpd
 1574 www           1  44    0   154M  8360K pfault  1   0:07  0.00% httpd
 1560 www           1  44    0   154M 10156K pfault  0   0:07  0.00% httpd
 1603 www           1  44    0   156M  4968K pfault  0   0:07  0.00% httpd
 1590 www           1  44    0   154M 11860K pfault  0   0:07  0.00% httpd
 1592 www           1  44    0   156M 14280K pfault  1   0:07  0.00% httpd
 1581 www           1  44    0   156M  6412K pfault  0   0:07  0.00% httpd
 1586 www           1  44    0   154M 11176K pfault  0   0:07  0.00% httpd
 1608 www           1  44    0   156M  7256K pfault  1   0:07  0.00% httpd
 1568 www           1  44    0   154M 10784K pfault  1   0:07  0.00% httpd
 1591 www           1  44    0   154M 13256K pfault  0   0:07  0.00% httpd
 1610 www           1  44    0   156M  3440K pfault  0   0:07  0.00% httpd
 1650 www           1  44    0   156M     0K sbwait  1   0:07  0.00% <httpd>
 1616 www           1  44    0   154M 13724K pfault  0   0:07  0.00% httpd
 1611 www           1  44    0   154M 10084K pfault  0   0:06  0.00% httpd
 1564 www           1  44    0   154M 12220K pfault  1   0:06  0.00% httpd
 1607 www           1  44    0   156M     0K sbwait  0   0:06  0.00% <httpd>
 1601 www           1  44    0   154M 13728K pfault  1   0:06  0.00% httpd
 1634 www           1  44    0   156M     0K sbwait  0   0:06  0.00% <httpd>
 1606 www           1  44    0   156M     0K sbwait  1   0:06  0.00% <httpd>
 1626 www           1  44    0   156M  3340K pfault  1   0:06  0.00% httpd
 1633 www           1  44    0   156M     0K sbwait  0   0:06  0.00% <httpd>
 1579 www           1  44    0   154M  8508K pfault  1   0:06  0.00% httpd
 1596 www           1  44    0   156M     0K sbwait  1   0:06  0.00% <httpd>
 1584 www           1  44    0   156M  2388K pfault  1   0:06  0.00% httpd
 1641 www           1  44    0   156M  3060K pfault  0   0:06  0.00% httpd
 1632 www           1  44    0   156M  3144K pfault  1   0:06  0.00% httpd
 1657 www           1  44    0   156M     0K sbwait  1   0:06  0.00% <httpd>
 1630 www           1  44    0   156M  6872K pfault  0   0:06  0.00% httpd
 1609 www           1  44    0   156M     0K sbwait  1   0:06  0.00% <httpd>
 1661 www           1  44    0   156M  5688K pfault  0   0:05  0.00% httpd
 1612 www           1  44    0   156M 17816K sbwait  1   0:05  0.00% httpd
 1670 www           1  44    0   156M  4904K pfault  0   0:05  0.00% httpd
 1576 www           1  44    0   154M 13980K pfault  1   0:05  0.00% httpd
 1673 www           1  44    0   156M     0K sbwait  0   0:05  0.00% <httpd>
 1645 www           1  44    0   156M  3312K pfault  0   0:05  0.00% httpd
 1597 www           1  44    0   156M  6004K pfault  1   0:05  0.00% httpd
 1587 www           1  44    0   156M  6008K sbwait  1   0:05  0.00% httpd
 1628 www           1  44    0   156M     0K sbwait  0   0:05  0.00% <httpd>
 1660 www           1  44    0   156M  6180K pfault  0   0:05  0.00% httpd
 1647 www           1  44    0   158M     0K sbwait  1   0:05  0.00% <httpd>
 1582 www           1  44    0   156M     0K sbwait  0   0:05  0.00% <httpd>
 1664 www           1  44    0   156M     0K sbwait  0   0:05  0.00% <httpd>
 1651 www           1  44    0   156M     0K WAIT    1   0:05  0.00% <httpd>
 1674 www           1  44    0   156M  7560K pfault  0   0:05  0.00% httpd
 1631 www           1  44    0   156M  6464K pfault  0   0:05  0.00% httpd
 1655 www           1  44    0   158M     0K sbwait  0   0:05  0.00% <httpd>
 1669 www           1  44    0   156M     0K sbwait  0   0:04  0.00% <httpd>
 1659 www           1  44    0   156M  5936K pfault  1   0:04  0.00% httpd
 1663 www           1  44    0   156M  5828K pfault  0   0:04  0.00% httpd
 1672 www           1  44    0   156M  5764K pfault  0   0:04  0.00% httpd
 1613 www           1  44    0   156M  5784K pfault  1   0:04  0.00% httpd
 1667 www           1  44    0   156M  3696K pfault  1   0:04  0.00% httpd
 1662 www           1  44    0   156M  3336K pfault  0   0:04  0.00% httpd
 1671 www           1  44    0   156M  5036K sbwait  0   0:04  0.00% httpd
 1675 www           1  44    0   156M  6228K pfault  1   0:04  0.00% httpd
 1629 www           1  44    0   156M  5788K pfault  0   0:04  0.00% httpd

Is there any way to find what the httpd processes are getting stuck doing?

Answer 1

First, as ETL wrote, enable mod_status with ExtendedStatus On to see current requests and their states, execution times... etc. You can also use an apachetop to get information which file is most popular. If it's not enough information to you, as timmeyh wrote, you can use strace but I will suggest -ff and -p switches (make sure that you specified main apache proccess id).

-ff switch

If the -o filename option is in effect, each processes trace is written to filename.pid where pid is the numeric process id of each process. This is incompatible with -c, since no per-process counts are kept.

-p switch

Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running.

Answer 2

You could also use the status module and extended status page to see what are the current requests.

Answer 3

You can use the common strace command to trace what your httpd is doing. your command would be like the folloing:

strace -f -o trace.txt /etc/rc.d/init.d/httpd start

The "-f" strace option traces child processes as they are created by currently traced processes as a result of the 'fork' system call. The -o trace.txt is your outputfile to which you want to dump your trace.

Hope this helps