0

We have this apache process which becomes io-bound time to time. Using atop, we can see it is a write operation.

Using lsof -p <PID> we can see a list of files open by the httpd process. First we thought "log" files must be the problem. So we turned them off just to test. However write operations still continues.

We will continue testing a few other things. For instance we use php session variables a lot. Maybe php session files are getting all the writing.

But is there a way to quickly identify files which get written to by the httpd process? This way we can focus our efforts on those files.

UPDATE:

We used the strace command as suggested. Here are two lines from the output.

write(23, "\27\0\0\0\3SET CHARACTER SET utf8", 27) = 27
write(23, "\17\0\0\0\3SET NAMES utf8", 19) = 19

We do not have a mysql process on this server. So is strace also showing what is being written to an ethernet port?

UPDATE2: During high io load, the process which consumes most of the write resources gives the following output to strace -e trace=write -p <PID>:

--- SIGCHLD (Child exited) @ 0 (0) ---
write(9, "!", 1) = 1
write(19, "OPTIONS * HTTP/1.0\r\nUser-Agent: Apache (internal dummy connection)\r\n\r\n", 70) = 70

However I cannot figure out where these are being written to.

UPDATE3: Tried the following command which will show me all files open for writing:

lsof | grep -e "[[:digit:]]\+w"

and below are the only two lines I cannot check. Other files do not show any significant write activity. Could the items below temporarily being written to the hard drive?

httpd     14173       apache    9w     FIFO                0,6           205676000 pipe
httpd     14173       apache    1w      CHR                1,3                3346 /dev/null

REASON FOUND:
Finally we found out that the reason for io-boundness were the php session files. Moving the php session files to another hard drive also moved io-boundness to that hard drive immediately. Probably we will move the sessions to redis or another memory based system. Interestingly, even though the answers below are technically correct, they did not lead us to the session files and kept us looking at different points of the system.

2
  • The last one with the User-Agent is clearly being written to a socket, not a file. Maybe they all are.
    – user207421
    Sep 17, 2012 at 21:31
  • Is there any chance these could be temporarily written to the hard drive, hence creating iowait?
    – Haluk
    Sep 17, 2012 at 21:48

3 Answers 3

2
strace -e trace=write -p <PID>
1
  • Updated with information after using strace.
    – Haluk
    Sep 12, 2012 at 12:31
3

I think lsof is the best tool for it to see the files it currently has opened

lsof -p <pid>
1

Are you sure to look at the right process? Every apache "server" is a separate process, so you should loop the control on every child (you start with the number defined in the StartServers directive).

To see the connections:

 netstat -nap | grep <pid>
2
  • We use "atop" to find the process creating the writes. Once we see an "httpd" process doing a significant amount of writes we check its PID with strace.
    – Haluk
    Sep 12, 2012 at 13:00
  • With lsof you can see every "file" opened by a process, your I/O problem is there.
    – Nosax
    Sep 12, 2012 at 13:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .