Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have this apache process which becomes io-bound time to time. Using atop, we can see it is a write operation.

Using lsof -p <PID> we can see a list of files open by the httpd process. First we thought "log" files must be the problem. So we turned them off just to test. However write operations still continues.

We will continue testing a few other things. For instance we use php session variables a lot. Maybe php session files are getting all the writing.

But is there a way to quickly identify files which get written to by the httpd process? This way we can focus our efforts on those files.

UPDATE:

We used the strace command as suggested. Here are two lines from the output.

write(23, "\27\0\0\0\3SET CHARACTER SET utf8", 27) = 27
write(23, "\17\0\0\0\3SET NAMES utf8", 19) = 19

We do not have a mysql process on this server. So is strace also showing what is being written to an ethernet port?

UPDATE2: During high io load, the process which consumes most of the write resources gives the following output to strace -e trace=write -p <PID>:

--- SIGCHLD (Child exited) @ 0 (0) ---
write(9, "!", 1) = 1
write(19, "OPTIONS * HTTP/1.0\r\nUser-Agent: Apache (internal dummy connection)\r\n\r\n", 70) = 70

However I cannot figure out where these are being written to.

UPDATE3: Tried the following command which will show me all files open for writing:

lsof | grep -e "[[:digit:]]\+w"

and below are the only two lines I cannot check. Other files do not show any significant write activity. Could the items below temporarily being written to the hard drive?

httpd     14173       apache    9w     FIFO                0,6           205676000 pipe
httpd     14173       apache    1w      CHR                1,3                3346 /dev/null

REASON FOUND:
Finally we found out that the reason for io-boundness were the php session files. Moving the php session files to another hard drive also moved io-boundness to that hard drive immediately. Probably we will move the sessions to redis or another memory based system. Interestingly, even though the answers below are technically correct, they did not lead us to the session files and kept us looking at different points of the system.

share|improve this question
    
The last one with the User-Agent is clearly being written to a socket, not a file. Maybe they all are. –  EJP Sep 17 '12 at 21:31
    
Is there any chance these could be temporarily written to the hard drive, hence creating iowait? –  Haluk Sep 17 '12 at 21:48

3 Answers 3

up vote 2 down vote accepted
strace -e trace=write -p <PID>
share|improve this answer
    
Updated with information after using strace. –  Haluk Sep 12 '12 at 12:31

Are you sure to look at the right process? Every apache "server" is a separate process, so you should loop the control on every child (you start with the number defined in the StartServers directive).

To see the connections:

 netstat -nap | grep <pid>
share|improve this answer
    
We use "atop" to find the process creating the writes. Once we see an "httpd" process doing a significant amount of writes we check its PID with strace. –  Haluk Sep 12 '12 at 13:00
    
With lsof you can see every "file" opened by a process, your I/O problem is there. –  Nosax Sep 12 '12 at 13:14

I think lsof is the best tool for it to see the files it currently has opened

lsof -p <pid>
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.