While troubleshooting a performance issue on a Debian Linux box running ext3 on software RAID 1 I redirected the output of a process writing useless temporary log files to /dev/null (ln -s /dev/null/ foo) which has reduced write operations by around 3MB/s. However the change seems to have also lead to a big increase in Write and Overall IOwait times which I'm at a loss to explain and would really appreciate some input on...

iostat: http://oi45.tinypic.com/5l2mx0.jpg

write latency (change applied at ~3am): http://oi46.tinypic.com/33be1kw.jpg

Update 18.04.13:

Write IOwait remains higher than before but I've applied the following all of which have been at least somewhat beneficial...

  • Remounted the filesystem with noatime option.
  • Changed to deadline ioscheduler.
  • Made vm.swappiness 30 (host is home to ~1GB of database, a CMS and streaming audio).
  • Turned readahead down to 256 down from 4096 for sda and sdb.
  • Turned readahead up to 4096 from 256 for /dev/md2 (the main data partition).

Update 19.04.13:

  • Converted the main filesystem from ext3 to ext4
  • Which IO scheduler are you using? – Gazzonyx Apr 17 '13 at 2:56
  • I'm using the Linux default: CFQ – Fat Finger Apr 17 '13 at 3:11
  • «I redirected the output … which has reduced read operations» — output reduced read? O_o – poige Apr 17 '13 at 3:36
  • Thats a typo it should be writes of course, corrected. – Fat Finger Apr 17 '13 at 3:49
  • Could you post an iostat before and after you route output to /dev/null? – Gazzonyx Apr 17 '13 at 5:24

Logically, the only thing that can increase response time (I/O wait) is increasing the number of operations that have to be carried out on the "same" thing.

Hypothesis: the I/O to /dev/null causes more operations at the driver level than the (possibly more buffered) I/O to a disk

Test: try switching it back to a disk file. If response time falls, there is something to do with /dev/null that takes longer than it should. If response time stays the same, then something happened at the same time as the change that affected response time.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.