Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

we have a NFS sitting on top of XFS and drbd which delivers us a horrible performance (about 1MB/s read / write as shown in iostat/iotop) the xfs volume properties are:

meta-data=/dev/drbd0             isize=256    agcount=4, agsize=52427198 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=209708791, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

and we have a Dell Box with an SAS1068E Controller and 2 WD 1TB Disks The volume is currently mounted with the properties:

rw,noatime,nodiratime,attr2,nobarrier,logbufs=8,noquota 

The filesystem contains tons of small files all about 50-100k in size, they are spread around in the directory tree.

We tried playing around with ReadAhead Values (currently disabled) and the xfs mount options but nothing seemed successful so far.

We noticed in the iotop that kdmflush is the process that causes the iowait Any suggestions to improve the Performance of this Setup?

share|improve this question
add comment

3 Answers

The short answer is that your disk system is woefully underspec for what you're trying to do.

1MB/sec is fairly typical of random IO performance on RAID1 on SATA disks. EG, see wmarow's iops anr raid calculator here. Putting two Barracuda ES.2 SATA disks in a RAID10 (effectively the same as a RAID1), setting 100% writes with 0% write cache hit shows an estimated 0.57MB/sec of throughput. Real-world performance may be different, but it's not going to be massively different.

The fact that you identify kdmflush as the responsible kernel process reinforces this - if your disk system is not able to handle the load, it will result in more time spend in iowait in this process. kdmflush is the device-mapper flush process, which handles deferred work due to loading elsewhere.

There are a couple of ways of improving this - get more disks, get better disks, or turn on write caching on the controller.

If you turn on write caching, you will want to get a BBU as well. The BBU may not be an option for the onboard SAS1068E though, so you may have to get a PCI-e controller.

I saw abysmal performance with DRBD when the RAID controllers I was using (3ware 9550 I believe) did not have write cache enabled. Your DRBD loading will be mostly random IO, so write caching will have a significant effect on performance.

The SAS1068E is very low end, and may also be contributing to the problem. If you get more disks or better disks, I'd suggest getting a better controller as well.

A quick google search reveals similarly poor performance with the same model RAID controller you are using.

share|improve this answer
add comment

1 MB/s sounds familiar. At a guess, your problem is less XFS and more in the DRBD layer. If block replication on DRBD is slow for some reason, it is entirely reasonable for kdmflush to be causing lots of IOWAIT. That speed sounds like the network connection between the two DRBD hosts is not being negotiated well.

Again, a guess, but that speed sounds a lot like a TCP connection without TCP Windows working correctly. It should be pretty obvious on a network trace, as the traffic will look like packet, ack, packet, ack, packet, ack rather than many packets and one ack.

If the iotop is being run on a client mounting the NFS share rather than the NFS server itself, when take a look at that connection as well as the DRBD connection.

share|improve this answer
    
The TCP Windows do not seem to be the Problem, the Trace looks normal to me, the IOTOP was run on the Server, but even the tcp windows for the nfs server look ok –  tmsmaster Aug 8 '11 at 19:38
add comment

Use more than a 10Mbps network for your DRBD replication. Your disk I/O on a DRBD device is limited to the network speed (unless you use a protocol other than C, which is what you do if you want your data to become corrupted and useless). To test that it is your network causing the problems, disconnect the primary from the secondary and your I/O rate will probably shoot right up.

share|improve this answer
    
Tried that, the DRBD speed does not seem to be the Problem, the Load did not drop and the speed is still around 1 MB/s, We have a 1Gbps Network link and did set a syncer rate of 40MB/s –  tmsmaster Aug 8 '11 at 4:55
    
Syncer rates are irrelevant for ongoing I/O. If it's not DRBD, then work through the layers to find your bottleneck, like you would with any other performance problem. –  womble Aug 8 '11 at 4:58
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.