Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I see a much larger performance hit with DRBD than their user manual says I should get. I'm using DRBD 8.3.7 (Fedora 13 RPMs).

I've setup a DRBD test and measured throughput of disk and network without DRBD:

dd if=/dev/zero of=/data.tmp bs=512M count=1 oflag=direct
536870912 bytes (537 MB) copied, 4.62985 s, 116 MB/s

/ is a logical volume on the disk I'm testing with, mounted without DRBD

iperf:

[  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

According to Throughput overhead expectations, the bottleneck would be whichever is slower, the network or the disk and DRBD should have an overhead of 3%. In my case network and I/O seem to be pretty evenly matched. It sounds like I should be able to get around 100 MB/s.

So, with the raw drbd device, I get

dd if=/dev/zero of=/dev/drbd2 bs=512M count=1 oflag=direct
536870912 bytes (537 MB) copied, 6.61362 s, 81.2 MB/s

which is slower than I would expect. Then, once I format the device with ext4, I get

dd if=/dev/zero of=/mnt/data.tmp bs=512M count=1 oflag=direct
536870912 bytes (537 MB) copied, 9.60918 s, 55.9 MB/s

This doesn't seem right. There must be some other factor playing into this that I'm not aware of.

global_common.conf

global {
usage-count yes;
}

common {
protocol C;
}

syncer {
al-extents 1801;
rate 33M;
}

data_mirror.res

resource data_mirror {
    device /dev/drbd1;
    disk   /dev/sdb1;

    meta-disk internal;

    on cluster1 {
       address 192.168.33.10:7789;
    }

    on cluster2 {
       address 192.168.33.12:7789;
    }
}

For the hardware I have two identical machines:

  • 6 GB RAM
  • Quad core AMD Phenom 3.2Ghz
  • Motherboard SATA controller
  • 7200 RPM 64MB cache 1TB WD drive

The network is 1Gb connected via a switch. I know that a direct connection is recommended, but could it make this much of a difference?

Edited

I just tried monitoring the bandwidth used to try to see what's happening. I used ibmonitor and measured average bandwidth while I ran the dd test 10 times. I got:

  • avg ~450Mbits writing to ext4
  • avg ~800Mbits writing to raw device

It looks like with ext4, drbd is using about half the bandwidth it uses with the raw device so there's a bottleneck that is not the network.

share|improve this question
    
Can you attempt this with a large file with actual data? Writing all zeros may be a special case handled by the disk for benchmarks. –  Jeff Strunk Nov 4 '11 at 17:11
    
The DRBD user manual recommends measuring throughput the way I did. drbd.org/users-guide-legacy/ch-benchmark.html That's a good idea, the only problem is that if I use a file then the read speed of the disk I read the file from comes into play. I'll see if I can figure out a way to do it that isn't too dependent on the disk I'm reading from. –  BHS Nov 4 '11 at 17:27
    
I tried the same dd commands except getting input from a file on another disk. I got similar results. –  BHS Nov 4 '11 at 20:01
    
Can you please post your drbd configuration files? –  Jeff Strunk Nov 7 '11 at 14:25
    
Can you please give more information about your hardware, too? –  Jeff Strunk Nov 7 '11 at 14:26

1 Answer 1

You are limiting the bandwith with "rate=33M" why? You are using the synchronous protocol "C" why?

I usally use protocol "A" and 8 MB buffer. For a Gigabit line and heavy traffic I limit to "rate=90M".

share|improve this answer
1  
Heavy traffic on a Gigabit line and you set rate to 90M?! You might want to double check that. From the DRBD User Manual: "A good rule of thumb for this value is to use about 30% of the available replication bandwidth." The reason for this is that the "rate" is sync rate --ie, how much B/W to use when sync'ing an Inconsistent device. Setting the sync rate too high can starve the target application of bandwidth. Additionally, DRBD User Manual recommends 33M for a Gig line, which is likely why he chose that value. –  Kendall Jan 1 '12 at 21:39
1  
@Kendall - I don`t replicate via an application interface. I use a dedicated interface/network for this kind of traffic. –  Nils Jan 1 '12 at 21:52
    
As to the "why" on Protocol C, Protocol A leaves open the possibility of data loss, since the write is considered completed when the local disk flush has occurred and the replication packet hits the local send buffer. –  Kendall Jan 1 '12 at 21:53
    
So you sync an Inconsistent disk over a different network interface than the primary DRBD interface ? –  Kendall Jan 1 '12 at 21:57
    
The disk can be inconsistent on the target side of drbd. This does not matter since I do not run databases on the primary side. So worst case is the loss of some logs. The servers have many interfaces used for different purposes. One interface exposes the service to the outside world. I use a DIFFERENT interface for drbd traffic. –  Nils Jan 2 '12 at 22:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.