I see a much larger performance hit with DRBD than their user manual says I should get. I'm using DRBD 8.3.7 (Fedora 13 RPMs).

I've setup a DRBD test and measured throughput of disk and network without DRBD:

dd if=/dev/zero of=/data.tmp bs=512M count=1 oflag=direct
536870912 bytes (537 MB) copied, 4.62985 s, 116 MB/s

/ is a logical volume on the disk I'm testing with, mounted without DRBD


[  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

According to Throughput overhead expectations, the bottleneck would be whichever is slower, the network or the disk and DRBD should have an overhead of 3%. In my case network and I/O seem to be pretty evenly matched. It sounds like I should be able to get around 100 MB/s.

So, with the raw drbd device, I get

dd if=/dev/zero of=/dev/drbd2 bs=512M count=1 oflag=direct
536870912 bytes (537 MB) copied, 6.61362 s, 81.2 MB/s

which is slower than I would expect. Then, once I format the device with ext4, I get

dd if=/dev/zero of=/mnt/data.tmp bs=512M count=1 oflag=direct
536870912 bytes (537 MB) copied, 9.60918 s, 55.9 MB/s

This doesn't seem right. There must be some other factor playing into this that I'm not aware of.


global {
usage-count yes;

common {
protocol C;

syncer {
al-extents 1801;
rate 33M;


resource data_mirror {
    device /dev/drbd1;
    disk   /dev/sdb1;

    meta-disk internal;

    on cluster1 {

    on cluster2 {

For the hardware I have two identical machines:

  • 6 GB RAM
  • Quad core AMD Phenom 3.2Ghz
  • Motherboard SATA controller
  • 7200 RPM 64MB cache 1TB WD drive

The network is 1Gb connected via a switch. I know that a direct connection is recommended, but could it make this much of a difference?


I just tried monitoring the bandwidth used to try to see what's happening. I used ibmonitor and measured average bandwidth while I ran the dd test 10 times. I got:

  • avg ~450Mbits writing to ext4
  • avg ~800Mbits writing to raw device

It looks like with ext4, drbd is using about half the bandwidth it uses with the raw device so there's a bottleneck that is not the network.

  • Can you attempt this with a large file with actual data? Writing all zeros may be a special case handled by the disk for benchmarks. Nov 4, 2011 at 17:11
  • The DRBD user manual recommends measuring throughput the way I did. drbd.org/users-guide-legacy/ch-benchmark.html That's a good idea, the only problem is that if I use a file then the read speed of the disk I read the file from comes into play. I'll see if I can figure out a way to do it that isn't too dependent on the disk I'm reading from.
    – BHS
    Nov 4, 2011 at 17:27
  • I tried the same dd commands except getting input from a file on another disk. I got similar results.
    – BHS
    Nov 4, 2011 at 20:01
  • Can you please post your drbd configuration files? Nov 7, 2011 at 14:25
  • Can you please give more information about your hardware, too? Nov 7, 2011 at 14:26

1 Answer 1


You are limiting the bandwith with "rate=33M" why? You are using the synchronous protocol "C" why?

I usally use protocol "A" and 8 MB buffer. For a Gigabit line and heavy traffic I limit to "rate=90M".

  • 2
    Heavy traffic on a Gigabit line and you set rate to 90M?! You might want to double check that. From the DRBD User Manual: "A good rule of thumb for this value is to use about 30% of the available replication bandwidth." The reason for this is that the "rate" is sync rate --ie, how much B/W to use when sync'ing an Inconsistent device. Setting the sync rate too high can starve the target application of bandwidth. Additionally, DRBD User Manual recommends 33M for a Gig line, which is likely why he chose that value.
    – Kendall
    Jan 1, 2012 at 21:39
  • 1
    @Kendall - I don`t replicate via an application interface. I use a dedicated interface/network for this kind of traffic.
    – Nils
    Jan 1, 2012 at 21:52
  • As to the "why" on Protocol C, Protocol A leaves open the possibility of data loss, since the write is considered completed when the local disk flush has occurred and the replication packet hits the local send buffer.
    – Kendall
    Jan 1, 2012 at 21:53
  • So you sync an Inconsistent disk over a different network interface than the primary DRBD interface ?
    – Kendall
    Jan 1, 2012 at 21:57
  • 1
    The disk can be inconsistent on the target side of drbd. This does not matter since I do not run databases on the primary side. So worst case is the loss of some logs. The servers have many interfaces used for different purposes. One interface exposes the service to the outside world. I use a DIFFERENT interface for drbd traffic.
    – Nils
    Jan 2, 2012 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.