0

This is my first question on serverfault. Please treat me kindly if I should miss any important information. Thanks!

I have two separate clusters, each with two nodes. Cluster #1 is running ubuntu 16.04, cluster #2 is running ubuntu 18.04.

On both clusters I use multiple drbd devices with thin provisioned lvm volumes as backing devices.

These drbd devices are used for virtual machines (kvm) with virtio-scsi controller and "discard" set to "unmap".

When I delete some large files on a virtual machine on cluster #1 and run "fstrim -av", the deleted blocks become unmapped and the usage of the thin volume decreases.

I recognized, that this is not working on cluster #2.

In fact, when I run "blkdiscard /dev/sda" on such a machine, volume usage raises to 100%. I believe blocks do not become unmapped, but are overwritten with zeros.

I tried to directly discard a lvm thin volume on the host system: This works without a problem! Next I tried to discard the drbd resource: This does not work! The thin volume is being fully allocated.

Therefore, I am sure this has nothing to do with the virtual machines, but with the combination of drbd and thin volumes.

On ubuntu 16.04 I use a linux kernel with version 4.4 and drbd 8.4.5 with drbdadm 8.9.6.

On ubuntu 18.04 kernel has version 4.15, drbd 8.4.10 and drbdadm 8.9.10.

I found out that there have been problems with drbd nodes with different unmap behaviour in the past. Since version 8.4.7 there is a new configuration option called "discard-zeroes-if-aligned".

It seems that this option, which defaults to yes, should fix exactly my problem. But it does not. Even if I explicitly set it to "yes".

To play safe, we have to disable discard support, if our local backend (on a Primary) does not support "discard_zeroes_data=true". [...]

There are some devices (notably the LVM/DM thin provisioning) that are capable of discard, but announce discard_zeroes_data=false. In the case of DM-thin, discards aligned to the chunk size will be unmapped, and reading from unmapped sectors will return zeroes. However, unaligned partial head or tail areas of discard requests will be silently ignored.

If we now add a helper to explicitly zero-out these unaligned partial areas, while passing on the discard of the aligned full chunks, we effectively achieve discard_zeroes_data=true on such devices.

Setting discard-zeroes-if-aligned to yes will allow DRBD to use discards, and to announce discard_zeroes_data=true, even on backends that announce discard_zeroes_data=false.

( https://docs.linbit.com/man/v84/drbd-conf-5/ )

LVM/DM thin does set discard_zeroes_data=0, because it silently ignores discards to partial chunks.

We can work around this by checking the alignment first. For unaligned (wrt. alignment and granularity) or too small discards, we zero-out the initial (and/or) trailing unaligned partial chunks, but discard all the aligned full chunks.

At least for LVM/DM thin, the result is effectively "discard_zeroes_data=1".

( https://lore.kernel.org/patchwork/patch/672051/ )

After I read this, I believe it might have something to do with the alignment and/or granularity of the discards in combination with the chunk size of the thin volumes!?

These are the values for the thin volume:

# for KEY in /sys/devices/virtual/block/dm-13/queue/discard_*; do
>     echo "${KEY##*/} = $(cat $KEY)";
> done

discard_granularity = 262144
discard_max_bytes = 17179869184
discard_max_hw_bytes = 2147450880
discard_zeroes_data = 0

# fdisk -l /dev/dm-13  |grep -i 'sector size'
Sector size (logical/physical): 512 bytes / 512 bytes

# lvs --unit=b -o lv_name,chunksize VG_V112/ThinPool
  LV       Chunk  
  ThinPool 262144B

And these are the values for the resulting drbd device:

# for KEY in /sys/devices/virtual/block/drbd60/queue/discard_*; do
>     echo "${KEY##*/} = $(cat $KEY)";
> done

discard_granularity = 262144
discard_max_bytes = 4194304
discard_max_hw_bytes = 2147450880
discard_zeroes_data = 0

# fdisk -l /dev/drbd60  |grep -i 'sector size'
Sector size (logical/physical): 512 bytes / 512 bytes

This is my global_common.conf :

# grep -vE '^\s*(#|$)' global_common.conf

global {
    usage-count no;
    minor-count 255;
}
common {
    handlers {
        fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
        before-resync-target "/usr/local/drbd/snapshot-resync-target-lvm8.sh";
        after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh ; /usr/local/drbd/unsnapshot-resync-target-lvm8.sh";
        split-brain "/usr/lib/drbd/notify-split-brain.sh root";
    }
    startup {
        wfc-timeout 20;
        degr-wfc-timeout 20;
        outdated-wfc-timeout 20;
    }
    options {
    }
    disk {
        on-io-error detach;
        disk-flushes no;
        md-flushes no;
        disk-barrier no;
        al-extents 6007;
        fencing resource-only;
        discard-zeroes-if-aligned yes;
    }
    net {
        protocol C;
        allow-two-primaries yes;
        verify-alg sha1;
        csums-alg sha1;
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
        ping-timeout 10;
        max-epoch-size 8000;
        max-buffers 8000;
        sndbuf-size 2M;
    }
}

And this is the resource:

# grep -vE '^\s*(#|$)' r01.res

resource r01 {
    net {
        shared-secret "XXXXXXXXXXXXXXXXXXXX";
        cram-hmac-alg sha1;
    }
    meta-disk internal;
    device minor 60;
    on v112 {
        address 10.22.25.130:7050;
        disk /dev/VG_V112/r01;
    }
    on v111 {
        address 10.22.25.129:7050;
        disk /dev/VG_V111/r01;
    }
}

I could believe that some blocks will not be unmapped after some small files have been deleted. But even if I discard the whole drbd device, the whole space will be allocated instead of being freed.

I hope, anyone has an idea, how to solve this??

Thanks!

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.