4

I'm seeing very strange performance characteristics on one of my servers. This server is running a simple two-disk software-RAID1 setup with LVM spanning /dev/md0. One of the logical volumes /dev/vg0/secure is encrypted using dmcrypt with LUKS and mounted with the sync and noatimes flag. Writing to that volume is incredibly slow at 1.8 MB/s and the CPU usage stays near 0%. There are 8 crpyto/1-8 processes running (it's a Intel Quadcore CPU).

I hope that someone on serverfault has seen this before :-(.

uname -a

2.6.32-5-xen-amd64 #1 SMP Tue Mar 8 00:01:30 UTC 2011 x86_64 GNU/Linux

Interestingly, when I read from the device I get good performance numbers:

reading without encryption:

$ dd if=/dev/vg0/secure of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes (6.6 GB) copied, 68.8951 s, 95.1 MB/s

reading with encryption:

$ dd if=/dev/mapper/secure of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes (6.6 GB) copied, 69.7116 s, 94.0 MB/s

However, when I try to write to the device:

$ dd if=/dev/zero of=./test bs=64k
8809+0 records in
8809+0 records out
577306624 bytes (577 MB) copied, 321.861 s, 1.8 MB/s

Also, when I read I see CPU usage, when I write, the CPU stays at almost 0% usage. Here is output of cryptsetup luksDump:

LUKS header information for /dev/vg0/secure

Version:        1
Cipher name:    aes
Cipher mode:    cbc-essiv:sha256
Hash spec:      sha1
Payload offset: 2056
MK bits:        256
MK digest:      dd 62 b9 a5 bf 6c ec 23 36 22 92 4c 39 f8 d6 5d c1 3a b7 37
MK salt:        cc 2e b3 d9 fb e3 86 a1 bb ab eb 9d 65 df b3 dd
                d9 6b f4 49 de 8f 85 7d 3b 1c 90 83 5d b2 87 e2
MK iterations:  44500
UUID:           a7c9af61-d9f0-4d3f-b422-dddf16250c33

Key Slot 0: ENABLED
    Iterations:             178282
    Salt:                   60 24 cb be 5c 51 9f b4 85 64 3d f8 07 22 54 d4
                            1a 5f 4c bc 4b 82 76 48 d8 a2 d2 6a ee 13 d7 5d
    Key material offset:    8
    AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
6
  • Are you running an AMD-optimized kernel on an Intel CPU? "2.6.32-5-xen-amd64" on "Intel Quadcore CPU"? Jul 25, 2011 at 21:31
  • good catch, most quad intels (i7 + Xeon for example) are EM64T based afaik, which x86_64/amd64 wouldn't take full advantage of. Dec 19, 2011 at 23:00
  • 1
    Have you tried mounting without the sync option as that will slow things down, Is it recommended to use sync with LUKS ? The output of pvdisplay,vgdisplay and lvdisplay may be informative ? If you could test write performance of /dev/vg0/secure that would be informative ( BUT DESTRUCTIVE ).
    – James
    Dec 22, 2011 at 20:49
  • can you post the mount and top cpu usage output? Dec 22, 2011 at 20:55
  • How does the write performance change when you mount it without sync?
    – Marcin
    Jan 7, 2013 at 14:12

1 Answer 1

2

Maybe you're using new hard drives using sectors larger then 512 Byte (like 4K)? If yes you should definitely care about disk alignments. For further reading have a look at the following links describing the problem:

http://bartsjerps.wordpress.com/2011/02/22/disk-alignment-linux/

http://tannerjc.net/wiki/index.php?title=Disk_Alignment

Thanks

1
  • 3
    Welcome to Serverfault. It's considered bad practice to just post a link as an answer. We would prefer it if you posted a summary of the solution in the answer, with attribution to any links. The aim is to have SF a repository of answers, rather than links to answers. Thanks.
    – pauska
    Jan 14, 2012 at 2:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .