Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm seeing very strange performance characteristics on one of my servers. This server is running a simple two-disk software-RAID1 setup with LVM spanning /dev/md0. One of the logical volumes /dev/vg0/secure is encrypted using dmcrypt with LUKS and mounted with the sync and noatimes flag. Writing to that volume is incredibly slow at 1.8 MB/s and the CPU usage stays near 0%. There are 8 crpyto/1-8 processes running (it's a Intel Quadcore CPU).

I hope that someone on serverfault has seen this before :-(.

uname -a

2.6.32-5-xen-amd64 #1 SMP Tue Mar 8 00:01:30 UTC 2011 x86_64 GNU/Linux

Interestingly, when I read from the device I get good performance numbers:

reading without encryption:

$ dd if=/dev/vg0/secure of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes (6.6 GB) copied, 68.8951 s, 95.1 MB/s

reading with encryption:

$ dd if=/dev/mapper/secure of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes (6.6 GB) copied, 69.7116 s, 94.0 MB/s

However, when I try to write to the device:

$ dd if=/dev/zero of=./test bs=64k
8809+0 records in
8809+0 records out
577306624 bytes (577 MB) copied, 321.861 s, 1.8 MB/s

Also, when I read I see CPU usage, when I write, the CPU stays at almost 0% usage. Here is output of cryptsetup luksDump:

LUKS header information for /dev/vg0/secure

Version:        1
Cipher name:    aes
Cipher mode:    cbc-essiv:sha256
Hash spec:      sha1
Payload offset: 2056
MK bits:        256
MK digest:      dd 62 b9 a5 bf 6c ec 23 36 22 92 4c 39 f8 d6 5d c1 3a b7 37
MK salt:        cc 2e b3 d9 fb e3 86 a1 bb ab eb 9d 65 df b3 dd
                d9 6b f4 49 de 8f 85 7d 3b 1c 90 83 5d b2 87 e2
MK iterations:  44500
UUID:           a7c9af61-d9f0-4d3f-b422-dddf16250c33

Key Slot 0: ENABLED
    Iterations:             178282
    Salt:                   60 24 cb be 5c 51 9f b4 85 64 3d f8 07 22 54 d4
                            1a 5f 4c bc 4b 82 76 48 d8 a2 d2 6a ee 13 d7 5d
    Key material offset:    8
    AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
share|improve this question
    
Are you running an AMD-optimized kernel on an Intel CPU? "2.6.32-5-xen-amd64" on "Intel Quadcore CPU"? –  ordnungswidrig Jul 25 '11 at 21:31
    
good catch, most quad intels (i7 + Xeon for example) are EM64T based afaik, which x86_64/amd64 wouldn't take full advantage of. –  Samuel Jones Dec 19 '11 at 23:00
1  
Have you tried mounting without the sync option as that will slow things down, Is it recommended to use sync with LUKS ? The output of pvdisplay,vgdisplay and lvdisplay may be informative ? If you could test write performance of /dev/vg0/secure that would be informative ( BUT DESTRUCTIVE ). –  James Dec 22 '11 at 20:49
    
can you post the mount and top cpu usage output? –  Giovanni Toraldo Dec 22 '11 at 20:55
    
How does the write performance change when you mount it without sync? –  Marcin Jan 7 '13 at 14:12

1 Answer 1

Maybe you're using new hard drives using sectors larger then 512 Byte (like 4K)? If yes you should definitely care about disk alignments. For further reading have a look at the following links describing the problem:

http://bartsjerps.wordpress.com/2011/02/22/disk-alignment-linux/

http://tannerjc.net/wiki/index.php?title=Disk_Alignment

Thanks

share|improve this answer
3  
Welcome to Serverfault. It's considered bad practice to just post a link as an answer. We would prefer it if you posted a summary of the solution in the answer, with attribution to any links. The aim is to have SF a repository of answers, rather than links to answers. Thanks. –  pauska Jan 14 '12 at 2:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.