Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have the following issue: I have an encrypted partition on a lvm on a plain partition on the hd. Now I craeted a RAID array to increase the performance. That means, that I have the following stack: HD(s) -- partition(s) -- RAID -- LVM -- Cryptsetup/LUKS. Now the performance is more or less the same with and without RAID (some measurements below).

Can anybody give me a hint why the performance does not gain a boost?

The measurements: First the output of hdparm -t ...:

/dev/server-multimedia/pics:
 Timing buffered disk reads: 208 MB in  3.00 seconds =  69.22 MB/sec

/dev/mapper/pics:
 Timing buffered disk reads: 198 MB in  3.01 seconds =  65.77 MB/sec

/dev/server_raid/pics:
 Timing buffered disk reads: 860 MB in  3.01 seconds = 286.09 MB/sec

/dev/mapper/pics_test:
 Timing buffered disk reads: 204 MB in  3.00 seconds =  67.98 MB/sec

/dev/server-multimedia/pics is the partition without raid and encryption. /dev/mapper/pics the partition opend by luks. /dev/server_raid/pics the raid based encrypted partition and /dev/mapper/pics_test is the luks-opened, raid based partition.

We can see that for the plain partition we do not loose much at all. Where the raid based partition seems to have a very bad performance.

I also checked the cpu state while the test ran. First with the plain partition:

top - 13:07:41 up 5 days,  2:41,  4 users,  load average: 0.22, 0.27, 0.16
Tasks: 287 total,   2 running, 285 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  1.2%sy,  0.3%ni, 95.1%id,  2.8%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:    900980k total,   752636k used,   148344k free,   178452k buffers
Swap: 26364332k total,   116044k used, 26248288k free,    95880k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25592 root      20   0     0    0    0 R   92  0.0   0:24.13 kcryptd
 6159 root      20   0  3824 3820 1492 D   20  0.4   0:00.83 hdparm
25591 root      20   0     0    0    0 S    4  0.0   0:00.38 kcryptd_io
 6168 christia  20   0  2612 1176  796 R    2  0.1   0:00.02 top

Now with raid:

top - 13:07:54 up 5 days,  2:41,  4 users,  load average: 0.25, 0.27, 0.16
Tasks: 287 total,   3 running, 284 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  1.2%sy,  0.3%ni, 95.1%id,  2.8%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:    900980k total,   619508k used,   281472k free,    86760k buffers
Swap: 26364332k total,   116044k used, 26248288k free,    55952k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
5594 root      20   0     0    0    0 R   84  0.0   0:06.76 kcryptd
6159 root      20   0  3824 3820 1492 D   18  0.4   0:02.94 hdparm
 404 root      20   0     0    0    0 R    4  0.0 102:20.06 md1_raid5
5593 root      20   0     0    0    0 S    4  0.0   0:00.39 kcryptd_io
6170 christia  20   0  2612 1180  796 R    2  0.1   0:00.01 top

I can see a quite high CPU load. But in the top lines most of the time the CPUs seems to do nothing (approx. 90% idle). So what? Is my CPU the bottle neck?

share|improve this question
    
What's your RAID setup and what's your hardware configuration? –  Spack Apr 24 '13 at 11:42
    
What is your CPU? Have you aligned your partitions with suitable (4kB?) sector size? –  ptman Apr 24 '13 at 12:20
    
@Spack Is is a software RAID5 with 3 identical disks. –  Christian Wolf Apr 24 '13 at 12:45
    
@ptman AMD Athlon 64 X2 5200+. Sector size of hd is 4KiB. RAID partitions start at (logical) sector 19531776. Should be aligned correctly for this layer. Then the RAID/LVM/partition layers I do not know how to check alignment. –  Christian Wolf Apr 24 '13 at 12:48

3 Answers 3

Most likely LUKS is the limit here, as AFAIK it uses only one thread per encrypted block device and it appears that kcryptd uses nearly 100% CPU power (and I would guess that with a load of ~0.25 you have 4 cores).

share|improve this answer

That looks like you have a CPU with 4 cores (or at least Linux treating it like 4) and dm-crypt takes one core completely without being able to use the others. If the CPU does not allow more than 70 MiB/s then increasing I/O speed does not make a difference, of course.

I am surprised, though. dm-crypt shall be multithreaded since kernel 2.6.38 (that was March 2011). Maybe you can increase the throughput by configuring a different cipher. Or you get a CPU with AES-NI (crypto in hardware, unlimited speed...). What is your cipher (cryptsetup luksDump /dev/... | grep Cipher)?

Edit 1

I just found this on http://code.google.com/p/cryptsetup/wiki/FrequentlyAskedQuestions:

Since version 1.60 cryptsetup supports the "benchmark" command. Simply run as root: cryptsetup benchmark

I have 1.4.2 (kernel 3.4.33) so I could not give that a try.

share|improve this answer
    
Debian Squeeze is 2.6.32 AFAIR. –  tim Apr 24 '13 at 12:36
    
@tim Doesn't happen often that an update dangles a 400% performance increase. ;-) –  Hauke Laging Apr 24 '13 at 12:38
    
It's only 200% as ist a dual core but that's quite enough... PS: The benchmark is impossible as debian used version 1.1.3. Bad. –  Christian Wolf Apr 24 '13 at 12:42
    
@ChristianWolf But probably a dual core with HT. If a task consumes about 100% on just two cores then the load would be about 0.5 instead of about 0.25. If you make a kernel update don't forget to update this thread. –  Hauke Laging Apr 24 '13 at 12:53
up vote 0 down vote accepted

So now, I updateds my machine. The result is an increased performance. I get valies of about 110 MB/sec. So it seems that the encrytion was a problem for the cpu.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.